CyberArk Labs Research: Securing Jenkins Java Web Start Agents

 

Jenkins: Distributed Builds

Jenkins is an open source automation server used to accelerate the software delivery process and is considered the leading CI/CD tool for automating projects. With thousands of plugins to choose from, Jenkins helps DevOps teams build agile software delivery pipelines that would otherwise be time-consuming and laborious to construct and maintain.

Out-of-the-box Jenkins comes as a server that runs all builds on the single master. Sometimes, however, an organization may require that builds be distributed across a cluster of machines for security reasons, to achieve scale, build across different platforms or provide for different test tools. Termed ‘distributed builds,’ this ability to seamlessly integrate and distribute delivery pipelines to different agents is one of Jenkins’ core benefits.

The support for a master/agent architecture, in which the workloads of integration and delivery pipelines are divided to multiple nodes called “agents,” allows a single Jenkins installation to automate as many projects as the organization requires. It also provides support for different environments needed to build or test. For example, building and testing a Windows application may require different agent settings than building and testing a Linux application.

The Jenkins automation server is delivered out-of-the-box with only the ‘master’ instance.
In this basic installation the master is handling all managerial tasks, such as job configuration, credentials storage, and managing the user’s database, plus pipeline-related tasks, which include building code, running tests and deploying artifacts.

In a previous research post, “Configuring and Securing Credentials in Jenkins,” we explained that running jobs on the master is risky because of security issues embedded in the Jenkins mode of operation, and should be avoided at all costs. A better Jenkins setup must include at least one ‘agent,’ hence why we use Jenkins distributed builds architecture.

The agent offloads most pipeline related tasks from the master, while the master handles managerial tasks. The master would still serve all HTTP requests and run the orchestration processes, but delegate pipeline-related tasks to build agents and distribute the workload to them automatically. The exact delegation behavior depends on each project’s configuration: some projects may choose to always run on a particular machine, while others may be indifferent as to which agent machines they run on and let the Jenkins master choose the agent for them.

There is no need to install the full Jenkins software on individual agents as each agent runs a special Java program that is in charge of establishing a bi-directional secure communication link between the master and the agent. This communication link is used to extend pipeline commands from the master to the agent.

In the next sections we show how a relatively low privileged user can leverage a distributed builds mechanism called ‘Java Web Start’ and exploit its weaknesses to take full control over a Web Start agent, gain persistency and eventually even escalate privileges to become a Jenkins admin.

It is fair to note that the issues described in this research are not specific to the Java Web Start technology, but apply equally to all connection methods that rely on agent machine initiating the connection, and therefore holding machine identity secrets.

Methods for Starting Agents

Jenkins supports a number of different methods for launching a master-agent connection. The three most common methods are:

1. Launch the master via SSH agent – this method uses the Jenkins preinstalled SSH client to initiate a connection from master (SSH client) to agent (SSH server). This is a convenient method for launching Linux agents, which normally have SSH server software installed out-of-the-box. This method is most common for launching and using Linux agents.
2. Master launches the agent on Windows – this method uses the remote management facility built into Windows Management Instrumentation (WMI) to launch and control a Windows agent remotely. This is a convenient setup for a Windows agent, however, it does not allow running programs that require display interactivity, such as tests, and seem to have some issues with the Jenkins-Windows integration.
3. Launch the agent using Java Web Start – this method uses a special Java protocol to start a connection from the agent to master, which is the opposite direction to the previous two methods. This method can be used for both Linux and Windows agents, but is commonly used with Windows agents. It is of particular interest to us since it is said to be more secure than the other methods. We will concentrate on this method in the next sections.

Using Java Web Start to Launch Jenkins Agents

Java Web Start is an application deployment technology for the Java platform, in which users start Java applications directly from a web browser.

Launching agents using this method starts at the agent, rather than at the master.
This method is convenient when the master cannot initiate a connection to the agent, for example, when the Jenkins master is accessible on a public network, while leaving the build agent within the firewall, typically for reasons of cost and security[i].

Using this approach, an admin user (or a user with Agent Connect permission) must logon interactively to the agent machine, open a browser and logon to the Jenkins master. Then, using a special page on the master, launch the agent using Java Web Start: the admin user would go to Manage Jenkins > Manage Nodes and click on the new agent’s name, to be connected to. At this point the following web page appears on the agent’s browser:

We can see in the image above that Jenkins supports two methods for launching a Java Web Start agent:

The admin user can either press the  button, or copy the following line and paste it to the agent’s terminal command line:

java -jar agent.jar –jnlpUrl <jenkinsURL>/computer/<slaveName>slave-agent.jnlp –secret <64 chars secret> -workDir "c:\JenkinsAgent"

We will analyze and try to exploit these two methods next, but before that, here is some background on how a Jenkins agent actually finds and connects to the master when a Java Web Start launch is requested. If you are not interested in the inner workings of Java Web Start, you can skip this section and pick up at the next section, “Abusing Java Web Start Agents.”

Inner Workings of the Jenkins Java Web Start

Earlier we mentioned that the reason Jenkins is using Java Web Start was mainly because agents are required to actively connect to a master. The Jenkins Project considers using Java Web Start a more secure practice because it places the agents behind a firewall in a segregated network, while allowing the master to serve users on a public network.

The Jenkins Java Web Start mechanism is composed of a number of sequential steps:

1. 1. First, the ‘JNLP[ii] agents’ option should be enabled in the Jenkins Configure Global Security When this option is enabled, Jenkins answers HTTP requests with a set of cookie objects that contain the target JNLP port the master is listening on. Sending an HTTP GET request to:

      <jenkinsURL>/tcpSlaveAgentListener

      Will return a list of cookies, among which is the port for Jenkins JNLP connections:

      X-Jenkins-JNLP-Port: 50893

      Another cookie the agent uses at a later stage is the X-Instance-Identity, which contains a Jenkins public key.

      This master is listening on this port (50893 in our example) for incoming agent connections.

   2. When the  button is pressed a special XML snippet, the JNLP file, is downloaded to the agent with instructions on how to launch the Java Web Start application and other important information. Here’s an example of a JNLP file:

      

      In the red frame above is the URL of the agent’s web page on the Jenkins master. In the blue frame is the security definitions of the Java Web Start. In green is the URL of the Java executable jar file on the master. In the purple frame is the secret, which will later be used as the token to identify the agent to the master.When using the second launch method, i.e. launching an agent via command line, copied from the Manage Nodes web page, the JNLP file is encrypted before downloading to the agent. The agent then uses (part of) the secret in the command line to decrypt the JNLP file and use it to establish connection to the master.

   3. Java Web Start is now launched by the Java Runtime Environment with the JNLP file as the executable reference. Java Web Start would load the jar file from the master (the URL in the green frame above) and start executing it with the arguments listed in the grey frame. One of these argument is, once again, the ‘secret’ used to identify the agent.
   4. The jar Java code is executed on the agent. It reads the port number from the master cookies (as described in section 1 above). The code uses this port to issue a HTTP GET to <jenkinsURL>:50893 in order to retrieve the following information from the master:

      Jenkins-Agent-Protocols: JNLP4-connect, Ping
      Jenkins-Version: 2.106-SNAPSHOT (private-05/06/2018 14:20 GMT-nimrod)
      Jenkins-Session: f5b1f99b
      Client: 192.168.190.1
      Server: 192.168.190.131
      Remoting-Minimum-Version: 2.60

      Jenkins is notifying the agent which protocols are supported (JNLP4-connect and ping), what is the minimum remoting version (2.60) and the current Jenkins version.

   5. Next, the agent opens a TCP socket to the master on the JNLP port (extracted in section 1) and start a TLS handshake. At the end of the handshake, the master’s TLS public key is compared with a fingerprint calculated from the X-Instance-Identity extracted from the cookies[iii].
   6. At this point, the agent and master are securely connected using the JNLP4-connect protocol over TLS. This means that communication from the master to the Java process are end-to-end encrypted.

Abusing Java Web Start Agents

As we’ve seen above, Jenkins supports two methods for launching the JNLP agent: (1) logging into a Jenkins master from a browser on the agent and pressing a launch button, or (2) running a special command from the agent’s terminal, usually copied from the agent’s browser.

Pressing the launch button saves the unencrypted JNLP file to disk, at least in the browser’s cache. Firefox, for example, stores cached downloaded files in:

Depending on the browser’s configuration, the JNLP file may also be saved to the user’s Downloads directory, and a third copy of the JNLP file is saved to the Java Web Start cache directory[iv]. A fourth copy of the JNLP file contents is created after using the “Install as a Service” option in the agent’s Working Directory on the Windows Operating System.

A job configurator user or an attacker with access to the agent can locate the unencrypted JNLP files in the browser cache (or elsewhere in the filesystem) and take full control of that agent.

Moreover, if a Jenkins user is running the java executable from the command line, the entire command line can be easily exposed, including the secret field, using a memory dump of the Java process or simply dumping the command line in Linux (E.g., by using  cat /proc/<pid>/cmdline).

A Windows Java process dump from a Java Web Start agent looks like:

 

By extracting the information (circled in red) from the dump, an attacker can ‘hijack’ the agent to transfer all jobs running on that agent to an attacker controlled machine.

This opens several new attack vectors:

(1) The first attack vector involves using the on-disk JNLP file. An attacker would copy the JNLP file to their own machine and run (on Windows):

javaws.exe <filename.JNLP>

This launches a Java Web Start session, which would connect to the master. The master would identify the secret and connect the attacker machine as the Jenkins agent.

What would an attacker gain by hijacking a Jenkins agent?

There are a number of reasons we believe replacing an agent may be dangerous:

a. A Jenkins agent may be running a number of jobs on several executors. Any access control restrictions imposed on a job configurator user are operating system related and are configured on the machine, unrelated to the Jenkins role the agent is filling.

If a rogue job configurator user can transfer all jobs to a machine they control, hence access control restrictions removed, the user would have control over all processes running on that machine. This entails access to workspaces, secrets, artifacts and more of other processes.

For example, running two Web Start agents on a single Linux agent using the command line method. Each Web Start agent is executed on a different Linux user so that workspaces of the two agents are separated. However, one user can easily dump the command line of the process of the other user and capture the entire command line used to execute the agent. An attacker can use the command line arguments (which includes the secret argument) to migrate the two ‘agents’ to a machine they control. This results in full control over both agents, practically canceling Linux user separation.

b. With control of the Jenkins agent, an attacker can replace a Java Web Start agent with their own machine, outside of the protected network and behind an onion router. An attacker gains anonymity, persistence and access to a valid Jenkins agent.

c. Control over a Jenkins agent allows an attacker to have control over the availability of all Jenkins processes on that agent. An attacker can, upon will, drop builds and fail tests running on that agent.

d. Having full control over a Jenkins agent makes it easier to replace the agent’s software, the agent.jar file, running the agent side of the remoting communications with the Jenkins master. This way an attacker can easily decrypt the TLS communications and uncover credentials transferred from master to agent[v].

(2) A second attack vector involves hijacking the agent by rebuilding the Java Web Start command line using the information in the Java process dump or the command line in Linux. Practically, we only need the secret and the master’s JNLP file URL to hijack the agent again.

(3) A third attack vector is related to Linux machines. If running from command line, CloudBees recommends using the following line on the terminal:

nohup java -jar /opt/jenkins/slave.jar -jnlpUrl http://YourJenkinsMaster:8080/computer/NameOfYourSlave/slave-agent.jnlp -jnlpCredentials userJenkinsMaster:passwordJenkinsMaster &

Note that instead of the secret used on the Windows agents, on Linux CloudBees recommends using the master’s admin username and password for launching the Java Web Start agent. This would expose the admin’s username and password to all users on the Linux machine, since the command line parameters are not considered a boundary on Linux[vi] and may be dumped using a simple bash command[vii].

(4) Another risk associated with Java Web Start agents is the exposure of privileged users’ credentials by Key Loggers installed on the agent. Since a privileged user must interactively login to the Java Web Start agent every time the agent needs to be re-launched and then login to the Jenkins Web API with their Jenkins user’s credentials, these may be exposed if a simple key logger is installed on the agent.

An unsuspecting admin approaching an offline Java Web Start agent (due to a long power-outage for instance) will type their password in the browser to get the  button, which launches the agent. This may be intercepted by an attacker, retrieving the username and password of the Jenkins admin.

This method effectively opens a window for privilege escalation for a job configurator user or an attacker on the machine, and eventually allows them access to tier zero assets, becoming the admin of a Jenkins master.

Conclusion

The Java Web Start technology is used today as a method for launching Jenkins agents, especially Windows build and test machines. Due to reasons unrelated to our research, Java Web Start will no longer be included in future Oracle Java versions. However, Oracle intends to extend support for the Web Start technology through March 2025.

In this research, we’ve introduced the Jenkins Java Web Start method of launching agents, which is especially effective for Windows agents. Jenkins documentation considers it to be more secured than other forms of agent launching[viii]. However, our research shows that a job configurator or an attacker on the agent are able to expose the Jenkins admin’s credentials and replace the agent with their own machine, while ‘hijacking’ the Jenkins master to believe it is still connected to the original agent.

Replacing an agent allows attackers access to secrets, read and write files in workspaces associated with other jobs and generally have control over an organization’s build and test infrastructure as demonstrated above.

It is imperative that Jenkins admins are aware of these risks inherent to the Java Web Start protocol, and design their Jenkins infrastructure to minimize over-exposure. Mitigations are offered in the next section.

Mitigations

The Jenkins Project security team was made aware of these issues. Their general response was that this is as-designed behavior since users are responsible for protecting the JNLP file just like they should protect private keys, and that in untrusted environments admins should use master-initiated connection protocols, rather than Java Web start, which is agent-initiated.

The mitigations proposed are therefore:

1. Where possible, use only master-initiated connection protocols, such as SSH. It is important to note that the issues described in this research are not specific to Java Web Start, but apply equally to all connection methods which rely on agent machine initiating the connection, and therefore holding machine-identity secrets.
2. Admin user credentials should be protected by multi-factor-authentication (MFA). The breadth of an admin’s capabilities and permissions in Jenkins and the risk of misuse warrant that. However, if you must use Java Web Start, it is recommended to setup a Jenkins Agent Connect user with restricted permissions to deal with Java Web Start agent launching, as shown in the next image:

Only two permission are required, read and connect agent. This will dramatically reduce the attack surface and limit exposure from credential leakage, key logger attacks and command line exposure, as detailed above.

3. When running Java Web Start under a Windows agent, Java offers an option to install the agent process as a Windows service. Installing as a service is good for cases of agent reboot (e.g. after a power failure), at which time Windows will re-execute the service automatically, thus not requiring an admin to relaunch the agent, limiting credential leakage.However, Jenkins admins must be aware of the fact that this is not true in other cases such as network failure or even user interruption[ix]. In those cases an admin user is still required to step up and relaunch the agent, thus exposing their credentials on the agent to key loggers and command line disclosure.
4. Generally, use the ‘master-as-a-fortress’ paradigm as a security strategy: Focus your resources on protecting the master while treating the agents as completely untrusted. Therefore, storing secrets on, and initiating communications from, an untrusted agent cannot be considered a security best practice. On the other hand, initiating communication from and storing private keys on a protected Jenkins master is far more secured.

 

---

[i] See https://wiki.jenkins.io/display/JENKINS/Distributed+builds#Distributedbuilds-AccessanInternalCIBuildFarm(Master+Agents)fromthePublicInternet

[ii] JNLP stands for Java Network Launch Protocol. It is also the extension of the file containing the protocol’s information.

[iii] This provides another very good reason to use a properly configured HTTPS encrypted protocol for the Jenkins master. Using a non-encrypted protocol might allow a man-in-the-middle to hijack and replace the Jenkins master, by offering a different X-Instance-Identity to the agent.

[iv] On our Windows 10 installation a cache file containing the JNLP XML is located at: %HOMEPATH%\AppData\LocalLow\Sun\Java\Deployment\cache\6.0\3

[v] See CyberArk research post #1: Configuring and Securing Credentials in Jenkins

[vi] “How to create an agent in Linux from console” https://support.cloudbees.com/hc/en-us/articles/115003929412-How-to-create-an-agent-in-Linux-from-console

[vii] Any user can dump other processes’ command line arguments on Linux. See, for example, https://unix.stackexchange.com/questions/163145/how-to-get-whole-command-line-from-a-process

[viii] See footnote i above

[ix] Running an agent ‘as a service’ is not recommended because it elevates privileges of the agent process to Windows administrator. A rogue job configurator running jobs on the agent will therefore have Windows administrator privileges on the agent. This is contrary to the concept of least-privilege. Furthermore, using the Java ‘install as a service’ only compounds the problem, since it copies the JNLP arguments to files created on disk. Now the poor admin has to deal with another file to protect.