Tripping the Jenkins Main Security Circuit-Breaker: An Inside Look at Two Jenkins Security Vulnerabilities

 

CyberArk Labs has discovered several vulnerabilities in the leading open source automation server Jenkins. This blog examines two of the more significant issues.

Following responsible disclosure, the vulnerabilities were reported to CloudBees, which supports Jenkins. To date, four vulnerabilities have been addressed and fixed, as noted in Jenkins’ July and August security advisories.

Jenkins is a valuable tool, and it’s CyberArk Labs’ goal to educate organizations on security risks and offer recommended mitigations and best practices. This is the fourth in a series of Jenkins-focused research posts from CyberArk Labs, following a recent post on potential security risks introduced by Jenkins plugins.

CVE-2018-1999001

Let’s first look at CVE-2018-1999001, which is a high severity issue rated with a CVSS 3.0 of 8.8. We’ll explain how this vulnerability was identified using actual code samples and offer possible mitigations.

This vulnerability can be exploited when unauthenticated attackers remove files from the Jenkins master file system, causing Jenkins to trip its main security switch, and in some cases completely abandon all security safeguards – including those that are meant to protect the Jenkins server from data breaches, allowing unauthenticated users admin access to Jenkins.

How It Works

Every new user of Jenkins, whether it is a user of Jenkins’ own user database or other user databases like Active Directory, is welcomed with special gifts: a new directory is created for the username and a new config.xml file is written there with information regarding the new user, such as an encrypted API token and the user’s email address and full name. The encrypted API token is a randomly generated token that may be used to make scripted clients, such as bots, impersonate the new user and invoke operations that require authorization (such as scheduling a build or changing a job script)[i].

For every new user, Jenkins prepares room on disk to save the details of the new user. The term ‘database’ is a bit misleading as Jenkins saves the new user’s information in a special directory called “users” as seen here:

Every username is given a directory under JENKINS_HOME/users/, and inside there is a single config.xml file. If Jenkins is set to use its own users’ database, this config.xml file contains the hashed user’s password, its encrypted API token, full name and more. See examples of user’s config file in appendix B.

The Jenkins Main Security Switch

With this vulnerability we aim to (ab)use the Jenkins authentication mechanism in order to flip the Jenkins main security switch and allow any user admin access. In this section we explain what this main security switch is all about.

The Jenkins Global Security Configuration page opens with an ‘enable security’ checkbox. This checkbox is selected by default since Jenkins 2.0:

The ‘enable security’ checkbox of the Jenkins web UI allows an administrator to enable, configure or disable all security features of Jenkins, including those of user authentication and authorization. By unchecking the box, an administrator is able to make Jenkins totally accessible to anonymous, unauthenticated users.

The enable security state of the Jenkins master is stored in another file on the JENKINS_HOME directory. It is the Jenkins master configuration file. Quite interestingly, the name of this file is also config.xml. See an example file in appendix A.

Removing the Main config.xml File

The fact that every Jenkins user can authenticate using their password or by using an API token makes the authentication processes a little complicated, and requires a dual authentication mechanism.

The dual authentication mechanism is implemented in Hudson.Security.BasicAuthenticationFilter.java.

Why dual? Jenkins supports two types of authentication: HTTP basic authentication and form-based authentication.
The HTTP basic authentication is reserved for scripted clients (e.g., bots impersonating human users to run scripts on Jenkins), and the form-based authentication is for humans to login with their usernames and passwords via the web UI.

Using the HTTP basic authentication, a client authenticates to the Jenkins master with a username and password or API token combination. In this basic authentication, the client’s username and password/API token should be concatenated, base-64 encoded and passed in the authorization HTTP header as follows:

Authorization: Basic dm9yZGVsOnZvcmRlbA==

The Jenkins master authenticates the user against a users’ database, in case of a username:password combination, or by comparing tokens with the local user’s API token, stored in the user’s config.xml file, in case of username:APItoken authentication.

When HTTP GET is issued with the “Authorization: Basic” header, the doFilter function is called in the Jenkins Hudson.Security.BasicAuthenticationFilter module (code snippet 1, line 3). This function extracts the “Authorization” header at line 6 of code snippet 1 and decodes the base64 information by calling (line 29):

String uidpassword = Scrambler.descramble(authorization.substring(6));

The string in uidpassword now contains the decoded base64 string in the format: “username:password.”
Next, username and password variables are populated (lines 32 and 33, code snippet 1).
Jenkins then calls the getById function (at line 45):

User u = User.getById(username, true);

This is called with two parameters: the username, extracted from the authorization HTTP header, and ‘true.’ The ‘true’ parameter is quite interesting, and we’ll return to it shortly.

public class BasicAuthenticationFilter implements Filter {
private ServletContext servletContext;
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
HttpServletRequest req = (HttpServletRequest) request;
HttpServletResponse rsp = (HttpServletResponse) response;
String authorization = req.getHeader("Authorization");

String path = req.getServletPath();
if(authorization==null || req.getUserPrincipal() !=null || path.startsWith("/secured/")
|| !Jenkins.getInstance().isUseSecurity()) {
// normal requests, or security not enabled
if(req.getUserPrincipal()!=null) {
// before we route this request, integrate the container authentication
// to Acegi. For anonymous users that doesn't have user principal,
// AnonymousProcessingFilter that follows this should create
// an Authentication object.
SecurityContextHolder.getContext().setAuthentication(new ContainerAuthentication(req));
}
try {
chain.doFilter(request,response);
} finally {
SecurityContextHolder.clearContext();
}
return;
}
// authenticate the user
String username = null;
String password = null;
String uidpassword = Scrambler.descramble(authorization.substring(6));
int idx = uidpassword.indexOf(':');
if (idx >= 0) {
username = uidpassword.substring(0, idx);
password = uidpassword.substring(idx+1);
}

if(username==null) {
rsp.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
rsp.setHeader("WWW-Authenticate","Basic realm=\"Jenkins user\"");
return;
}

{// attempt to authenticate as API token
// create is true as the user may not have been saved and the default api token may be in use.
// validation of the user will be performed against the underlying realm in impersonate.
User u = User.getById(username, true);
ApiTokenProperty t = u.getProperty(ApiTokenProperty.class);
if (t!=null && t.matchesPassword(password)) {
UserDetails userDetails = u.getUserDetailsForImpersonation();
Authentication auth = u.impersonate(userDetails);

SecurityListener.fireAuthenticated(userDetails);

SecurityContextHolder.getContext().setAuthentication(auth);
v {
chain.doFilter(request,response);
} finally {
SecurityContextHolder.clearContext();
}
return;
}
}

path = req.getContextPath()+"/secured"+path;
String q = req.getQueryString();
if(q!=null)
path += '?'+q;

// prepare a redirect
rsp.setStatus(HttpServletResponse.SC_MOVED_TEMPORARILY);
rsp.setHeader("Location",path);

// ... but first let the container authenticate this request
RequestDispatcher d = servletContext.getRequestDispatcher("/j_security_check?j_username="+
URLEncoder.encode(username,"UTF-8")+"&j_password="+URLEncoder.encode(password,"UTF-8"));
d.include(req,rsp);
}

Code snippet 1: Hudson.Security.BasicAuthenticationFilter.java

Switching to the Hudson.model.User module, we see that the GetById function in Hudson.model.User calls another function:

public static @Nullable User getById(String id, boolean create) {
return getOrCreate(id, id, create);
}

Code snippet 2: Hudson.model.User.java

 

Notice that the interesting ‘true’ parameter is now called “create.”

The getOrCreate function in the User.java module calls another getOrCreate function, but adds another interesting parameter, returned from getUnsanitizedLegacyConfigFileFor function:

… User getOrCreate(@Nonnull String id, @Nonnull String fullName, boolean create) {
return getOrCreate(id, fullName, create, getUnsanitizedLegacyConfigFileFor(id));
}

Code snippet 3: Hudson.model.User.java

 

This is where it gets interesting. It seems that there was some Jenkins legacy file system that was used to populate the users’ database.

Digging in, we found two sister functions relating to this legacy database:

private static final File getConfigFileFor(String id) {
return new File(getRootDir(), idStrategy().filenameOf(id) +"/config.xml");
}

private static File getUnsanitizedLegacyConfigFileFor(String id) {
return new File(getRootDir(), idStrategy().legacyFilenameOf(id) + "/config.xml");
}

Code snippet 4: Hudson.model.User.java

 

The first function – getConfigFileFor(id) –  gets the users database directory of Jenkins (JENKINS_HOME/users) and adds some form of the username (id) and returns a file name consisting of:

JENKINS_HOME/users/{username}/config.xml

The second function is doing quite the same thing, but this time taking the username part from idStrategy().legacyFilenameof().

As the name suggests, the legacyFilenameof() function generally leaves the username string unchanged, while the filenameOf() function used by getConfigFileFor(), sanitizes the username string significantly in order to prevent unsafe use of user names as directory names or other reserved names (See SECURITY-499).

Next, the Jenkins code calls the getOrCreate() function:

private static @Nullable User getOrCreate(@Nonnull String id, @Nonnull String fullName, boolean create, File unsanitizedLegacyConfigFile) {

String idkey = idStrategy().keyFor(id);

byNameLock.readLock().lock();
User u;
try {
u = AllUsers.byName().get(idkey);
} finally {
byNameLock.readLock().unlock();
}

final File configFile = getConfigFileFor(id);
if (unsanitizedLegacyConfigFile.exists() && !unsanitizedLegacyConfigFile.equals(configFile)) {
File ancestor = unsanitizedLegacyConfigFile.getParentFile();
if (!configFile.exists()) {
try {
Files.createDirectory(configFile.getParentFile().toPath());
Files.move(unsanitizedLegacyConfigFile.toPath(), configFile.toPath());
LOGGER.log(Level.INFO, "Migrated user record from {0} to {1}", new Object[] {unsanitizedLegacyConfigFile, configFile});
} catch (IOException | InvalidPathException e) {
LOGGER.log(
Level.WARNING,
String.format("Failed to migrate user record from %s to %s", unsanitizedLegacyConfigFile, configFile),
e);
}
}

// Don't clean up ancestors with other children; the directories should be cleaned up when the last child
// is migrated
File tmp = ancestor;
try {
while (!ancestor.equals(getRootDir())) {
try (DirectoryStream<Path> stream = Files.newDirectoryStream(ancestor.toPath())) {
if (!stream.iterator().hasNext()) {
tmp = ancestor;
ancestor = tmp.getParentFile();
Files.deleteIfExists(tmp.toPath());
} else {
break;
}
}
}
} catch (IOException | InvalidPathException e) {
if (LOGGER.isLoggable(Level.FINE)) {
LOGGER.log(Level.FINE, "Could not delete " + tmp + " when cleaning up legacy user directories", e);
}

Code snippet 5: Hudson.model.User.java

This function takes a string unsanitizedLegacyConfigFile as a parameter, and at line 13 of code snippet 5, calls getConfigFileFor(id) to compute the ‘sanitized’ username directory. ‘Sanitization’ in this context means the removal of malicious data from user names in order to avoid security issues.

Now, things get really interesting (code snippet 5, lines 14-19). When (1) an unsanitized (legacy) file exists, (2) the legacy configuration file is not equal to the sanitized configuration file, and (3) the sanitized configuration file does not exist on disk, Jenkins creates a new sanitized directory for the user. The unsanitized legacy config file is MOVED into the newly created directory, line 18 in snippet 5 creates a new directory and line 19 moves the unsanitized configuration file to the new directory.

Can we exploit this logic? Knowing that we have a config.xml file in the JENKINS_HOME directory that contains the Jenkins main security switch, we set out to check the code and see if we can somehow convince Jenkins to remove the main configuration file from disk.

Let’s first chart the code (see flow-chart below) and then analyze it:

We check the flow chart with the username “..”  To do that, we prepare a base64 string in the format “username:password” that contains username of .. (dot dot) and any password:

“..:ANYPASSWORD” = Li46QU5ZUEFTU1dPUkQ=”

Next, we send it using CURL:

Curl JenkinsURL –H “Authorization: Basic Li46QU5ZUEFTU1dPUkQ=”

Jenkins code will compute the unsanitized configFile to be:

/JENKINS_HOME/users/../config.xml

Which is equivalent to –

/JENKINS_HOME/config.xml

We know this file exists, as this is the Jenkins main configuration file we discussed previously, so we’ve successfully passed test #1.

The sanitized Jenkins (post SECURITY-499) does not allow “..” in the username due to path traversal issues, so Jenkins replaces each dot with the string “$002e.” Therefore, the sanitized configuration file is computed as:

/JENKINS_HOME/users/$002e$002e/config.xml

Obviously, the sanitized configuration file is not equal to the unsanitized one. We just passed test #2.

The 3rd and last test checks if the sanitized file exists – I don’t suppose some Jenkins installation will have a user by the name of “$002e$002e”^[ii]… Well, we just passed test #3.

At this point, the Jenkins main configuration file – config.xml  – that contains statements about the main security switch has been removed from the JENKINS_HOME directory. Note that Jenkins did not require us to authenticate to get to this point. This means that any unauthenticated person with access to the Jenkins master can delete the Jenkins main configuration file and eventually gain admin access to Jenkins resources, secrets and infrastructure.

Code snippet 1 shows that Jenkins is actually authenticating the user, i.e., comparing API tokens, only at line #47. Since the username we supplied does not exist in Jenkins users’ database, the Curl GET HTTP command will always return an error since the authentication was unsuccessful.

At this point, the Jenkins main configuration file has been removed from the Jenkins home directory by an unauthenticated user.

This issue has been fixed in Jenkins version 2.121.1 LTS (2.132 weekly).

Main Config Removed; What’s Next?

There is another somewhat important issue we’ve yet to address. The Jenkins master software is working off the Java virtual machine cache memory. That means all files are actually read to the Java cache when the Jenkins server starts running, and the fact that one file is removed, as important as it may be, does not immediately change the Jenkins security status. This is important because Jenkins must be restarted for the changes in config.xml to take effect.

An attacker can now wait silently until the Jenkins server is restarted (e.g., when a new Jenkins version is released and updated), at which time the main security of Jenkins will revert to the legacy defaults of granting administrator access to anonymous users.

However, an attacker doesn’t really need to wait that long…

CVE-2018-1999043: Crashing the Java Virtual Machine

If you recall, the User.java module function User.getById() in code snippet 2 was called with username and another intriguing parameter (set to ‘true’). That parameter we later found was called ‘create,’ and it seemed like we could create new users in Jenkins Java cache even though we were not authenticated to Jenkins (the actual authentication check is performed a few lines afterwards).

Each user takes a certain amount of space in the Java virtual machine memory, and if the users have long names, the amount of space may be increased. By using the same Curl command with very long usernames, we were able to crash the Java virtual machine due to low memory, and force the Jenkins admin to restart the Jenkins server upon our will.

This second vulnerability has been fixed in version 2.121.3 LTS (2.137 weekly).

Final Thoughts: Exploiting Systematically

We’ve seen how the Jenkins main config.xml file, a file containing the most important Jenkins security configurations, was moved from the JENKINS_HOME to a new directory, effectively removing it, and how we were able to crash the Java virtual machine by filling up its memory with new users so when Jenkins is restarted it will be running in “Security Disabled” mode. This mode requires no authentication, and considers any actor with access to the Jenkins master an administrator.

At this point, a sophisticated attacker can further exploit this situation to become stealthier. By reading the config.xml file from the JENKINS_HOME/users/$002e$002e/ directory and making necessary changes to the files on the Jenkins filesystem, such as adding a new admin user, or reading and decrypting the admin’s API token, our attacker can copy the previously moved config.xml file back to its original place.

By doing that quickly enough – and at the right time – the Jenkins master would be returned to its original security configurations and nobody would notice the fact that the Jenkins master reverted to its legacy default (i.e., security disabled) for a short while. This would be extremely dangerous for an organization as it would make the entire attack on the network virtually undetectable.

Conclusion

Both of these vulnerabilities have been addressed and fixed. It is worth noting, that as part of the fix, the Jenkins Project programmers overhauled the internal user storage in a visible way (numeric directory suffixes and the use of users.xml file).

However, constant vigilance and strong cyber security hygiene are critical in driving down risk in dynamic DevOps environments. Organizations leveraging Jenkins and other important DevOps methodologies and tools to deliver products and services to market should consider the following best practices:

1. Make the Jenkins master your fortress. Make sure all administrative access to the Jenkins master is isolated, monitored and controlled using CyberArk’s Privileged Session Manager (PSM).
2. Monitor incoming network communications to the Jenkins master. Make sure requests that may be masking brute force attacks on Jenkins or the Java virtual machine, such as trying to login using wrong passwords or user names, are stopped and investigated.
3. Always update your Jenkins master and agents software with the latest version available on the Jenkins website. Updating and patching with latest software will make sure attackers cannot leverage published exploits against the organization’s DevOps infrastructure.

 

Disclosure Timeline

• May 24, 2018: Two issues were reported to CloudBees Jenkins security group
• May 25, 2018: Issues were designated as SECURITY-897 and SECURITY-672
• July 18, 2018: SECURITY-897 was resolved and a security advisory was released; Issue was assigned
  CVE2018-1999001
• August 15, 2018: SECURITY-672 was resolved and a security advisory was released; Issue was assigned
  CVE2018-1999043

 

 

 

APPENDIX A

Example Jenkins main config.xml (main security switch at line 11):

<?xml version='1.1' encoding='UTF-8'?>
<hudson>
<disabledAdministrativeMonitors/>
<version>2.106</version>
<installState class="jenkins.install.InstallState$6">
<isSetupComplete>true</isSetupComplete>
<name>DOWNGRADE</name>
</installState>
<numExecutors>0</numExecutors>
<mode>NORMAL</mode>
<useSecurity>true</useSecurity>
<authorizationStrategy class="hudson.security.GlobalMatrixAuthorizationStrategy">
<permission>hudson.model.Computer.Connect:agentconnect</permission>
<permission>hudson.model.Hudson.Administer:admin</permission>
<permission>hudson.model.Hudson.Administer:attacker</permission>
<permission>hudson.model.Hudson.Read:agentconnect</permission>
<permission>hudson.model.Hudson.Read:jobconfig</permission>
<permission>hudson.model.Item.Configure:jobconfig</permission>
<permission>hudson.model.Item.Read:readonly</permission>
<permission>hudson.model.View.Read:readonly</permission>
</authorizationStrategy>
<securityRealm class="hudson.security.HudsonPrivateSecurityRealm">
<disableSignup>true</disableSignup>
<enableCaptcha>false</enableCaptcha>
</securityRealm>
<disableRememberMe>false</disableRememberMe>
<projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
<workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
<buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
<markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
<jdks/>
<viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
<myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
<quietPeriod>5</quietPeriod>
<scmCheckoutRetryCount>0</scmCheckoutRetryCount>
<views>
<hudson.model.AllView>
<owner class="hudson" reference="../../.."/>
<name>all</name>
<filterExecutors>false</filterExecutors>
<filterQueue>false</filterQueue>
<properties class="hudson.model.View$PropertyList"/>
</hudson.model.AllView>
</views>
<primaryView>all</primaryView>
<slaveAgentPort>0</slaveAgentPort>
<label></label>
<crumbIssuer class="hudson.security.csrf.DefaultCrumbIssuer">
<excludeClientIPFromCrumb>false</excludeClientIPFromCrumb>
</crumbIssuer>
<nodeProperties/>
<globalNodeProperties/>
</hudson>

 

APPENDIX B

Example user’s config.xml:

Jenkins main configuration file config.xml

<?xml version='1.1' encoding='UTF-8'?>
<user>
<fullName>Read Only User</fullName>
<properties>
<jenkins.security.ApiTokenProperty>
<apiToken>{AQAAABAAAAAwPze9D9ahyB+RUYTTOijSYJZoyXaZNWa47ZTfuo1jDJ6LM3IvTSMNNWdMPBg3PYFIJmMIripwGdaSxvzfJGH04Q==}</apiToken>
</jenkins.security.ApiTokenProperty>
<com.cloudbees.plugins.credentials.UserCredentialsProvider_-UserCredentialsProperty plugin="[email protected]">
<domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash"/>
</com.cloudbees.plugins.credentials.UserCredentialsProvider_-UserCredentialsProperty>
<hudson.plugins.emailext.watching.EmailExtWatchAction_-UserProperty plugin="[email protected]">
<triggers/>
</hudson.plugins.emailext.watching.EmailExtWatchAction_-UserProperty>
<hudson.model.MyViewsProperty>
<views>
<hudson.model.AllView>
<owner class="hudson.model.MyViewsProperty" reference="../../.."/>
<name>all</name>
<filterExecutors>false</filterExecutors>
<filterQueue>false</filterQueue>
<properties class="hudson.model.View$PropertyList"/>
</hudson.model.AllView>
</views>
</hudson.model.MyViewsProperty>
<org.jenkinsci.plugins.displayurlapi.user.PreferredProviderUserProperty plugin="[email protected]">
<providerId>default</providerId>
</org.jenkinsci.plugins.displayurlapi.user.PreferredProviderUserProperty>
<hudson.model.PaneStatusProperties>
<collapsed/>
</hudson.model.PaneStatusProperties>
<hudson.search.UserSearchProperty>
<insensitiveSearch>true</insensitiveSearch>
</hudson.search.UserSearchProperty>
<hudson.security.HudsonPrivateSecurityRealm_-Details>
<passwordHash>#jbcrypt:$2a$10$Ooy6fm9jjrNofQQH914tVe8Fat4g6dwbd3v1/g3ZwtSGWyzo4txha</passwordHash>
</hudson.security.HudsonPrivateSecurityRealm_-Details>
<hudson.tasks.Mailer_-UserProperty plugin="[email protected]">
<emailAddress>[email protected]</emailAddress>
</hudson.tasks.Mailer_-UserProperty>
<jenkins.security.LastGrantedAuthoritiesProperty>
<roles>
<string>authenticated</string>
</roles>
<timestamp>1521644405116</timestamp>
</jenkins.security.LastGrantedAuthoritiesProperty>
</properties>

 

[i] The Jenkins API Token mechanism has changed in the Jenkins 2.129 version, however legacy API tokens remain on the Jenkins users’ database.

[ii] Even if a user by that name does, for some strange reason, exist, a sophisticated attacker would be able to launch another attack, now with a username: “../users/..”. The unsanitized Config File is still at /JENKINS_HOME/config.xml, while the sanitized Config File is now “/JENKINS_HOME/ ..$002fusers$002f../config.xml”