The Jenkins automation server is widely considered the de-facto standard in open source continuous integration tools. It offers a wealth of integration possibilities for user authentication and authorization, such as Unix user database, OpenID, Github authentication and SAML. However, enterprises most commonly use Active Directory Domain Services (AD DS) by Microsoft as their preferred users’ database.
This blog post highlights potential issues that users could run into with the Jenkins AD DS integration. These include disabled AD users retaining their access to the Jenkins master, issues with extending the Jenkins user session indefinitely and last, but not least, the dangers inherent in updating new Jenkins software.
This is the fifth in a series of Jenkins-focused research posts from CyberArk Labs exploring best practices for securing Jenkins. It follows the most recent post highlighting two security vulnerabilities discovered by CyberArk Labs and addressed by Jenkins. Jenkins is a valuable DevOps tool and it’s CyberArk Labs’ goal to educate organizations on security risks and offer recommended mitigation and best practices.
Active Directory Integration with Jenkins
AD DS solutions are so popular among enterprises that many would find it difficult to imagine life without it. At the end of 2015, only 5% of Fortune 1,000 companies didn’t use AD DS to organize their users and computers[i] — and the number of users has likely grown in recent years.
As enterprise usage of Jenkins expands, users may want to implement better security and restrictions around their Jenkins installation. While Jenkins offers a comprehensive users’ database, if an organization is already using AD then switching Jenkins to using AD is only natural.
Jenkins offers an easy integration with AD either by using the LDAP plugin, already installed by default with the Jenkins core package, or using the AD plugin, loaded from the Manage Plugins menu. In either case, Jenkins switches to using the AD database for user names, passwords and group memberships, but still remains dependent on its own user database for localized information, such as email addresses, the last granted authorities to a user and the user’s API token.
When an employee leaves a company, that status must immediately be reflected in the AD database to prevent unauthorized use of Jenkins.
It seems that there are no guidelines released by Microsoft on account management. However, some organizations tend to disable an account rather than delete it when a user leaves the organization. The account and data are kept for a period of time in disabled state, according to the data retention policy defined by the organization, and then deleted from the AD database. Disabling AD accounts may also take place during a long vacation or as a pre-layoff measure.
According to CyberArk Labs research, a user that has been disabled from AD will still be able to access the Jenkins master with their original permissions using their API token without alerting the Jenkins admin. In special cases, even deleting the account from the Jenkins console doesn’t seem to help, as users can continue accessing their account using the same method even after their accounts have been deleted[ii].
If the suspended employee had a chance to write down their API token, they could continue using the token for as long as they wish. For example, user ‘employee’ with API token afb5bcb1c6d6ff5fb10d7ec6a8bd4929 will be able to gain access, although their token has been disabled, by issuing the following CURL:
curl jenkinURL/manage -H “Authorization:Basic ZW1wbG95ZWU6YWZiNWJjYjFjNmQ2ZmY1ZmIxMGQ3ZWM2YThiZDQ5Mjk=”
The remote Jenkins API can be used to do just about anything on Jenkins and, therefore, although the Jenkins admin has the notion of security, a disabled user using their API token may still have unabridged access right under the radar .
AD Integration Mitigation
This issue was reported to CloudBees, which supports Jenkins, on May 23, 2018, and was resolved on Jan 16, 2019 with the release of Jenkins version 2.160 weekly (2.150.2 LTS).
This issue may be mitigated by either using the new “Terminate All Sessions” button in the user’s page, which allows the user or admin to invalidate user’s sessions or by invalidating the disabled user’s API token, if they are using the ‘new’ API token system (introduced in Jenkins version 2.129), or changing the disabled user’s API token for legacy API tokens. Both methods prevent the API token from being used in the future.
Image 1 shows the user’s configuration page:
Use the link in the red circle to invalidate the user’s ‘new’ API token or the Change API Token button (in blue) for legacy tokens.
Jenkins Session Mechanism Allows Access to a Deleted User
Whenever a user logs into the Jenkins server via the login webpage, a Jenkins session is created. This session accompanies the user while logged in. Jenkins attaches a session ID to each session opened.
A session ID is a unique number that a Web site’s server assigns a specific user for the duration of that user’s visit (session). Jenkins stores the session ID as a JSESSIONID cookie in the user’s browser.
After 30 minutes of inactivity[iii], the session is invalidated and the user is required to create a new session by logging into Jenkins again using their password.
During the login process, the user’s existence and group memberships are checked against the user’s database and, once their password is confirmed, Jenkins issues a JSESSIONID cookie to the user’s browser. During the time a session is in progress, no further checks are performed by Jenkins and the user is considered logged in.
By clicking the right mouse button and selecting Inspect on the Chrome pop up menu:
our Employee X can access cookie details belonging the Jenkins webpage:
Simply by copying the two marked fields, i.e. the name of the cookie (JSESSIONID.nnnnn) and the value (nodeNNNNNNN…NNN.nodeN) to another internet browser running on another machine, employee X can now run their Jenkins webpage from home[iv] as long as the original work session has not been invalidated. This is pretty easy to accomplish. Employee X only needs to shut down Chrome without pressing the ‘log out’ link (at the top-right of the Jenkins webpage) and the session is duplicated.
To keep the session open, employee X only needs to do something on the Jenkins webpage once every 30 minutes. A script refreshing the page would do just fine for that purpose.
Another way to keep a session open is by using the Jenkins option ‘Enable Auto Refresh:’
This option makes sure the browser will automatically refresh the Jenkins webpage every few seconds. So, by clicking this option, employee X can make sure that his session remains valid for as long as possible, practically circumventing the 30 minute inactivity timeout (or the 14 days Remember-Me option). Jenkins would consider the user still logged into the server and thus continue waiting for the open session to end indefinitely.
Keeping a session open has another important security implication: by using these tricks employee X maintains stealthy access to the Jenkins server – even after their account is deleted from the Jenkins users’ database[v]! A terminated employee, for example, can access the enterprise’s Jenkins server even after their account has been completely deleted from the Jenkins users’ database, as image 5 shows.
Session Management Mitigation
This issue was also reported to CloudBees on May 23 and was resolved on Jan 16, 2019 with Jenkins version 2.160 weekly (2.150.2 LTS).
The new Jenkins version introduces a new session revocation mechanism utilizing a new “Terminate All Sessions” button that allows the user or admin to invalidate the user’s sessions.
Another mitigation method is to invalidate the sessions of past employees immediately after deleting their records from the user’s database using the monitoring plugin.
Both methods make sure that all open sessions are terminated.
A Short Epilogue: Updating Jenkins Core Software
Jenkins code is being continually developed by the open source community. The long-term support package of Jenkins is released every 12 weeks and the weekly version is released, well, every week.
This means that the Jenkins code must be updated pretty regularly, if not to include new features, then to protect installations from security bugs. So, there must be a pretty structured automatic update procedure for the Jenkins code, right? Well, apparently not entirely.
Image 7 demonstrates one type of Jenkins installation: when a new version of Jenkins is available, the Jenkins admin gets a message on the Manage Jenkins page where the admin is encouraged to download a new version (in the blue circle) or upgrade automatically (the red circle).
In another installation of Jenkins we get a different picture:
Where did the ‘Upgrade Automatically’ option go? And what’s the difference between upgrading the software automatically and just downloading it?
When the Jenkins server sees that it can replace the current Jenkins.war file, the file holding the Jenkins core software, then the ‘Upgrade Automatically’ option is shown in the Manage Jenkins window. However, when Jenkins cannot replace the Jenkins.war file, e.g., where Jenkins does not have the required permissions, then the only option Jenkins offers is just the ‘download’ one.
So what’s the big deal, do the job yourself, you say… Well, not so fast.
It appears that due to some historical reasons, all Jenkins core software files (as opposed to Jenkins plugins) are always served via HTTP. When using the ‘Update Automatically’ option, the file’s integrity is confirmed internally before applying the update, even though the Jenkins file is still downloaded via HTTP. However, in Jenkins installations that do not support the ‘Update Automatically’ option on the Manage Jenkins page, Jenkins core executable files are still downloaded via the HTTP protocol, which is unencrypted and vulnerable to Man-in-the-Middle attacks.
An attacker on the local network can intercept the clear-text communications between the Jenkins master and the Jenkins download site and replace the code with their own, achieving code execution on the Jenkins master!
Mitigation
This was reported to CloudBees on May 23. CloudBees responded that this issue is not a vulnerability and that it is for the user (the Jenkins admin) to verify the integrity of the downloaded file.
We feel that the fact that downloading new software is accomplished via the vulnerable HTTP with no internal integrity confirmation should have been stated clearly, so the Jenkins admin is aware of that fact.
In this case, our mitigation is just that: confirm the integrity of the downloaded Jenkins.war file by using the following line:
$ jarsigner -verbose –certs –verify jenkins.war | grep ‘Signed by’ –A 50
In our installation, this is the return if all is okay with Jenkins.war file:
– Signed by “CN=Infradna Inc (Kohsuke Kawaguchi), O=Infradna Inc (Kohsuke Kawaguchi), STREET=4438 Hilton Ave, L=San Jose, ST=California, OID.2.5.4.17=95130, C=US”
Digest algorithm: SHA-256
Signature algorithm: SHA256withRSA, 2048-bit key
Timestamped by “CN=COMODO SHA-256 Time Stamping Signer, O=COMODO CA Limited, L=Salford, ST=Greater Manchester, C=GB” on Sun Nov 04 22:14:20 UTC 2018
Timestamp digest algorithm: SHA-256
Timestamp signature algorithm: SHA256withRSA, 2048-bit key
jar verified.
Conclusion
This blog post highlights three Jenkins security issues: AD disabled users integration into Jenkins, Jenkins session mechanism abuse and downloading Jenkins core software.
On top of the mitigation offered above, here are some general best practices for DevOps engineers in charge of Jenkins management:
Make sure that session leftovers and API tokens of terminated or suspended employees are completely removed from the Jenkins master.
- Since Jenkins is downloading new software via unencrypted channels that are vulnerable to man-in-the-middle attacks, care must be taken when using the manual ‘download’ button and verifying the war file is mandatory.
[i] https://www.itsmdaily.com/active-directos-infographics/
[ii] Wadeck Follonier of CloudBees informed us that this is true in a special case where the legacy tokens are generated at user creation and the API token used is the default one. Since Jenkins 2.129, a new API Tokens system has been introduced and should be used instead of the legacy token system.
[iii] This 30 minute timeout may be lengthened by using the Remember-Me option in the Jenkins login page.
[iv] Naturally, we are assuming that the Jenkins master is accessible from the user’s home.
[v] Wadeck Follonier of CloudBees informed us that a user’s account will only be recreated if the user has a valid remember-me cookie on the Jenkins database.
