Red vs. Blue: Best Practices for Jenkins Credentials Management
Over the past year, CyberArk Labs has conducted extensive research on Jenkins – an open source automation server used to accelerate the software delivery process. With more than 223,000 active installations worldwide, Jenkins is considered the de facto standard in open source continuous integration tools, effectively acting as the DevOps engine, or “butler,” and addressing everything from source code management to delivering code to production. Because of this, it’s no surprise that Jenkins needs unconstrained access to privileged credentials – or secrets – to do its job. These secrets are prime targets for cyber attackers. If exploited, secrets may allow attackers to take full control of an organization’s IT infrastructure, disable security controls, steal confidential information, commit financial fraud and disrupt operations.
Our goal with this Jenkins research was to educate organizations on potential security risks around unmanaged secrets and misconfigured environments, and share best practices for protecting privileged access while maintaining DevOps velocity. In our recent webinar, Red vs. Blue: Best Practices for Jenkins Credentials Management, we showcased research highlights and demonstrated several common attack techniques using Red and Blue Team tactics, including the ones below. For an in-depth look, tune in to the on-demand webinar.
External Attackers
- Red Team: Last February, reports emerged that attackers had successfully exploited an unpatched Jenkins vulnerability, stealthily mined Monero coins for about 18 months, and ultimately made $3 million in one of the biggest malicious cryptocurrency mining operations of all time.
- Blue Team: To protect against outsider threats, such as crypto mining attacks, organizations must treat their Jenkins master as a fortress –constantly protecting, monitoring and auditing it. Further, staying on top of regular patches for Jenkins software and plugins is a must. While this cybersecurity hygiene practice should be table stakes, thousands of Jenkins installations have not yet been patched, meaning the software remains vulnerable.
Malicious Insiders: The Jenkins Admin
- Red Team: Jenkins secrets are protected by encryption and may not be exposed. Same goes for Jenkins API tokens and other credentials stored in Jenkins. But can Jenkins keep its secrets from its admin? Unfortunately, the answer is no. Red Teamers will quickly discover that Jenkins admins have access to all of this critical information. Using the Script Console, admins can run code on any machine controlled by Jenkins – including the Jenkins master – to decrypt secrets and display them in clear text.
- Blue Team: The Jenkins admin is a very powerful user in Jenkins. To help mitigate the risk of an insider attack, organizations should extract secrets from Jenkins to the furthest extent possible, placing them in a centralized vault where they can be secured, rotated and controlled. To keep any one person from having all of the secrets, use multiple access tokens stored in different repositories. In extreme cases, use two Jenkins infrastructures, for example, one server for dev and another one for production. You can learn more about this here.
Malicious Insiders: Malicious Pipeline Config User
- Red Team: The Jenkins pipeline is a suite of Jenkins plugins supporting the implementation of continuous delivery automation in Jenkins. It is a computer-readable expression of all software build stages: build, test and deploy. If a malicious insider has control over a pipeline – either via the Jenkins web user interface or by controlling the Jenkins file – they can request secrets using the functionality provided by the Jenkins credentials binding plugin and receive them in clear text.
- Blue Team: One way to mitigate risk is to limit the scope of credentials for specific pipeline jobs. Set up the Jenkins folders plugin so that pipelines in one folder cannot access the secrets in another folder.
Interested in learning more? You can check out our full Jenkins research series on the CyberArk Threat Research Blog (or in the individual links below). There you’ll review the Jenkins vulnerabilities discovered by CyberArk Labs and subsequently addressed by CloudBees, examine weaknesses of misconfigured environments and explore best practices for configuring privileged credentials and securing web start agents and plugins:
- Configuring and Securing Credentials in Jenkins
- Securing Jenkins Java Web Start Agents
- Jenkins Plugins – Aladdin’s Lamp and the Sultan of Threats
- Tripping the Jenkins Main Security Circuit-Breaker: An Insider Look at Two Jenkins Security Vulnerabilities
- Securing Jenkins: Active Directory and LDAP Services in a Jenkins Environment
