Editor’s Note: This post was co-authored by John Walsh, senior product marketing manager, DevOps security, CyberArk and Tim Johnson, director, product marketing, CloudBees, and the original version was published on CloudBees.com. The post has been updated to include recent industry guidance.
By the time you hear about a software supply chain attack, it will have already become a massive problem and public relations nightmare. However, these attacks don’t start out that way. In fact, they almost always begin with a tiny gap, like an exposed Git credential or a misconfigured CI/CD tool, earlier in the supply chain. And that’s exactly what cyber adversaries are looking for.
Attackers infected more than 18,000 conscientious SolarWinds customers by injecting malicious code into an otherwise legitimate software update. The April 2021 Codecov breach was traced back to a process error that enabled the bad actors to extract credentials and modify scripts. In the massive Kaseya ransomware attack, trusted software was compromised to reach into the company’s global customer base. And things are only expected to get worse. According to Gartner®, “By 2025, 45% of organizations worldwide will have experienced attacks on their software supply chains, a three-fold increase from 2021.”[1]
These recent headline-grabbing attacks have resulted in significant financial and reputational damage and prompted many organizations to re-examine their own development environments and delivery practices to identify vulnerabilities. And they’re getting to the heart of the issue: unprotected secrets (credentials, SQL/LDAP passwords, SSH keys and API tokens) that provide privileged access to the organization’s most valuable data and resources. A single, unprotected access point is all it takes for an attack to rip through an entire IT environment and move downstream to customers and partners.
The “Secret” to Software Delivery Automation
Jenkins — plus all the other tools, platforms and containers used by developers to automate software delivery — must be able to interact with numerous systems and applications throughout the DevOps environment to accomplish its role as the “butler.” To do this, it requires access to the powerful secrets used to source control and deploy artifacts.
If not configured properly and secured, these secrets become easy — and valuable — targets for attackers. CI/CD pipeline and other DevOps tools are “Tier Zero” assets: access them and you get access to more privileged credentials. Using compromised privileged credentials, attackers can take confidential data, inject malware into the codebase, make significant changes to an application’s functionality or steal code and valuable IP from repositories.
Gartner® has issued the following guidance: “We recommend privileged access management (PAM) tools to monitor and protect privileged service accounts that run builds and automation jobs. In the SolarWinds attack, attackers modified trusted domains and abused privileged roles. We see that as a confirmation that privileged accounts are a primary target. Mitigating this risk often requires privileged accounts to be managed by a PAM.”[2]
It’s clear that protecting the privileged access embodied in secrets and credentials across the DevOps pipeline is key to protecting software integrity and securing the supply chain. But who ultimately bears this responsibility?
Who Actually Owns Software Supply Chain Risk?
The shift to modern infrastructure and DevOps has created complexity and differences in how security needs are met. Regulations and industry standards such as Sarbanes-Oxley (SOX) require organizations to take ownership of supply chain risk, implement specific DevOps security controls and provide greater transparency to minimize risk. Meanwhile, customer demands for security assurance, coupled with modern digital experiences, are increasing rapidly.
Security teams are charged with securing digital value chains that protect customer data, starting with who, or what, is allowed to access what within the CI/CD pipeline and production environments, when and for how long. The challenge is finding a way to achieve that goal without slowing down developers.
Developers face their own security dilemma: with a laser-focus on bringing innovations to market fast, the DevOps culture emphasizes high velocity, intensive sharing of code, ad-hoc tooling and full-on automation. In this culture, low-security shortcuts often flourish, such as putting unprotected secrets in JenkinsFiles to save time or reusing open-source code from the internet without sufficient scrutiny. The last thing they want is someone else preventing them from getting their work done, and they don’t really want to have to “own” security, yet they do not want to be held responsible for a costly breach-related outage.
As a result, some types of DevOps security, vulnerability and code scanning are being integrated into DevOps processes. Yet in many organizations, secrets management remains a basic functionality that’s fragmented across individual DevOps tools, making them difficult for developers (who don’t have time to become security experts) to manage and secure. When security issues are discovered — sometimes right before code is scheduled to go into production — developers feel the pain in terms of last-minute code changes and delayed releases.
The evolving development landscape requires security and development to work together to secure secrets across the CI/CD environment to reduce risk and minimize damage from the next supply chain threat. A centralized approach to secrets management makes it easier for developers to maintain their existing workflows and keep moving fast.
Follow These Three Steps to Protect the Pipeline
Most organizations already have established requirements for securing credentials with secrets management in traditional IT systems, such as automatically rotating, centrally storing and continuously monitoring credentials and secrets. However, to protect your software supply chain and secure the CI/CD pipeline, you also need to implement these secrets management best practices throughout the Jenkins pipeline, within DevOps tools and admin consoles, on developer workstations and in applications and scripts. The most effective strategy follows these steps:
- Enforce the “least privilege principle” to limit secrets exposure. Apply role-based access control (RBAC) so the developer, tool, application, container, script or automation process only has access to the credentials it needs. This, for example, will help prevent untrusted parties from configuring jobs and help ensure valuable credentials, such as deployment tokens, are segregated by appropriate access restrictions or deployed from a second secure Jenkins server.
- Remove all hard-coded secrets in code, DevOps tools, configuration files and scripts. It’s also important to never use default passwords. For example, some tools establish a developer default user to create projects; other tool consoles can be accessed using http or with the default password.
- Authenticate access to secrets. This extra layer of protection requires attackers to take extra steps beyond stealing a single secret known as “secret zero.”
A centralized secrets management platform will make these best practices far easier to implement and manage by giving your organization a complete and continuous picture of “who has access to what.”
Orchestrating Trust in the DevOps Pipeline: Where to Go from Here?
While the notion of “shift left” to build safer and more efficient CI/CD practices and processes is widely understood, putting it into practice can be difficult. It’s important for the security team to take the lead in integrating security into the DevOps processes before poor practices become entrenched. And instead of viewing security as the “release prevention department,” developers should make it a practice to bring security in at the planning stage — and keep security tightly integrated in the CI/CD process from end to end.
Here are four practical recommendations for making this collaborative approach a reality:
1. Security teams must get up to speed on DevOps tools and techniques to credibly communicate risk and best practices to their development counterparts. And by using agile and DevOps methods within their own security practices, such as automating security tasks and delivering security capabilities in smaller, more frequent increments, they can gain a deeper understanding of DevOps.
2. Implement a security team-owned solution that provides critical capabilities, such as secrets management, in machine-consumable ways that can be integrated into automated processes, like security policy as code. This makes it easier for developers to get the access they need to do their jobs, while doing the “right” thing from a security perspective.
3. Teams should jointly consider how best to deploy security resources into existing or new organizational models and structures. For example, establishing centers of excellence, community leaders and security champions and embedding security team members inside development teams. “Results from Gartner’s 2021 Enabling Cloud-Native DevSecOps survey showed that creating a cloud center of excellence (COE) was considered as the single most effective measure to improve security, with 31% of respondents naming it as their first choice.”[3]
4. Offer developer training on specific attacker tactics and show how sample code modules could expose secrets and provide examples as user stories. For example, “As an attacker, I would scan the organization’s code repositories looking for secrets.” Take the team through a penetration testing exercise or engage a Red Team to demonstrate how an attacker could compromise a CI/CD pipeline.
With the right tools and strategy in place, security teams can more effectively partner with developers to establish more agile, secure and productive dev environments downstream in your software supply chains.
[1] Gartner, “How Software Engineering Leaders Can Mitigate Software Supply Chain Security Risks,” 15 July 2021. Manjunath Bhat, Dale Gardner, Mark Horvath
[2] Gartner, “How Software Engineering Leaders Can Mitigate Software Supply Chain Security Risks,” 15 July 2021. Manjunath Bhat, Dale Gardner, Mark Horvath
[3] Gartner, “Survey Analysis; Enabling Cloud-Native DevSecOps,” 13 September, 2021. Dionisio Zumerle
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
