As software supply chain attacks surge in frequency and scale, it’s become apparent that cyber criminals are looking for stealthy ways to make malicious changes or inject malware into software — before it’s deployed — often by compromising developers’ credentials or the administrative privileges of a development tool. Once a powerful password or key is stolen, it’s easy to tweak a software library or modify a few image registry settings to inject malware or compromise pipelines in a way that is often exceedingly difficult to detect.
More troubling still, many users — including developers — are over-privileged users who have access to broad swaths of an IT environment — in other words, they’re highly attractive targets. And in many cases, they’re shouldering more responsibility for managing these environments. According to Brandon Traffanstedt, senior director in CyberArk’s global technology office, cyber criminals have taken note that some of the easiest targets are developers with access rights that allow them to configure a wide range of systems to provision applications — but without the greatest appreciation for security. “Cyber criminals know developers are given as much access as they need,” he says. “We need to secure their identities — and their access — without constraining the amazing work they’re doing.”
The Trouble with Securing Identities Across a DevOps Pipeline
Cyber criminals target developers for two primary reasons. The first is to exploit the elevated privileges they require to access a wide range of applications and platforms strewn across an extended enterprise. The second is to uncover as many application secrets as possible that might reside within code, in environmental variables or anywhere outside of a secure secret store or vault. A recent example of this is the April 2021 Codecov breach, during which attackers gained access to a credential based on an error in the company’s Docker image creation process. They used this credential to modify Codecov’s Bash Uploader script and used that to harvest exposed credentials and secrets — too often easy and unprotected targets — from numerous customer CI/CD environments.
Some organizations have a large number of users who are privileged users “by default,” with elevated access rights whether they need them or not. Developers, however, have privileged access to more crucial platforms than almost anyone else and are relied upon to ensure that modern businesses remain competitive. As responsibility for managing applications on an end-to-end basis becomes the norm, the more control over IT environments a developer is likely to have. “There’s been an organic shift left with developers and pipelines making up the lifeblood of successful businesses — from pizza delivery to insurance,” says Traffanstedt.
The challenge is finding a way to ensure applications are built and deployed securely without slowing down the application development process, notes Traffanstedt. “Security needs to be part of the overall quality assurance process already in place to provide a foundation that developers can use to make their lives easier,” he says. Otherwise, the notion of security as a roadblock to agile development will persist.
Fortunately, developers are making progress. As organizations increasingly embrace DevSecOps best practices, there’s a greater awareness of how vulnerable software supply chains really are and what can be done to incorporate transparent security. The simplest phishing attack can result in losing a set of credentials that provides cyber criminals with the keys to the entire development environment.
Unfortunately, organizations can’t wait for developers to master all the nuances of cybersecurity on their own or expect developers to keep security common practices top-of-mind. Security teams have been given a clear mandate to protect the software supply chain, starting with who (or what) is allowed to access what, when and for how long. The challenge is finding a way to achieve that goal while having the least possible impact on developer productivity. After all, as organizations become more dependent on software to drive digital processes, the rate at which software is built, deployed and updated directly correlates to corporate financial performance.
Risk Management in the Age of Shift Left
Cybersecurity has always been about limiting risk. Securing software supply chains in that regard is no different than in any other IT environment — but it has to be done fast. Here are steps to bolster DevOps security while maintaining developer velocity, focusing on addressing the most risk most efficiently.
The first order of security business is always going to be securing the workstations and tools developers use regularly.
1. Secure Developer Workstations and Tools
Sometimes, developers need to elevate certain tools they use to write and commit code. Many organizations solve this by providing full local administrative rights. This should never be the default, and the principle of least privilege should be consistently enforced. “The goal is to reduce, if not completely eradicate, the number of local administrators that you have while providing just the right level of elevation based on policy,” says Traffanstedt.
And since many credential theft attacks can be executed without local admin rights and from a single developer’s workstation, it’s also important to protect against credential theft from browsers where application secrets might be stored to access repositories or tools with command line interfaces (CLI) — and where those same secrets can be stored right in configuration files. The key to this approach is leveraging solutions that do not require archaic allow/deny listing policies but instead provide proactive and predictive controls
2. Secure the Automation
Another threat vector that needs to be secured is the automation frameworks developers now routinely employ to configure infrastructure as code, along with the ever-growing list of developer tools used throughout the pipeline. It’s just as critically important to both manage and rotate the secrets used by these tools and platforms, as well as to ensure that applications and processes authenticate securely to retrieve those managed secrets. Otherwise, you’ve done a significant amount of work for very little value.
Many of the tools and the services offered by cloud vendors and container platforms have some built-in secrets management capabilities. However, each tool vendor takes a different approach to security, has a varying level of maturity and experience and uses different interfaces — effectively establishing its own “island of security.” Using these native secrets management capabilities is certainly better than no security at all, says Traffanstedt. “But at some point, when it comes to simplifying compliance and configuration in addition to supporting code portability, having one centralized store of record can allow you to identify gaps in policy and push changes much faster than liaising with disparate tools and providers.”
3. Determine Who Really Needs Access to What
The next step is to determine who precisely on any development team can commit source code and elevate binaries as part of a DevOps workflow. Not every member of the DevOps team needs to have full administrative rights to every system and tool that make up the pipeline.
Just-in-time privileged access controls can give specific developers the flexibility to access what they need, when they need it, based on role. When privileges must be elevated, cybersecurity teams should monitor the resources that are being accessed — and by whom. In many cases, it becomes apparent that many developers only need local admin rights to write code. Only a handful of members of the development team are often responsible for managing code commits and elevating code into production environments. “Organizations need to make sure they take a least privilege approach to managing access across the DevOps pipeline,” says Traffanstedt.
4. Secure Tier 0 Access to Resources and VMs
In a similar vein, developers should not have direct access to virtual machines in production environments. If they require access, they should broker it through session management or a secure web access gateway to minimize risks. As with all security controls, exceptions will occur and those should be subject to a heightened level of approval and scrutiny. Under no circumstances should exceptions made to policies be allowed to become permanent. If a policy activation is permanent or very loosely restricted, then it is essentially the equivalent of standing access. There needs to be a strict process for approving the assignment and activation of policies across the entire DevOps workflow. Keep in mind that in a perfect world, human hands would not touch infrastructure directly in day-to day-operations — instead, they would leverage elements of automation to make changes.
Pay extra attention to cloud platforms that provide users with several levels of access options, many of which amount to yet another instance where a developer is overprivileged. The same least privilege rules and session isolation and monitoring processes applied in an on-premises environment need to be extended to the cloud, planning especially for the continued evaluation of how human and non-human entities are permissioned as well as for the heightened scale that the cloud can provide.
The Zero Trust Future-Present for Supply Chain Security
Securing the development pipeline — the heart of the digital supply chain — is not a one-time event. Just like the development of modern applications, security needs to be a continuous process and tackled in risk-prioritized phases. Security and Identity and Access Management (IAM) teams must constantly reassess permissions, regardless of whether developers (and their applications) are employing identity management frameworks or not. Organizations should assume no identity — person or machine — is to be trusted as part of a larger Zero Trust approach to security.
At the same time, application secrets need to be closely guarded. A vault that both encrypts application secrets and safeguards how those secrets are accessed is the last line of defense for any software supply chain.
The challenge and opportunity for security professionals is to present themselves as allies in the effort to secure the software supply chain. While CI/CD pipelines leverage more automation than ever, the fact of the matter is that developers are people too — people who have a profoundly powerful impact on the success of our organizations and whose identities and processes deserve the same level of security and operational considerations as any executive’s.
