Editor’s Note: Part 2 of a 5 Part series on securing DevOps environments based on insights from Global 1000 Chief Information Security Officers in the CISO View report.
- Transform the security team into DevOps partners
- Prioritize securing DevOps tools and infrastructure
- Establish enterprise requirements for securing secrets and credentials
- Adapt processes for application testing
- Evaluate the results
The power of the DevOps tool chain and their associated privileged accounts and secrets make them a top priority for security teams, along with protecting the development and production environments themselves. In The CISO View – Protecting Privileged Access in DevOps and Cloud Environments, published earlier this year, our panel of expert CISOs weighed in on their experiences securing key tools and infrastructure to help their organizations achieve successful DevOps outcomes and progress on their digital transformation journeys.
Here are five of their recommendations for how CISOs and their security teams should prioritize protection of DevOps tools and processes:
1. Set and enforce policies for the selection and configuration of tools. Since you can’t protect what you don’t know about, first take a full inventory of the DevOps tools being used by the development teams. While challenging, this is especially important for open source tools, since 58 percent of today’s businesses are utilizing open source heavily to flatten learning curves and speed release cycles. Once these tools are accounted for, conduct a thorough evaluation to identify any existing security deficiencies and address them promptly. For example, make sure tools are not being used in an unsecure default configuration and that they are kept up to date.
When it comes to evaluating new tools, security teams should find a way to get a seat at the table by collaborating with the group responsible for tool selection and configuration (e.g., a COE or Engineering Council) or working closely with Procurement to select the best tools for the organization and establish enterprise security standards from the start.
2. Control access to DevOps tools. Since attackers may only need to exploit one vulnerability to carry out their mission, it’s important to address security requirements and potential vulnerabilities holistically. This starts by securing DevOps and cloud management tools’ secrets and credentials in an encrypted vault protected with multi-factor authentication (MFA). Provide “just in time” access so users can gain high-level access only when it’s needed to perform certain tasks – and ensure that this temporary usage is closely monitored. Additionally, limit access to high-risk commands within DevOps tools. For instance, Docker users often run a Docker container with the —privileged flag, which gives the container direct access to host elements. Ensure users are not able run containers with this flag – and if it’s a “must,” severely limit user access and monitor and record all activities with the –privileged flag.
Follow other cyber hygiene best practices, such as setting up access controls that segregate DevOps pipelines so attackers cannot gain access to one and then move to another; ensuring that credentials and secrets are not shared between DevOps tool accounts and Windows sysadmin accounts and removing all unnecessary accounts with access to DevOps tools (i.e., accounts for developers who have changed roles, don’t actually require access to tools or have left the company).
3. Reduce the concentration of privilege. Ensure least privilege by limiting each user’s level of access to DevOps tools to the minimum necessary for their role. But, don’t stop there. Configure DevOps tools to require dual authorization for certain critical functions. For instance, you can require that before a change to a Puppet manifest file goes live, a second person must review and approve the change. Additionally, implement separation of duties for build automation tools such as Jenkins, which often end up over-privileged and able to perform all duties without restriction – from building and testing to packaging. In the case of Jenkins, separate duties by implementing multiple Jenkins nodes, each dedicated to one function (build or test or package) for each application. Each node will have a unique identity and a limited set of privileges, which minimizes the impact of a potential compromise.
4. Ensure code repositories do not expose secrets. Develop risk-based policies for developers around the use of code repositories. Remember that, beyond credentials, code may contain details about the organization’s internal network that could be useful to attackers. If you can do so without adversely affecting workflow, use an on-premises rather than a cloud-based code repository. Scan the environment to make sure that any on-premises code repositories are inaccessible from outside the network. If cloud-based repositories are used, ensure they are configured to be private. Finally, before checking code into any repository, implement automated scanning to ensure code does not contain secrets.
5. Protect and monitor infrastructure. Cyber attackers seek the path of least resistance. Often, a well-crafted phishing email will do the trick, so make sure that all workstations and servers undergo regular patching, vulnerability scanning and security monitoring. Additionally, monitor your cloud infrastructure for signs of unusual credential usage or configuration changes (such as making private data stores public). Ensure VM and container images used in development and production environments come from a sanctioned source and are kept up to date. To ensure security remains “baked in” to countless rounds of automatic rebuilds, security teams should work with their DevOps counterparts to automate the configuration of VMs and containers so that, when a new machine or container is spun up, it is automatically configured securely and given appropriate controls – without requiring human involvement.
Want to dig in deeper? Download the full CISO View report, watch a brief highlights video, tune-in to our related webinar or check out our post on aligning security and DevOps teams. And watch this space for our next installment covering tips for establishing enterprise requirements for securing DevOps secrets and credentials.
