The recent CircleCI breach highlights the risk of storing secrets in places like private code repositories (GitHub), scripts, configuration files, files encrypted at rest, CI/CD pipeline code or anywhere where they cannot easily be rotated, audited, authenticated and secured.
CircleCI is a leading continuous integration and continuous delivery (CI/CD) platform with over a million users. After a recent security breach, CircleCI is advising customers to “[i]mmediately rotate any and all secrets stored in CircleCI” and to “review internal logs for their systems for any unauthorized access.” CircleCI recommends that their customers do not store any “long-lived” credentials in CircleCI.
In addition, we recommend that organizations affected by the breach check for suspicious activity on any integrated SaaS applications and cloud providers, which could have been impacted.
The details are still developing however, what’s more important is how organizations can proactively prevent of this type of attack going forward. For those affected by this breach, highly privileged access for machine identities is now exposed via their secrets to potential attackers, making it difficult to know what the attackers might have access to. Companies impacted need to immediately rotate their secrets, but any secrets that are long-lived or embedded in code will require extra development time to manually find and rotate them.
The Problem With Hard-coded Secrets Applications and Scripts
Secrets are credentials such as passwords, SSH keys, API keys or any private information that grants access to protected resources. These secrets grant non-human/machine identities highly privileged access to sensitive company resources. Controlling the secret allows you to assume the non-human identity.
The most recent CyberArk Identity Security Threat Landscape Report shows that machine/non-human identities outnumber human identities by a factor of 45x, and 68% of non-humans have access to sensitive data and assets, highlighting the importance of protecting secrets and the non-human identities they represent.
To protect non-human identities, organizations should not directly embed secrets in application code, scripts, configuration files or in any static location where unauthorized users can read them or where they cannot quickly be rotated when needed during a security incident. One of the things to keep in mind is that if anyone gains access (authorized or unauthorized) to the source code, they also have access to anything the secrets protect. With that access, they can simply impersonate the identity of the non-human user that secret represents and leverage this identity to escalate privileges and expand the impact of the initial attack.
CircleCI isn’t alone. There are several high-profile breaches related to non-human access in the news recently. Therefore, the focus should be on how to mitigate the most risk and limit the spread of future attacks. Let’s look at some key lessons from this attack and other recent breaches:
- Source code doesn’t stay secret: CircleCI isn’t alone. During the same week, Slack reported a “threat actor downloaded private code repositories” after a “limited number of Slack employee tokens were stolen and misused.” Thankfully, no “downloaded repositories contained customer data, means to access customer data.” Sadly, this isn’t the only high-profile leak of source code, but this proves that source code is a terrible place to keep secrets.
- Attackers target secrets to escalate privileges: The Uber breach reported on September 19, 2022 was spread throughout the enterprise because hard-coded secrets embedded in a PowerShell script allowed the attacker to gain high-level access and escalate privileges. This breach is yet another good example of why you shouldn’t put secrets in code.
- Development time is wasted rotating secrets: Developers everywhere are now busy searching their code and updating hundreds or thousands of secrets spread across their development and production environment. One developer on Reddit said, “there goes the rest of my week…My company deploy[s] several hundred websites through CircleCI, each with their own secrets…”
Why Developers Still Write Secrets into Source Code
Storing secrets in code is certainly not recommended, but many people sheepishly admit to seeing secrets in production code before, even if they personally would never do it. Some of the reasons for this are:
- Test code improperly moved to production: The code was only meant to be part of a test application, sandbox environment or the secret was meant to be temporary (“I will remove it later”).
- Lack of secure development training: The developer who wrote the code isn’t aware of the best practices for safely handling secrets in code.
- False security assumptions: The developer falsely assumes the secrets are protected by something else like the code repository (GitHub) access controls, base64 encoding, Kubernetes Secrets or encryption at rest only.
Attackers Are Gonna Attack, so Hunters Need to Hunt
Under constant cybersecurity threats, enterprises need to focus on what they can control to mitigate their risk and potential exposure. Because there is no well-defined IT perimeter, the threat landscape is now asymmetric, making a Zero Trust security mindset essential. Protecting non-human/machine identities and the secrets that represent them is critical to defending against the next attack. Any static credential or secret need to be removed from all scripts and source code. Security teams need to holistically manage access across the entire enterprise without silos or blind spots, with the ability to automatically rotate credentials as needed. A centralized secrets management system is the best option for authenticating, authorizing and auditing non-human access because it allows organizations to fully understand who has access to what and to automatically rotate or revoke access as needed.
Editor’s note, Jan. 17, 2022: After this blog post was published, CircleCI disclosed additional details on how the initial breach started – with malware on an employee laptop that allowed the attackers to keep the employee’s sessions open and active, without reauthorizing or running MFA precautions. This new information only underscores the importance of a holistic Identity Security strategy that stops breaches where they start – at the endpoints – by preventing credential, password, cookie and other types of tokens (especially post-MFA tokens) theft with a solution like CyberArk Endpoint Privilege Manager. A robust Identity Security solution also limits the blast radius of the initial phase of a breach by managing the secrets attackers frequently use to expand breaches deep into the heart of the enterprise.
