AWS is one of the most successful cloud solutions available today. As a pioneer in the infrastructure-as-a-service (IaaS) scene, AWS has more than a million customers. Part of that success is because of the rich set of features and services that comes with it – including authentication services like STS.
STS is an essential service that runs across many user and application actions as well as in automated actions done by AWS itself in the background.
Besides the regular AWS customers that use its services, many attackers also use AWS – just not in the way you might think. Most of the time, they compromise a legitimate user for their non-legitimate actions, either by using it as a proxy to hide their trails or by stealing or corrupting valuable information of the compromised account.
Today, we’ll cover a few attack vectors impacting STS via temporary tokens. We will show how a temporary token can be converted to a permanent token, and how an attacker that compromises an EC2 instance can steal the temporary role token and act consistently, even outside of that VM instance. In addition, we will demonstrate ways to detect these attacks manually or through a free, open source tool we developed.
What is STS?
AWS STS – Security Token Service – is a web service that enables you to request temporary security credentials for AWS for operating time-limited and restricted privileges. It could also be used as just-in-time granting privileges mechanism.
One of the fascinating things about STS is that you are using it every day without even knowing it. Are you familiar with AssumeRole? For those of you who aren’t, AssumeRole is one of the most useful API calls in your AWS environment. AssumeRole returns a set of temporary security credentials that you can use to access AWS resources.
When you use “AssumeRole,” you use STS. When you run Lambda with an associated role, the Lambda uses STS. Even your EC2 machine uses STS for getting its temporary credentials for its metadata service.
And yet, could something so basic also be dangerous? Yes! Even though those tokens are time limited, they still might have access to critical or sensitive information and systems that would be enticing to a potential attacker or malicious actor.
Since many services use temporary tokens, an attacker has a big attack surface for stealing or getting their hands on one of those temporary tokens.
The Dark Side of the Temporary Tokens
STS is designed to generate temporary and limited-privilege credentials, but is that always what happens? Does STS always enforce those credentials to be temporary?
With that in mind, let’s talk about the following scenario.
Alice is an employee of a large finance company. Alice uploaded, by mistake, her secret access key of a privileged user to a public repository on GitHub.
Evil Bob is a malicious attacker who noticed this and “stole” the privileged credentials from the public repo.
Evil Bob is a clever attacker, and he knows that it is easy to track the activity of a specific access key ID, and he could leave trails of his actions.
Moreover, the security team could be alerted through seeing non-regular actions done by a specific access key – which would raise some questions.
Evil Bob wants to hide his trails or at least make it much harder to track them. Therefore, he creates a temporary token to a privileged role and uses that instead.
But wait, there is a problem! The IAM Role token is a temporary one, and it will last only for a minimum of 15 minutes or a maximum of 1 hour by default and could last up to 12 hours depending on the role configuration. So he creates another temporary token, but this time he uses his already created temporary token to generate another one. And he does it again, and again, and again. You can see the tokens’ creation flow in Figure 1.
Figure 1: The creation flow of endless temporary tokens
If we summarize what Evil Bob did, he managed to create a permanent token out of the temporary one, and now he can finally start to create the known and well branded “Evil Bob’s crypto mining farm.” Or he could do other things like deleting all the VMs in the environment, encrypting data (ransomware in a cloud environment), modifying information in databases, or just keeping persistence for future usage.
After a week, the information security team in Alice’s company finds out that something odd happened in their AWS environment. The team saw strange EC2 machines using many resources. They investigate it, and discover that a privileged access key had been leaked. The team immediately revokes the privileged access key to stop Evil Bob and deletes any sign of his farm.
Was that enough to get Evil Bob out of the AWS environment? Well, not quite.
Revoking the “AKIA” access key (the permanent key), which was leaked, isn’t enough to get Evil Bob out of the environment because of Evil Bob’s token refreshment method.
Moreover, in a case that the initial token leaked from a compromised EC2 machine – rather than a public code repository like in this scenario – there wasn’t a key to revoke as the first token was a temporary key.
Let’s look at this as a hierarchical “tree branches” structure. The first node is the first leaked AKIA token, then for every time Evil Bob requests a new temporary token, a new “branch” is created on the tree. Every branch is a standalone token that isn’t influenced by revoking its ancestors.
Figure 2: A tree structure of temporary token relations
Turn on the Flashlight
If an attacker uses this method, it could be very challenging for the victim to detect the attacker’s moves and actions.
So how can you detect this type of attack?
Every time “AssumeRole” is used, the session is recorded in the CoudTrail (the logging service of AWS), and you can look for all the creations of every token. But, if you look to this specific event (“AssumeRole”), you will probably see many records of this as it is a very basic and popular action. For example, a mid-sized AWS environment could have hundreds of AssumeRole actions every day.
Another issue is that you can only see the past 90 days – which might be a problem if you would like to recognize and analyze these attacks in retrospect. What would you do if the attack described above happened more than 90 days ago, and since then, the attacker just maintaining valid temporary tokens as persistence?
To solve these issues, you need to create a trail bucket in which to save the logs, and for making advance queries on the log data by using the “Amazon Athena Service.”
You can set a trail bucket via the CloudTrail dashboard. You will also need to create a table in Athena for the trail bucket (which you can also do from the CloudTrail dashboard).
Now, when everything is set and you are ready to go, we can better understand what we are looking for.
We are looking for temporary tokens (begins with ‘ASIA’) that were created by other temporary tokens. Event records have a property called ‘useridentity.’ This is an object that contains a lot of information about the token used for acting, including the access key ID that was used.
We can find those access keys by using the following SQL query:
Figure 3: SQL query for detection of a temporary token that created another temporary token
In the case the query returns results, we should omit any legitimate records and keep the suspicious records for further investigation.
How can I identify suspicious records?
There might be many variations of anomalous records. We will focus on those we mentioned in the blog post.
1. Temporary tokens, which are created by long term keys, create other temporary tokens. Long term keys, commonly known as Access Key Id, and they have the “AKIA” prefix. The behavior of a temporary token that’s created by an AKIA token and creates other temporary tokens is suspicious, and might be indicative of malicious activity occurring within the account.
2. A temporary token which created for EC2 machine and was used outside of that machine and the temporary token was used for persistence. Detecting a temporary token that is suspected as “leaked” or “stolen” from EC2 could be a very intense, inconvenient action. We target tokens that were created for EC2 instances and used to persistence outside of the machine. We are looking after the following records structure:
- Token that in its useridentify object the ARN attribute looks like “arn:aws:sts:{account_id}:assumed-role/aws:ec2-instance/i-{instance_id}”
- The token used in a non-EC2 machine IP address.
Figure 4: Results of the Athena query
Now, after analyzing the results and removing the legitimate records – any records left indicate malicious or suspicious activity in our account.
For determining the root cause of this suspicious activity, we will need to better understand how or, in which way, those tokens were created and what started this persistent loop. Was it a leak of a user’s secret access key, or maybe it started from a token stolen from a compromised EC2?
Temporary tokens that are created by a permanent access key ID (AKIA keys) can be recognized by looking for the AKIA prefix in the access key ID used in the “useridentity” object.
As we mentioned before, the “useridentity” object contains much information about the performer of the recorded action.
After we are having the suspicious token list and discovered all the creation of the temporary tokens, we can find out the root source of the attack, or in this case, the user this key was stolen from.
For those tokens that we couldn’t find their AKIA root source keys, they might be temporary tokens, which haven’t been created by AKIA token and might be stolen tokens from an EC2 machine.
The attacker can create many tokens, which will force you to match every node in the token tree to find your path to the root cause. As you can understand by now, this is a Sisyphean and very inconvenient task.
*NOTE: The detection methods are according to the current AWS logging events format and behavior.
Introducing SkyWrapper
Looking after such attacks is a tough challenge. To make it easier for everybody, we developed an open-source tool called SkyWrapper.
SkyWrapper is a tool that analyzes behaviors of all the temporary tokens created in a given AWS account.
The tool aims to find suspicious creation forms and uses of temporary tokens to detect malicious activity in the account.
The tool analyzes the AWS account, and creates an excel sheet that includes all the currently living temporary tokens and the suspected compromised AKIA keys used to “refresh” temporary tokens.
A summary of the finding is printed to the screen after each run.
In addition, a detailed separate report file is saved, which includes a summary of the results, with the list of the privileged live tokens, a list of the privileged roles attached to the live tokens, and a list of “exploitable” roles to the mentioned attack.
Roles are considered as exploitable/vulnerable roles when they are configured in a way that every temporary token that is created via AssumeRole of these roles can create another temporary tokens.
Figure 5: SkyWrapper results Excel
Figure 6: SkyWrapper run demo
SkyWrapper is available on CyberArk’s GitHub: https://github.com/cyberark/SkyWrapper
AWS STS is a widely used and essential service in any AWS environment, but having the ability to create temporary tokens can present material risk if it is misconfigured or not being monitored and observed. Moreover, AWS STS, as we mentioned before, can be an open door for many risks if misused. You can read more about potential risks that exist in STS in the blog post Ian Williams wrote in the following link: https://blog.netspi.com/gaining-aws-console-access-via-api-keys/. The attack scenario we presented is just one example of an attacker creating a non-ordinary back door in a compromised AWS environment.
We showed how an attacker could convert a temporary token into a long-term token, how they can “hide” their activity trail and actions in the compromised AWS environment, and how this can be very challenging for the security team to address. This attack technique is very stealthy and hard to detect – imagine how long an attacker can have access to your account while you might not even notice anything
However, this technique is just one example of a much bigger problem of managing the temporary tokens in cloud environments. Temporary tokens sometimes get less attention and monitoring than a permanent one, which is a great place for an attacker to take advantage.
Observing and early detection of misconfigured roles could eventually prevent attackers who managed to get access to achieve their goals. I encourage you to run SkyWrapper and get a better view of the temporary tokens that exist in your environment, detect the privileged ones, and handle the ones that are marked as suspicious. You will be surprised to see how many temporary tokens you have in your account.
What are you waiting for? Scan your AWS environment today: https://github.com/cyberark/SkyWrapper.
