I Exposed AWS Access Keys, On Purpose: Here’s What I Learned and How I Boosted Incident Response

Introduction

Who doesn’t know that, you just write a quick fix for your code? After a few hours, the function is finally finished.

Proud to have finally done it. Your Brain, taost. Your commit, fast. Pushing your credentials without realising, even faster! Let’s find out what happens next.

It happens like that every day. However, I wanted to experience this myself. What is happening on the AWS side, and what steps can I add to increase our security posture here. So join me on my journey of making the Access Keys public so we can both learn from it.

Setup

The setup for this experiment is simple:

• A public repository on GitHub
• A clumsy user (me)
• Access key from that user
• SNS
• SES
• Step Functions
• EventBridge

So the first thing I did was to create a repository on GitHub. Nothing special. It contains a README.md and .env file. That should be enough. Usually the .env file shouldn’t be committed in any repository, but clumsy me wanted to finally finish the work. So I forgot to add this to the .gitignore. Whoopsie!

Luckily, for me, I’m using a specific user for this case alone. Since I knew beforehand that I will expose those keys “by accident”, I was prepared. After all, exposing any keys is always dangerous. Don’t try this at home!

The dummy user is not allowed to do anything, nada, niente. Not only implicitly, but explicitly! AWS has already created a role for this purpose AWSDenyAll. So I’m using this policy for that clumsy user.

Now that that’s done, quickly created a SNS Topic and put my email in it as a subscriber.

Now I have to get the corresponding event. The magic word is therefore: EventBridge Rule. For this, I followed this tutorial. After I set up everything, I did the first test. A little Test and the result: One support ticket, one email from AWS, and no SNS notification. Interesting.
Like the good first class programmer I am, let’s not change anything and run it again. Still no change. Very interesting.

A small adventure

A few exposed keys and just as many support tickets later, I figured it out. Let me explain it to you. You see, the event is triggered by the Support Center. More precisely, as soon as an issue is created. See the following screenshot:

As you can see the event is called AWS_RISK_IAM_QUARANTINE and NOT AWS_RISK_CREDENTIALS_EXPOSED. Glad nobody told me this before! But that is not all. Under the EventBridge rule AWS_RISK_CREDENTIALS_EXPOSED is not available in the dropdown either! Isn’t that great?

Fortunately, the solution here is relatively simple, do it yourself:

Here is the JSON

{
  "source": ["aws.health"],
  "detail-type": ["AWS Health Event"],
  "detail": {
    "service": ["RISK"],
    "eventTypeCategory": ["issue"],
    "eventTypeCode": ["AWS_ACCESS_KEY_EXPOSED", "AWS_RISK_IAM_QUARANTINE"]
  }
}

Yep, there is still more to do. The region plays a decisive role here as well because the Support Center is a global service, but it directly means that the events arrive in us-east-1. Global usually means us-east-1. Remember that! So the EventBridge Rule as well as the SNS Topic and whatever you need to automate in this case must be created in us-east-1!

Some AWS Health events are not Region-specific. Events that aren’t specific to a Region are called global events. These include events sent for AWS Identity and Access Management (IAM). To receive global events, you must create a rule for the US East (N. Virginia) Region.

Source

So again short and sweet:

1. the EventBridge Rule to receive a global event must be in us-east-1.
2. the SNS Topic must be in us-east-1.
3. the event pattern must be as described above

Now this is all set up, time to make a whoopsie-daisy and expose our access key.

The Exposure

As described before, the key is made public in the .env file “by accident”. To do this, I simply enter the Access Key and Secret Access Key and use the magic words git commit -m "exposium!" && git push origin main.

After the push, some processes are initiated at GitHub and AWS. This happens on every commit and this allows AWS to react in time if they find access keys in a repository. You can read here about GitHubs secret scanner. In the default case, 3 things happen:

1. a support ticket is opened on AWS Support
2. you get an email with the info that access keys have been exposed
3. the user gets the policy AWSCompromisedKeyQuarantineV2 attached.

The email describes some steps you can do in this case. Here is an excerpt:

And here is the event that gets created: