Keeping an Eye on your IaC

Business problem to solve: Monitor our AWS IaC for potential changes outside of the IaC workflows.

Context: an AWS object is considered to be a part of our IaC if it has a proper tag managed-by set.

False Start

I first hoped to use CloudTrail Lake event data store with event enrichment enabled so that the event responsible for the change already had a proper tag (if any) attached. This did not really work.

Solution

This article outlines the solution.

It all starts with the AWS Config with the recording on. Check the AWS Console Config Settings section. We will monitor changes to Configuration Item, or CI for short. CIs support a wide range of AWS objects.

We add an Event Bridge rule to to pass a CI change event to an SQS queue. My first thought was to handle this change event in lambda, only to realize that the event itself does not carry enough information, hence the need to look for the corresponding change events in the Cloud Trail.

An SQS queue with DLQ ensures retries and ultimately scalability.

A lambda is attached to the SQS queue and does the bulk of the work:

1. Lambda consumes the the CI change event from the queue,
2. calculates the timeframe for the lookup of the corresponding events in the CLoudTrail - expected to be within 90 seconds.
3. Performs the CloudTrail lookup - we are interested in the user identity first and foremost.
4. Checks the tags of the object affected
5. Finally publishes message to the predefined SNS topic outlining who changed which IaC object and when.

One can consume SNS messages via e-mail or other means such as Slack messaging.