Keeping an Eye on your IaC
Business problem to solve: Monitor our AWS IaC for potential changes outside of the IaC workflows.
Context: an AWS object is considered to be a part of our IaC if it has a proper tag managed-by set.
False Start
I first hoped to use CloudTrail Lake event data store with event enrichment enabled so that the event responsible for the change already had a proper tag (if any) attached. This did not really work.
Solution
This article outlines the solution.
It all starts with the AWS Config with the recording on. Check the AWS Console Config Settings section. We will monitor changes to Configuration Item (CI). CIs support a wide range of AWS objects.
We add an Event Bridge rule to to pass a CI change event to an SQS queue. My first thought was to handle this change event in lambda, only to realize that the event itself does not carry enough information. Corresponding CloudTrail event(s) DO have this information, but may take up to 10min to become available.
An SQS queue with DLQ and a 10min delay ensures retries and ultimately scalability.
A lambda is attached to the SQS queue and does the bulk of the work:
- Lambda consumes the the CI change event from the queue,
- Checks the tags of the object affected, if the object is not IaC, the change is ignored.
- Calculates the timeframe for the lookup of the corresponding events in the CLoudTrail - expected to be within 5 min.
- Performs the CloudTrail lookup - we are interested in the user identity first and foremost.
- Finally, lambda publishes the message to Slack outlining who changed which IaC object and when.