Automating tasks and continuously improving workflows are key parts of DevOps. In a CI/CD pipeline, being able to simplify processes is a huge plus. Not only will simplifying systemic processes make life easier for you as a developer, but there are other added benefits to gain as well. For example, you can significantly reduce the cost of running services in the cloud with refined and automated scaling of infrastructure.
AWS Lambda is the latest in cloud computing refinement which offers the ability to execute "serverless" code. Meaning, that AWS will provide the run-time platform for us. We’re now used to using Amazon Machine Images (AMIs) to store software configuration and other variables. AMIs certainly minimize human errors in a CI/CD pipeline. Problems such as misconfiguration can be avoided entirely. AWS Lambda simply supercharges the use of AMIs by making them completely serverless.
Let’s not forget that AMIs need to run off of an instance, and most of the time that instance needs to be kept alive for the scripts to run properly. Automation becomes an added cost, despite bringing a lot of benefits. Switching to a serverless design makes AMIs that much more powerful.
At the same time, scripts designed to automate cloud instance creation, update, maintenance, and deletion can be triggered by events. This means you can configure the entire cloud environment to be more agile as well.
This leads to another major benefit: more automation opportunities. With AMIs, you can automate the deletion of old AMI backups without having to keep an EC2 instance running all the time. You can also trigger actions based on storage usage or other metrics captured by CloudWatch.
In this post, we will cover how to automate AMIs for your instances using AWS Lambda and CloudWatch. We have built two functions to achieve this, one is to create AMIs and the other is to delete older AMIs with retention.
AWS Lambda is perfect for these tasks. The service is designed for small and on-demand apps or services that are meant to be triggered by specific events. AWS handles the constant monitoring for you, giving you more resources to focus on the rest of the process. Lambda currently supports the following languages: Node.js, Java, C#, and Python. We'll be using Python3.7 to write our functions in this post. We will use CloudWatch Events to trigger the Lambda function, and we can create CloudWatch rules using Cron expressions.
Similar to other Amazon services, you need to start the configuration process by giving AWS Lambda sufficient service roles to access the rest of the environment. Lambda needs access to describe instances, create/deregister images, and delete snapshots. You can do this from the IAM menu from the AWS Management Console or by creating a policy directly. Log on to AWS Management Console and click on the IAM Service which will redirect you to the IAM console.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:DeleteSnapshot",
"ec2:CreateSnapshots",
"ec2:CreateSnapshot"
],
"Resource": [
"arn:aws:ec2:*:*:instance/*",
"arn:aws:ec2:*::snapshot/*",
"arn:aws:ec2:*:*:volume/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"ec2:DescribeImages",
"ec2:DeregisterImage",
"ec2:DescribeInstances",
"ec2:DescribeSnapshotAttribute",
"ec2:DescribeInstanceAttribute",
"ec2:DescribeImageAttribute",
"logs:*",
"ec2:CreateImage",
"ec2:DescribeSnapshots",
"ec2:DescribeInstanceCreditSpecifications",
"ec2:DescribeInstanceStatus"
],
"Resource": "*"
}
]
}
The next step is adding functions to the Lambda instance. Depending on what you are trying to achieve, you can add functions that perform specific tasks under certain conditions. For example, you can create a function for making and managing AMI backups based on tags. You can use tags like “Backup” and “Delete” to organize AMI backups.
Or mention the ‘instance id’ directly as in below screenshot
As well as CloudWatch, Lambda can capture triggers from various sources, including SNS. It also supports API calls and AWS IoT for more complex operations.
As per the above steps, the Lambda functions you create will cycle through your EC2 cycles, create a list of all instances with a suitable tag, and begin the process of creating AMIs for those instances. The created AMI backups can have metadata and tags added to them, including information about how long they need to be retained.
As shown, the same function can also automate the deletion of AMI it creates after a certain period of time has passed. Older AMIs are automatically deleted and new AMI backups are created as part of the cycle. It is a simple routine, but one that saves you a lot of time and money on maintaining AMI backups.
Ibexlabs is an experienced DevOps & Managed Services provider and an AWS consulting partner. Our AWS Certified DevOps consultancy team evaluates your infrastructure and make recommendations based on your individual business or personal requirements. Contact us today and set up a free consultation to discuss a custom-built solution tailored just for you.