AMI & Snapshot Management Using AWS Lambda

Automating tasks and continuously improving workflows are key parts of DevOps. In a CI/CD pipeline, being able to simplify processes is a huge plus. Not only will simplifying systemic processes make life easier for you as a developer, but there are other added benefits to gain as well. For example, you can significantly reduce the cost of running services in the cloud with refined and automated scaling of infrastructure.

AWS Lambda is the latest in cloud computing refinement which offers the ability to execute "serverless" code. Meaning, that AWS will provide the run-time platform for us. We’re now used to using  Amazon Machine Images (AMIs) to store software configuration and other variables. AMIs certainly minimize human errors in a CI/CD pipeline. Problems such as misconfiguration can be avoided entirely. AWS Lambda simply supercharges the use of AMIs by making them completely serverless.

Why Go Serverless?

Let’s not forget that AMIs need to run off of an instance, and most of the time that instance needs to be kept alive for the scripts to run properly. Automation becomes an added cost, despite bringing a lot of benefits. Switching to a serverless design makes AMIs that much more powerful.

At the same time, scripts designed to automate cloud instance creation, update, maintenance, and deletion can be triggered by events. This means you can configure the entire cloud environment to be more agile as well.

This leads to another major benefit: more automation opportunities. With AMIs, you can automate the deletion of old AMI backups without having to keep an EC2 instance running all the time. You can also trigger actions based on storage usage or other metrics captured by CloudWatch.

In this post, we will cover how to automate AMIs for your instances using AWS Lambda and CloudWatch. We have built two functions to achieve this, one is to create AMIs and the other is to delete older AMIs with retention.

AWS Lambda is perfect for these tasks. The service is designed for small and on-demand apps or services that are meant to be triggered by specific events. AWS handles the constant monitoring for you, giving you more resources to focus on the rest of the process. Lambda currently supports the following languages: Node.js, Java, C#, and Python. We'll be using Python3.7 to write our functions in this post. We will use CloudWatch Events to trigger the Lambda function, and we can create CloudWatch rules using Cron expressions.

Configuring AWS Lambda

Similar to other Amazon services, you need to start the configuration process by giving AWS Lambda sufficient service roles to access the rest of the environment. Lambda needs access to describe instances, create/deregister images, and delete snapshots. You can do this from the IAM menu from the AWS Management Console or by creating a policy directly. Log on to AWS Management Console and click on the IAM Service which will redirect you to the IAM console.

• Click on ‘Policies’ on the left pane of the console.

1. Click on ‘Create policy’, then choose the ‘JSON’ tab and paste the below JSON data into the field.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": [
               "ec2:DeleteSnapshot",
               "ec2:CreateSnapshots",
               "ec2:CreateSnapshot"
           ],
           "Resource": [
               "arn:aws:ec2:*:*:instance/*",
               "arn:aws:ec2:*::snapshot/*",
               "arn:aws:ec2:*:*:volume/*"
           ]
       },
       {
           "Sid": "VisualEditor1",
           "Effect": "Allow",
           "Action": [
               "ec2:DescribeImages",
               "ec2:DeregisterImage",
               "ec2:DescribeInstances",
               "ec2:DescribeSnapshotAttribute",
               "ec2:DescribeInstanceAttribute",
               "ec2:DescribeImageAttribute",
               "logs:*",
               "ec2:CreateImage",
               "ec2:DescribeSnapshots",
               "ec2:DescribeInstanceCreditSpecifications",
               "ec2:DescribeInstanceStatus"
           ],
           "Resource": "*"
       }
   ]
}

• Click on ‘Visual editor’ to double check the given access , then select ‘Review policy’.
• Give your new policy a unique name and check the constraints for the policy name. Then click on ‘Create policy’.
• Now, back in the IAM Console, go to ‘Roles’ to create an IAM Role attach the policy we have created. Start by clicking on ‘Create role’.
• Click on ‘Lambda’ as the service that will use this role.

•  Search for your created IAM Policy and click on the checkbox.

•  Review the IAM Role and Click on Create Role.

Create Lambda Functions

The next step is adding functions to the Lambda instance. Depending on what you are trying to achieve, you can add functions that perform specific tasks under certain conditions. For example, you can create a function for making and managing AMI backups based on tags. You can use tags like “Backup” and “Delete” to organize AMI backups.

•  Go to the ‘Lambda’ service in your AWS Management Console.

•  Go to ‘Functions’ and click on ‘Create function’.

•  Fill the function details in as per the below screenshot and attach the role that you have just created.

•  Scroll down to ‘Function code’ and copy this code into the field.

• Next, scroll down to ‘Environment variables’ and create the environment variables as below.

       Or  mention the ‘instance id’ directly as in below screenshot

•  Set the timeout value to 15 mins; this is the threshold value of execution time for our Lambda function.

•  Save the changes by clicking on ‘Save’ in the top-right corner, and then click on ‘Test’.

•  Now to ‘Configure a test event’.

•  Create a tag “backup” for instances that you wanted to schedule for image creation.

• Now go to the Lambda function and run it by clicking ‘Test’. We can check the output in log output and we can also check in our CloudWatch logs. Track the created AMIs in the AMI console and you’ll be able to see your respective snapshots in the Snapshot console too. 

• Next, go to the CloudWatch console, click on logs and search for your Lambda functions. We can explore all the logs and also set ‘Expire Events After’ to 30 days.

•  Go back to the Lambda function. And click on the ‘Add trigger’ in the designer console and select the CloudWatch Events.

• We can add a Cron expression to the Lambda function, this will automatically trigger the function as per the Cron instructions. The Cron expression mentioned in the screenshot is for a daily launch at 5 pm. For more details on Cron expressions, please refer to AWS Schedule Expressions.

•  Similarly, repeat the above steps and create a ‘Delete function’ and copy this code to the ‘Function’ code and add ‘Environment variables’ again as per the below screenshot. We can add a retention period as 30. It means the function scans the list for AMIs which are older than 30 days and deletes them accordingly. We can change the retention to 15 or 30 based upon our requirement. Also, add timeout values and triggers as a create AMI function.

•  Congratulations, you have successfully created Lambda functions and configured them to automatically manage AMIs and EBS Snapshots.

As well as CloudWatch, Lambda can capture triggers from various sources, including SNS. It also supports API calls and AWS IoT for more complex operations.

As per the above steps, the Lambda functions you create will cycle through your EC2 cycles, create a list of all instances with a suitable tag, and begin the process of creating AMIs for those instances. The created AMI backups can have metadata and tags added to them, including information about how long they need to be retained.

As shown, the same function can also automate the deletion of AMI it creates after a certain period of time has passed. Older AMIs are automatically deleted and new AMI backups are created as part of the cycle. It is a simple routine, but one that saves you a lot of time and money on maintaining AMI backups.

Ibexlabs is an experienced DevOps & Managed Services provider and an AWS consulting partner. Our AWS Certified DevOps consultancy team evaluates your infrastructure and make recommendations based on your individual business or personal requirements. Contact us today and set up a free consultation to discuss a custom-built solution tailored just for you.