I’ve recently been spending some time learning about Amazon EC2 Image Builder. The idea behind EC2 Image Builder is that you can automatically build an operating system image or container image and store it in the cloud for future deployments. Container images are stored in Amazon Elastic Container Registry (ECR), which you can think of as your own private Docker Hub. OS images are stored as Amazon Machine Images (AMI) in the Amazon Simple Storage Service (S3).
Although EC2 Image Builder works fairly well, there are several improvements that I would like to see around it. Let’s explore what some of these proposed improvements are.
Stream Logs to Amazon CloudWatch
EDIT: Upon further inspection, it turns out that Image Builder actually does send logs to Amazon CloudWatch. The service automatically creates the necessary CloudWatch Log Group and Streams, provided it has the necessary permissions to do so.
As of today, EC2 Image Builder stores several different log files in Amazon S3 buckets. While this is an acceptable storage medium, you need to download the individual log files from S3 in order to view their contents. Additionally, the log format is not terribly easy to read, as many of the logs are in JSON format.
Instead of simply storing the logs in S3, I would like to see easy to image builder store logs in a human readable format within the Amazon CloudWatch service. To take it a step further, the logs could be exposed inside the EC2 Image Builder service console, so I don’t have to navigate to a different service in order to see them.
As with most Amazon services, EC2 Image Builder provides a raw interface. Amazon seems to expect other people to build utilities around their services, instead of providing ready-to-use services. In some cases, this is a nice approach, but I surmise most people just want to use the service as-is. Building utilities takes time, and customers of AWS need their services to “just work.”
Real-time Linting for YAML Components
EC2 Image Builder utilizes a concept known as components to install software, perform system configuration, and perform validation tests. These components are built by the end customer using YAML syntax, not too different from Ansible. Each component contains a build or test phase, which is broken down into individual “steps” that perform the actual work. Each step of a phase can invoke certain “action modules,” each of which performs a specific task.
When you create a component YAML configuration inside the AWS Management Console, there is no immediate feedback about whether or not the configuration is valid. When you click the button to Create Component, however, you will receive an error if any validation fails.
Instead, I would like to see some tooling, either directly inside the AWS Management Console, for added to the AWS Toolkit extension for Microsoft Visual Studio Code. The tooling should perform real-time linting on EC2 Image Builder component YAML files, ensuring the following.
- Overall document structure is correct – no typos in YAML properties
- Only valid action modules are used
- Required parameters (aka. “inputs”) to the action modules are specified
- Values for inputs match the expected values
Real-time linting tools will reduce the burden on cloud engineers to validate their configurations using the AWS Management Console, and reduce the feedback interval.
Trigger EC2 Image Builder Pipelines from Amazon EventBridge
Today, you can manually invoke a pipeline in EC2 Image Builder, or you can schedule it to run on a time-based interval using cron expressions. Manually invoking a pipeline is possible using the AWS SDKs, including the AWS Tools for PowerShell, however this requires writing extra code.
The Amazon EventBridge (CloudWatch Events) service can trigger a variety of other AWS services directly, although EC2 Image Builder is not one of them. Security teams may want to integrate vulnerability and patching feeds with EC2 Image Builder, to immediately trigger a fresh build once a patch is released. Instead of writing some additional code in AWS Lambda to “glue” EventBridge and EC2 Image Builder together, EventBridge should be able to trigger a pipeline directly.
There are already several other integrations between EventBridge and EC2, providing functions such as stopping or rebooting EC2 instances, creating snapshots, and more.
So there you have it, those are just a few ideas I had off the top of my head, to improve the EC2 Image Builder service and its integration with other AWS services. What kind of functionality would you like to see in EC2 Image Builder, sometime in the future?