Tommi's Scribbles
Using AWS for Container CD/CI
- Published on 2023-07-07
Containerization is the popular kid on the block. AWS has many services built around containers. In this write up, I will cover how CodePipeline can be used with ECS to continuously build and deploy container images to a private container repository.
Initial Setup
To begin with, you will need an AWS account. I expect you to be resourceful enough to find out how to create that on your own. Do note that use of AWS services outside of the free tier can and will likely incur costs.
The services this write up uses are AWS CodeBuild, AWS CodeCommit, AWS CodePipeline, and AWS ECS. You can find a simple skeleton AWS CDK stack on my GitHub profile that you can use as the starting point. Deploying the stack will create a new AWS repository with the contents of the skeleton, including a full build pipeline.
CDK Configuration
AWS has good instructions for how to set up a Java CDK project that uses gradle. Whether you use that or my skeleton project as a starting point is up to you. However, I will not walk through the initial steps, but only the stack. I also expect you to be able to figure out what dependencies you need to add in gradle so you can build the stack.
We start the stack with some includes:
import software.amazon.awscdk.services.codebuild.*; import software.amazon.awscdk.services.codecommit.IRepository; import software.amazon.awscdk.services.codepipeline.Pipeline; import software.amazon.awscdk.services.ecr.Repository; import software.amazon.awscdk.services.s3.Bucket; import software.amazon.awscdk.services.s3.IBucket; import software.constructs.Construct; import software.amazon.awscdk.Stack; import software.amazon.awscdk.StackProps; import software.amazon.awscdk.services.codepipeline.Artifact; import software.amazon.awscdk.services.codepipeline.IStage; import java.util.Map;
Next, we create the stack class:
public class ImgCDCIBaseStack extends Stack
{
}
In the class, we put the basic stack constructors:
public ImgCDCIBaseStack(
final Construct scope,
final String id
)
{
this(scope, id, null);
}
public ImgCDCIBaseStack(
final Construct scope,
final String id,
final StackProps props
)
{
super(scope, id, props);
}
After the call to super, we start crafting a stack. First, we define a bucket to hold the build artifacts. I like having a separate "storage" stack, which includes a bucket for builds. Replace the bucket creation code as needed.
IBucket artifactBucket = Bucket.fromBucketName( this, id + "BuildBucket", "com.yourorg.builds" );
We will obviously need an ECR repository since we are working with containers. We will also need a CodeCommit repository. I am calling my project img-cdci-base.
Repository ecr = CreateContainerRepository(this, id,"img/cdci-base"); IRepository codeCommit = CreateRepository( this, id, "img-cdci-base", "Base AWS CodeBuild stack and image." );
I like to use a custom Java library for my common CDK code. Below are the function excerpts for what is used in this container write-up. I leave it up to you to decide how you want to implement the functions: library, utility class, or just as class methods.
Note: Both ECR and CodeCommit use "Repository". If you have them in the same file, you likely need to disambiguate more.
Also note: when CreateRepository is called, it uses the contents of your CDK project as the initial commit. This requires the following task in your gradle build file, in case you didn't use my skeleton project. Also note, the gradle code is provided in Kotlin, not Groovy.
task("createCommit", Zip::class) {
archiveFileName.set("commit.zip")
destinationDirectory.set(layout.buildDirectory.dir("../../initial"))
val tree = fileTree(project.rootDir) {
exclude("initial", "**/cdk.out/**")
}
from(tree)
}
tasks.named("run") {
dependsOn("createCommit")
}
These are the actual helper snippets.
public static IRepository CreateRepository(
Construct scope,
String parentID,
String repositoryName,
String description)
{
return Repository.Builder.create(
scope,
parentID + repositoryName + "Repository"
)
.code(Code.fromZipFile("../initial/commit.zip", "dev"))
.description(description)
.repositoryName(repositoryName)
.build();
}
public static Repository CreateContainerRepository(
Construct scope,
String parentID,
String repoName
)
{
return Repository.Builder.create(
scope,
parentID + repoName + "Containers"
)
.imageScanOnPush(true)
.repositoryName(repoName)
.build();
}
public static BuildEnvironment GetCustomBuildEnvironment(
IRepository buildImageECR,
String image
)
{
return BuildEnvironment.builder()
.buildImage(
LinuxArmBuildImage.fromEcrRepository(
buildImageECR,
image
)
)
.privileged(true)
.build();
}
public static BuildEnvironment GetDefaultBuildEnvironment()
{
return BuildEnvironment.builder() .buildImage(LinuxArmBuildImage.AMAZON_LINUX_2_STANDARD_2_0)
.privileged(true)
.build();
}
public static Project CreateBuildProject(
Construct scope,
String parentID,
String buildName,
String description,
BuildEnvironment buildEnv,
ISource source,
IBucket artifactBucket)
{
IArtifacts outputArtifacts = Artifacts.s3(
S3ArtifactsProps.builder()
.bucket(artifactBucket)
.includeBuildId(false)
.packageZip(false)
.path(buildName)
.build()
);
return Project.Builder.create(
scope,
parentID + buildName
)
.artifacts(outputArtifacts)
.description(description)
.environment(buildEnv)
.projectName(buildName)
.source(source)
.build();
}
public static void AddBuildActionToStage(
@NotNull IStage stage,
Project buildProject,
Artifact input,
Artifact output)
{
CodeBuildAction buildAction = CodeBuildAction.Builder.create()
.actionName("CodeBuild")
.input(input)
.outputs(List.of(output))
.project(buildProject)
.type(CodeBuildActionType.BUILD)
.variablesNamespace("BuildVariables")
.build();
stage.addAction(buildAction);
}
public static IStage AddNewStageToPipeline(
Pipeline pipeline,
String stageName)
{
return pipeline.addStage(StageOptions.builder()
.stageName(stageName)
.build());
}
public static void AddSourceActionToStage(
@NotNull IStage stage,
IRepository repo,
String branch,
Artifact output)
{
CodeCommitSourceAction sourceAction
= CodeCommitSourceAction.Builder.create()
.actionName("CodeCommit")
.branch(branch)
.output(output)
.repository(repo)
.trigger(CodeCommitTrigger.EVENTS)
.variablesNamespace("SourceVariables")
.build();
stage.addAction(sourceAction);
}
public static Pipeline CreatePipeline(
Construct scope,
String parentID,
String pipelineName,
IBucket artifactBucket)
{
return Pipeline.Builder.create(scope, parentID + pipelineName)
.artifactBucket(artifactBucket)
.crossAccountKeys(false)
.pipelineName(pipelineName)
.build();
}
public static PolicyStatement ECRPermission()
{
List actionList = Arrays.asList(
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:DescribeImages",
"ecr:DescribeRepositories",
"ecr:GetAuthorizationToken",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:InitiateLayerUpload",
"ecr:ListImages", "ecr:PutImage",
"ecr:UploadLayerPart"
);
List resourceList = List.of("*");
return PolicyStatement.Builder.create()
.actions(actionList)
.resources(resourceList)
.build();
}
Armed with the snippets above and the repositories, we can setup CodeCommit. We assume two branches: dev for staging work, and master for release work.
CodeCommitSourceProps propsDev = CodeCommitSourceProps.builder()
.branchOrRef("dev")
.repository(codeCommit)
.build();
ISource codeSourceDev = Source.codeCommit(propsDev);
CodeCommitSourceProps propsProd = CodeCommitSourceProps.builder()
.branchOrRef("master")
.repository(codeCommit)
.build();
ISource codeSourceProd = Source.codeCommit(propsProd);
Next we create the CodeBuild part. We create two build projects, one for "dev" and one for "production". The key things here are that if you check the custom snippets from above, the development project will use the default Amazon Linux 2 Arm64 image, while the production will use the image the staging builds.
Another key thing are the permissions. You won't be able to push or deploy to ECR without the right permissions. Note though: If you plan to use this code in production, please review that the permissions are not too loose.
Project codeBuildDev = CreateBuildProject(
this,
id,
"img-cdci-base-stg",
"Base AWS CodeBuild stack and image builder for dev branch.",
Map.of(
"IMAGE_REPO_NAME",
BuildEnvironmentVariable.builder()
.type(BuildEnvironmentVariableType.PLAINTEXT)
.value(ecr.getRepositoryName())
.build(),
"IMAGE_TAG",
BuildEnvironmentVariable.builder()
.type(BuildEnvironmentVariableType.PLAINTEXT)
.value("beta")
.build(),
"AWS_ACCOUNT_ID",
BuildEnvironmentVariable.builder()
.type(BuildEnvironmentVariableType.PLAINTEXT)
.value("REPLACE_WITH_YOUR_ACCOUNT_ID")
.build()
),
GetDefaultBuildEnvironment(),
codeSourceDev,
artifactBucket
);
codeBuildDev.addToRolePolicy(ECRPermission());
Project codeBuildProd = CreateBuildProject(
this,
id,
"img-cdci-base-gold",
"Base AWS CodeBuild stack and image builder for master branch.",
Map.of(
"IMAGE_REPO_NAME",
BuildEnvironmentVariable.builder()
.type(BuildEnvironmentVariableType.PLAINTEXT)
.value(ecr.getRepositoryName())
.build(),
"IMAGE_TAG",
BuildEnvironmentVariable.builder()
.type(BuildEnvironmentVariableType.PLAINTEXT)
.value("latest")
.build(),
"AWS_ACCOUNT_ID",
BuildEnvironmentVariable.builder()
.type(BuildEnvironmentVariableType.PLAINTEXT)
.value("REPLACE_WITH_YOUR_ACCOUNT_ID")
.build()
),
GetCustomBuildEnvironment(ecr, "beta"),
codeSourceProd,
artifactBucket
);
codeBuildProd.addToRolePolicy(ECRPermission());
If you are wondering what the maps are, they are environment variables usable later on in the buildspec files.
Once we have a repository and a build project, we can create a CodePipeline to implement CD/CI: whenever there is a push to CodeCommit, we do a CodeBuild.
Pipeline pipelineDev = CreatePipeline(
this,
id,
"img-cdci-base-stg-CDCI",
artifactBucket
);
IStage sourceStageDev = AddNewStageToPipeline(
pipelineDev,
"source"
);
Artifact sourceOutputDev = new Artifact("SourceArtifact");
AddSourceActionToStage(
sourceStageDev,
codeCommit,
"dev",
sourceOutputDev
);
IStage buildStageDev = AddNewStageToPipeline(
pipelineDev,
"build"
);
Artifact devBuildOutput = new Artifact("BuildArtifact");
AddBuildActionToStage(
buildStageDev,
codeBuildDev,
sourceOutputDev,
devBuildOutput
);
Pipeline pipelineProd = CreatePipeline(
this,
id,
"img-cdci-base-gold-CDCI",
artifactBucket
);
Artifact sourceOutput = new Artifact("SourceArtifact");
IStage sourceStage = AddNewStageToPipeline(
pipelineProd,
"source"
);
AddSourceActionToStage(
sourceStage,
codeCommit,
"master",
sourceOutput
);
IStage buildStage = AddNewStageToPipeline(
pipelineProd,
"build"
);
Artifact buildOutput = new Artifact("BuildArtifact");
AddBuildActionToStage(
buildStage,
codeBuildProd,
sourceOutput,
buildOutput
);
The code above in essence does the same steps for both development and release pipelines: hook to the right CodeCommit repository and add build step.
Base CodeBuild Container Image
I like to create a base container image that contains all the common things my container images need, but which can be customized further for more specific use cases. You can use whatever image you want as your base, but I like the UBIs provided by Red Hat. This is due to the fact RHEL is "the enterprise linux", and UBIs are smaller in size than for example Oracle Linux's image.
For my container tool of choice I prefer Podman and Buildah. As they are RHEL products, their business model is vastly superior to Docker. They also work nicely with AWS without the need for extra accounts, arbitrary free use limits, or other nastiness.
As CD/CI is something that is useful for all projects, we will go through creating a base image that can be used for Arm64 CodeBuild builds. We will then create a derivative image of that base image for use in projects using Gradle as the build system, again deploying an ECR image we can use for aarch64 CodeBuild builds.
Container configuration for base image
Red Hat offers three different sized images. I like the standard one best as it keeps installing things simple and is not that much bigger in size.
So we start our Containerfile like so:
FROM registry.access.redhat.com/ubi8/ubi:latest AS os LABEL maintainer="your@admin.email" USER root
Now, while UBIs are open to use, Red Hat does have some limitations on packages bundled. So you want to be careful on the commands when you install packages.
RUN yum update \ --disableplugin=subscription-manager \ --disablerepo=* \ --enablerepo=ubi-8-appstream \ --enablerepo=ubi-8-baseos \ -y \ && yum install \ --disableplugin=subscription-manager \ --disablerepo=* \ --enablerepo=ubi-8-appstream \ --enablerepo=ubi-8-baseos \ buildah unzip tar \ -y \ && rm -rf /var/cache/yum
We start by updating the image using flags that make sure we don't infringe on the generous license provided by Red Hat. Next, we install the tools we will need:
- Buildah for building and publishing our container images.
- Unzip and tar for unpacking some archives.
Finally, we remove the yum cache to avoid unnecessarily bloating up the image. This created layer will be used as the base for creating our final base image.
FROM os AS awscli COPY awscli-key /opt ARG CLI_URL ARG CLI_SIG RUN gpg --import /opt/awscli-key \ && curl $CLI_URL \ -o awscliv2.zip \ && curl $CLI_SIG \ -o awscliv2.sig \ && gpg --verify awscliv2.sig awscliv2.zip || exit 1 \ && unzip awscliv2.zip \ && ./aws/install \ && rm -rf ./aws \ && rm awscliv2.zip \ && rm /opt/awscli-key
The final steps install AWS CLI to the image. For the install, you need to place a file called awscli-key next to your Containerfile. The contents of the file are available on the AWS CLI download page. It is the gpg signing key to make sure what you install is the real deal.
The key is copied to the container and added to the gpg keychain. The URL and archive hash, passed as arguments, are used to download the CLI install archive and the signature for the archive. The archive is then verified with the signature.
If all is well, the archive is unpacked, installed, and no longer needed files deleted.
Build configuration for base image
AWS CodeBuild builds based on a YML configuration file. I personally hate YML, but alas. Let's start crafting our buildspec.yml
version: 0.2 env: shell: /bin/sh variables: CLI_URL: "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" CLI_SIG: "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip.sig"
Here we define the variables for the container args to point to the AWS CLI files. Note that I am using ARM here as I prefer the cost effectiveness of ARM over AMD / Intel.
Next we start defining the different build phases.
phases: install: commands: - > yum install yum-utils -y - > yum-config-manager --add-repo=http://mirror.centos.org/altarch/7/os/aarch64/ - > yum-config-manager --add-repo=http://mirror.centos.org/altarch/7/extras/aarch64/ - > yum install --disableplugin=priorities --nogpgcheck buildah -y
As we are building our base build image, we do not yet have buildah available. Also, the default CodeBuild image does not have buildah available. Thus, we install yum-utils and use the config manager to add buildah from the CentOS 7 extras. Reason we use CentOS 7 extras here is that CentOS 8 extras didn't at the time have the ARM version of buildah available.
Disabling the gpg check with nogpgcheck flag is a bit nasty, but it saves us some hassle for the limited use of this initial build. Next we can move to the actual build steps.
pre_build:
commands:
- echo Logging in to Amazon ECR...
- REPO_URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com
- >
aws ecr get-login-password
--region $AWS_DEFAULT_REGION |
buildah login
--username AWS
--password-stdin
${REPO_URI}
We first create the pre-build step. The step simply logs in to AWS ECR using the account and region where the CodeBuild is run, and passes the credentials to buildah.
For the actual build, we construct a tag for the container image from some built-in and CodeBuild environment variables:
- IMAGE_REPO_NAME is an environment variable that has the ECR repository name.
- CODEBUILD_RESOLVED_SOURCE_VERSION is a CodeBuild provided variable holding the source version used for the build.
- CODEBUILD_BUILD_NUMBER is a CodeBuild provided variable holding the sequential number of builds.
We then use buildah to build the container image using the Containerfile we built before, passing in the defined variables for CLI_URL and CLI_SIG.
build:
commands:
- echo Build started on `date`
- BUILD_NAME=$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION-$CODEBUILD_BUILD_NUMBER
- echo Building the container image...
- >
buildah bud
--storage-driver=vfs
--build-arg CLI_URL=$CLI_URL
--build-arg CLI_SIG=$CLI_SIG
-f Containerfile
-t ${BUILD_NAME}
Finally, we add a post build step. The step tags the freshly built image, and then pushes the image to ECS.
We use the IMAGE_TAG environment variable provided by our CDK stack above. That way, staging build will deploy with tag beta, while release tags will deploy with tag latest. We also tag the images with an ALT_TAG using source version and build number, thus allowing locking and pulling specific versions as well.
post_build:
commands:
- echo Build completed on `date`
- IMG_URI=${REPO_URI}/${IMAGE_REPO_NAME}:${IMAGE_TAG}
- ALT_TAG=${REPO_URI}/${IMAGE_REPO_NAME}:${CODEBUILD_RESOLVED_SOURCE_VERSION}-${CODEBUILD_BUILD_NUMBER}
- echo Tagging image ${ALT_TAG}...
- >
buildah tag
--storage-driver=vfs
${BUILD_NAME}
${ALT_TAG}
- echo Pushing the container image ${ALT_TAG}...
- >
buildah push
--storage-driver=vfs
${ALT_TAG}
- echo Tagging latest image ${IMG_URI}...
- >
buildah tag
--storage-driver=vfs
${BUILD_NAME}
${IMG_URI}
- echo Pushing the latest container image ${IMG_URI}...
- >
buildah push
--storage-driver=vfs
${IMG_URI}
And voila! We have an image in ECR we can use as the image used for CodeBuild CD/CI projects.
Derivative CodeBuild Container Image
Now that we have a base container image, we can use that image as the base for derivative builds. That is, we can create another CodeCommit repository, and add CodeBuild project to build other containers, using images we have pushed to ECR as the source.
You get two benefits out of this. First, if there's a common set you need in all your container images, you can maintain everything in one image. Second, you get some isolation from the image you use as base (in this case UBI) and your own internal systems as instead of continuously pulling an image someone else is in control of, you only pull when you want to update the base image.
To help with the above, I like creating two different build-file versions: the "staging" version does the above; the "gold" version just uses one of the staging images as source and tags it with the latest tag. This way, any derivative images can just reference the latest base image instead of building everything from scratch.
If you read carefully, the CDK stack in fact implements this already. The production build project uses the beta tagged version of itself as the build image!
Container configuration for derivative image
So, let's create a derivative image from the base image we created. I want to create an image to use with Gradle-based builds. While gradle allows using gradlew, the system still needs Java installed. So that is what we are going to add.
As before, we start with the Containerfile. You can use the same skeleton project we used above, or duplicate the earlier project and modify accordingly.
The key difference to before is that we use our earlier deployed base image as the source instead of UBI. UBI is still at the bottom of everything as that is what the base image was based off.
Replace the AWSACCOUNTNUMBER with the account number of the ECR you pushed the image above to, and the AWSREGION with the region for the ECR. Here, I assume the image name is cdci-base. I also assume you did as I explained above, and did a "release" version of the image and tagged it latest.
FROM AWSACCOUNTNUMBER.dkr.ecr.AWSREGION.amazonaws.com/img/cdci-base:latest LABEL maintainer="your@email.address" USER root
We then just go straight to installing Java on the image. I like IBM Semeru and 17 is the current LTS, so that is what is assumed to be installed. However, you can easily override and install some other version instead.
ENV JAVA_HOME /usr/lib/jvm/ibm-semeru-open-17-jdk
ENV PATH "${JAVA_HOME}/bin:${PATH}"
COPY jdk-key /opt
ARG JDK_URL
ARG JDK_SHA
RUN curl $JDK_URL \
-LJO \
&& mv *.rpm jdk.rpm \
&& echo $JDK_SHA \
&& echo "$JDK_SHA jdk.rpm" | sha256sum --check || exit 1 \
&& rpm --import /opt/jdk-key \
&& rpm -K jdk.rpm || exit 1 \
&& yum install \
--disableplugin=subscription-manager \
--disablerepo=* \
--enablerepo=ubi-8-appstream \
--enablerepo=ubi-8-baseos \
jdk.rpm \
-y \
-v \
&& rm -rf /var/cache/yum \
&& rm /opt/jdk-key \
&& rm jdk.rpm \
&& export JAVA_HOME \
&& export PATH=$PATH:$JAVA_HOME/bin \
&& java -version
As above, you need to place the signing key for your choice of Java in a file called jdk-key alongside the Containerfile.
Build configuration for derivative image
With the Containerfile ready, it is time to configure the build. It is again very similar to the base image build configuration file.
version: 0.2 env: shell: /bin/sh variables: JDK_URL: "https://github.com/ibmruntimes/semeru17-binaries/releases/download/jdk-17.0.2<>2B8_openj9-0.30.0/ibm-semeru-open-17-jdk-17.0.2.8_0.30.0-1.aarch64.rpm" JDK_SHA: ed1d7b7096227faffa48c0154f5101cdd0f2055330eb77491d9725f6c5acc82f
We start by defining the args. I had some trouble with the SHA file, so I ended up copying it as is instead of downloading it as with the AWS SDK above. The Containerfile reflects that as well.
The build steps are pretty much a carbon copy:
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- REPO_URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com
- >
aws ecr get-login-password
--region $AWS_DEFAULT_REGION |
buildah login
--username AWS
--password-stdin
${REPO_URI}
build:
commands:
- echo Build started on `date`
- BUILD_NAME=$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION-$CODEBUILD_BUILD_NUMBER
- echo Building the container image...
- >
buildah bud
--storage-driver=vfs
--build-arg JDK_URL=$JDK_URL
--build-arg JDK_SHA=$JDK_SHA
-f Containerfile
-t ${BUILD_NAME}
post_build:
commands:
- echo Build completed on `date`
- BUILD_NAME=$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION-$CODEBUILD_BUILD_NUMBER
- IMG_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
- echo Tagging image...
- >
buildah tag
--storage-driver=vfs
${BUILD_NAME}
${IMG_URI}
- echo Pushing the container image...
- >
buildah push
--storage-driver=vfs
${IMG_URI}
And boom! Now we have another image in our ECR, which can be used as an image for any CodeBuild project building the source with Gradle!
What Next?
The above code should hopefully get you started in utilizing AWS services to create container and build automation architectures using infrastructure as code. However, you also probably saw a lot that was not implemented, or was implemented hastily: Lot of code duplication, hardcoded strings, missing dependencies for gradle...
Making the full thing work and tweaking the code should give you a lot of ideas and insights on how you can make this write-up best work for you. But just be careful: on AWS free tier, you get one CodePipeline for free. After that, they are one dollar a month per pop. So don't go too crazy. Or be prepared to pay.