Introducing the new Docker Build provider
Posted on
Deploying and managing containerized workloads is one of Pulumi’s fastest-growing areas. Standing up managed container services and Kubernetes clusters is a common area for automation, and many of our customers use Pulumi to automate building and publishing images to their registry of choice.
Given this critical use case, we are thrilled to introduce the latest addition to the Pulumi ecosystem: the new Docker Build provider, designed to streamline and modernize Docker image builds from your Pulumi programs.
This new addition expands upon the solid foundation of our highly utilized Docker provider. It elevates the docker.Image
resource, one of the most heavily used and most powerful resources in the Pulumi ecosystem, into its very own dedicated package. By fully embracing the power of BuildKit, the Docker Build provider brings many new features to support the best-in-class Docker capabilities developers love.
The new provider exposes Docker’s next-gen buildx interface, so now even your most complex Docker images can be handled by Pulumi. This is a remarkably flexible and versatile tool for building images and includes key features such as,
Multi-platform image support: Build images that run seamlessly across different hardware architectures.
Advanced caching mechanisms: Speed up builds and reduce resource consumption by utilizing cache backends like S3, GitHub Actions, local disk, and more.
Support for build secrets: Unlike building images with the
docker
CLI, which requires juggling environment variables or files on disk, Pulumi’s first-class support for secrets means you can now safely and efficiently incorporate sensitive information into your builds. With Pulumi ESC (Environments, Secrets, and Configuration), it’s easy to share build secrets with other developers and teams.Support for multiple export types: Export your images to registries, disk, or blob storage to increase the power and flexibility of your workflows.
Support for Docker Build Cloud: Use Docker Build Cloud to offload your builds and caches to the cloud, enhancing productivity and performance.
A brief history
Pulumi first introduced the Docker provider in 2018, and it has been widely adopted since. Over the years, Pulumi has invested in the Docker provider, notably introducing version 4.0 last year and additional improvements like build-on-preview
behavior.
At the same time, the Docker build ecosystem has seen tremendous advancements in how images can be built and distributed. These changes were initially experimental, but they became official when BuildKit graduated to become the default builder in Docker version 23.
We have heard many requests from users to expose more BuildKit functionality in the Docker provider. After some consideration, we decided that a new, standalone provider focused exclusively on building images would provide the best overall user experience. In addition, it allows Pulumi to stay current with the evolving Docker build landscape.
Getting started
Add the docker-build
package to your Pulumi program to take full advantage of modern image builds.
import * as docker_build from "@pulumi/docker-build";
import pulumi_docker_build as docker_build
using DockerBuild = Pulumi.DockerBuild;
import "github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild"
resources:
my-image:
type: docker-build:Image
import com.pulumi.dockerbuild.Image
Suppose your Pulumi program has a subfolder called ./app
that contains a file named Dockerfile
(its contents can be as simple as FROM alpine
). In that case, the example below shows:
- how to build a multi-platform image (
linux/amd64
andlinux/arm64
), - publish the image to a remote AWS ECR registry, and;
- use an inline cache to speed up subsequent builds.
import * as pulumi from "@pulumi/pulumi"
import * as aws from "@pulumi/aws";
import * as docker_build from "@pulumi/docker-build";
// Create an ECR repository for pushing.
const ecrRepository = new aws.ecr.Repository("ecr-repository", {});
// Grab auth credentials for ECR.
const authToken = aws.ecr.getAuthorizationTokenOutput({
registryId: ecrRepository.registryId,
});
// Build and push an image to ECR with inline caching.
const myImage = new docker_build.Image("my-image", {
// Tag our image with our ECR repository's address.
tags: [pulumi.interpolate`${ecrRepository.repositoryUrl}:latest`],
context: {
location: "./app",
},
// Use the pushed image as a cache source.
cacheFrom: [{
registry: {
ref: pulumi.interpolate`${ecrRepository.repositoryUrl}:latest`,
},
}],
// Include an inline cache with our pushed image.
cacheTo: [{
inline: {},
}],
// Build a multi-platform image manifest for ARM and AMD.
platforms: [
"linux/amd64",
"linux/arm64",
],
// Push the final result to ECR.
push: true,
// Provide our ECR credentials.
registries: [{
address: ecrRepository.repositoryUrl,
password: authToken.password,
username: authToken.userName,
}],
});
// Export a ref for the pushed images so we can deploy it.
export const ref = myImage.ref;
import pulumi
import pulumi_aws as aws
import pulumi_docker_build as docker_build
# Create an ECR repository for pushing.
ecr_repository = aws.ecr.Repository("ecr-repository")
# Grab auth credentials for ECR.
auth_token = aws.ecr.get_authorization_token_output(registry_id=ecr_repository.registry_id)
# Build and push an image to ECR with inline caching.
my_image = docker_build.Image("my-image",
# Tag our image with our ECR repository's address.
tags=[ecr_repository.repository_url.apply(lambda repository_url: f"{repository_url}:latest")],
context=docker_build.BuildContextArgs(
location="./app",
),
# Use the pushed image as a cache source.
cache_from=[docker_build.CacheFromArgs(
registry=docker_build.CacheFromRegistryArgs(
ref=ecr_repository.repository_url.apply(lambda repository_url: f"{repository_url}:latest"),
),
)],
# Include an inline cache with our pushed image.
cache_to=[docker_build.CacheToArgs(
inline=docker_build.CacheToInlineArgs(),
)],
# Build a multi-platform image manifest for ARM and AMD.
platforms=[
docker_build.Platform.LINUX_AMD64,
docker_build.Platform.LINUX_ARM64,
],
# Push the final result to ECR.
push=True,
# Provide our ECR credentials.
registries=[docker_build.RegistryArgs(
address=ecr_repository.repository_url,
password=auth_token.password,
username=auth_token.user_name,
)],
)
# Export a ref for the pushed images so we can deploy it.
pulumi.export("ref", my_image.ref)
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;
using DockerBuild = Pulumi.DockerBuild;
return await Deployment.RunAsync(() =>
{
// Create an ECR repository for pushing.
var ecrRepository = new Aws.Ecr.Repository("ecr-repository");
// Grab auth credentials for ECR.
var authToken = Aws.Ecr.GetAuthorizationToken.Invoke(new()
{
RegistryId = ecrRepository.RegistryId,
});
// Build and push an image to ECR with inline caching.
var myImage = new DockerBuild.Image("my-image", new()
{
// Tag our image with our ECR repository's address.
Tags = new[]
{
ecrRepository.RepositoryUrl.Apply(repositoryUrl => $"{repositoryUrl}:latest"),
},
Context = new DockerBuild.Inputs.BuildContextArgs
{
Location = "./app",
},
// Use the pushed image as a cache source.
CacheFrom = new[]
{
new DockerBuild.Inputs.CacheFromArgs
{
Registry = new DockerBuild.Inputs.CacheFromRegistryArgs
{
Ref = ecrRepository.RepositoryUrl.Apply(repositoryUrl => $"{repositoryUrl}:latest"),
},
},
},
// Include an inline cache with our pushed image.
CacheTo = new[]
{
new DockerBuild.Inputs.CacheToArgs
{
Inline = null,
},
},
// Build a multi-platform image manifest for ARM and AMD.
Platforms = new[]
{
DockerBuild.Platform.Linux_amd64,
DockerBuild.Platform.Linux_arm64,
},
// Push the final result to ECR.
Push = true,
// Provide our ECR credentials.
Registries = new[]
{
new DockerBuild.Inputs.RegistryArgs
{
Address = ecrRepository.RepositoryUrl,
Password = authToken.Apply(getAuthorizationTokenResult => getAuthorizationTokenResult.Password),
Username = authToken.Apply(getAuthorizationTokenResult => getAuthorizationTokenResult.UserName),
},
},
});
// Export a ref for the pushed images so we can deploy it.
return new Dictionary<string, object?>
{
["ref"] = myImage.Ref,
};
});
package main
import (
"fmt"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ecr"
"github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an ECR repository for pushing.
ecrRepository, err := ecr.NewRepository(ctx, "ecr-repository", nil)
if err != nil {
return err
}
// Grab auth credentials for ECR.
authToken := ecr.GetAuthorizationTokenOutput(ctx, ecr.GetAuthorizationTokenOutputArgs{
RegistryId: ecrRepository.RegistryId,
}, nil)
// Build and push an image to ECR with inline caching.
myImage, err := dockerbuild.NewImage(ctx, "my-image", &dockerbuild.ImageArgs{
// Tag our image with our ECR repository's address.
Tags: pulumi.StringArray{
ecrRepository.RepositoryUrl.ApplyT(func(repositoryUrl string) (string, error) {
return fmt.Sprintf("%v:latest", repositoryUrl), nil
}).(pulumi.StringOutput),
},
Context: &dockerbuild.BuildContextArgs{
Location: pulumi.String("./app"),
},
// Use the pushed image as a cache source.
CacheFrom: dockerbuild.CacheFromArray{
&dockerbuild.CacheFromArgs{
Registry: &dockerbuild.CacheFromRegistryArgs{
Ref: ecrRepository.RepositoryUrl.ApplyT(func(repositoryUrl string) (string, error) {
return fmt.Sprintf("%v:latest", repositoryUrl), nil
}).(pulumi.StringOutput),
},
},
},
// Include an inline cache with our pushed image.
CacheTo: dockerbuild.CacheToArray{
&dockerbuild.CacheToArgs{
Inline: nil,
},
},
// Build a multi-platform image manifest for ARM and AMD.
Platforms: dockerbuild.PlatformArray{
dockerbuild.Platform_Linux_amd64,
dockerbuild.Platform_Linux_arm64,
},
// Push the final result to ECR.
Push: pulumi.Bool(true),
// Provide our ECR credentials.
Registries: dockerbuild.RegistryArray{
&dockerbuild.RegistryArgs{
Address: ecrRepository.RepositoryUrl,
Password: authToken.ApplyT(func(authToken ecr.GetAuthorizationTokenResult) (*string, error) {
return &authToken.Password, nil
}).(pulumi.StringPtrOutput),
Username: authToken.ApplyT(func(authToken ecr.GetAuthorizationTokenResult) (*string, error) {
return &authToken.UserName, nil
}).(pulumi.StringPtrOutput),
},
},
})
if err != nil {
return err
}
// Export a ref for the pushed images so we can deploy it.
ctx.Export("ref", myImage.Ref)
return nil
})
}
description: Push to AWS ECR with caching
name: ecr
outputs:
ref: ${my-image.ref}
resources:
# Create an ECR repository for pushing.
ecr-repository:
type: aws:ecr:Repository
# Build and push an image to ECR with inline caching.
my-image:
type: docker-build:Image
properties:
# Tag our image with our ECR repository's address.
tags:
- ${ecr-repository.repositoryUrl}:latest
context:
location: ./app
# Use the pushed image as a cache source.
cacheFrom:
- registry:
ref: ${ecr-repository.repositoryUrl}:latest
# Include an inline cache with our pushed image.
cacheTo:
- inline: {}
# Build a multi-platform image manifest for ARM and AMD.
platforms:
- linux/amd64
- linux/arm64
# Push the final result to ECR.
push: true
# Provide our ECR credentials.
registries:
- address: ${ecr-repository.repositoryUrl}
password: ${auth-token.password}
username: ${auth-token.userName}
runtime: yaml
variables:
auth-token:
# Grab auth credentials for ECR.
fn::aws:ecr:getAuthorizationToken:
registryId: ${ecr-repository.registryId}
package myapp;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.ecr.Repository;
import com.pulumi.aws.ecr.EcrFunctions;
import com.pulumi.aws.ecr.inputs.GetAuthorizationTokenArgs;
import com.pulumi.dockerbuild.Image;
import com.pulumi.dockerbuild.ImageArgs;
import com.pulumi.dockerbuild.inputs.CacheFromArgs;
import com.pulumi.dockerbuild.inputs.CacheFromRegistryArgs;
import com.pulumi.dockerbuild.inputs.CacheToArgs;
import com.pulumi.dockerbuild.inputs.CacheToInlineArgs;
import com.pulumi.dockerbuild.inputs.BuildContextArgs;
import com.pulumi.dockerbuild.inputs.RegistryArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
// Create an ECR repository for pushing.
var ecrRepository = new Repository("ecrRepository");
// Grab auth credentials for ECR.
final var authToken = EcrFunctions.getAuthorizationToken(GetAuthorizationTokenArgs.builder()
.registryId(ecrRepository.registryId())
.build());
// Build and push an image to ECR with inline caching.
var myImage = new Image("myImage", ImageArgs.builder()
// Tag our image with our ECR repository's address.
.tags(ecrRepository.repositoryUrl().applyValue(repositoryUrl -> String.format("%s:latest", repositoryUrl)))
.context(BuildContextArgs.builder()
.location("./app")
.build())
// Use the pushed image as a cache source.
.cacheFrom(CacheFromArgs.builder()
.registry(CacheFromRegistryArgs.builder()
.ref(ecrRepository.repositoryUrl().applyValue(repositoryUrl -> String.format("%s:latest", repositoryUrl)))
.build())
.build())
// Include an inline cache with our pushed image.
.cacheTo(CacheToArgs.builder()
.inline()
.build())
// Build a multi-platform image manifest for ARM and AMD.
.platforms(
"linux/amd64",
"linux/arm64")
// Push the final result to ECR.
.push(true)
// Provide our ECR credentials.
.registries(RegistryArgs.builder()
.address(ecrRepository.repositoryUrl())
.password(authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult).applyValue(authToken -> authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult.password())))
.username(authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult).applyValue(authToken -> authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult.userName())))
.build())
.build());
ctx.export("ref", myImage.ref());
}
}
The Pulumi output, ref
, provides a convenient way to reference the pushed image in downstream Pulumi resources like ECS TaskDefinitions or Kubernetes Deployments.
Migrating from docker.Image to docker_build.Image
The new docker_build.Image
replaces the old docker.Image
resource. It adds new features and addresses many bugs with the previous resource by fully aligning with Docker’s underlying buildx commands. From now on, we will continue to invest in Docker Build and the docker_build.Image
resource.
We do not expect to make further changes to docker.Image
. It will remain available for the foreseeable future, so you can continue using it if it meets your needs. However, we recommend you migrate your images to docker_build.Image
to get the best possible support, features, and performance. We expect to deprecate the docker.Image
resource in the future.
The migration process is straightforward and detailed in the API documentation. Docker Build provides a superset functionality over the previous
docker.Image
resource; thus, you can migrate existing resources without issue. The new Docker Build Image options will look very familiar if you use the Docker command-line tool.
push: true
in the code just as you would include --push
on the command line.buildOnPreview
flag.Jump over to the applicable migration guide to learn how to update your existing Pulumi program:
Migrating from Docker provider v3
Version 3.x of the Docker provider is still widely used because it exposes BuildKit
functionality through raw command-line arguments. However, the new Docker Build provider exposes those arguments as top-level fields on the resource.
Reference the below TypeScript code to compare a v3 definition of a docker.Image
with the new dockerbuild.Image
. In particular, note the extraOptions
in v3 are now top-level fields, e.g., cacheFrom
, cacheTo
, platforms
, etc.
// v3 Image
const v3 = new docker.Image("v3-image", {
imageName: "myregistry.com/user/repo:latest",
localImageName: "local-tag",
skipPush: false,
build: {
dockerfile: "./Dockerfile",
context: "../app",
target: "mytarget",
args: {
MY_BUILD_ARG: "foo",
},
env: {
DOCKER_BUILDKIT: "1",
},
extraOptions: [
"--cache-from",
"type=registry,myregistry.com/user/repo:cache",
"--cache-to",
"type=registry,myregistry.com/user/repo:cache",
"--add-host",
"metadata.google.internal:169.254.169.254",
"--secret",
"id=mysecret,src=/local/secret",
"--ssh",
"default=/home/runner/.ssh/id_ed25519",
"--network",
"host",
"--platform",
"linux/amd64",
],
},
registry: {
server: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
},
});
// v3 Image after migrating to docker-build.Image
const v3Migrated = new dockerbuild.Image("v3-to-buildx", {
tags: ["myregistry.com/user/repo:latest", "local-tag"],
push: true,
dockerfile: {
location: "./Dockerfile",
},
context: {
location: "../app",
},
target: "mytarget",
buildArgs: {
MY_BUILD_ARG: "foo",
},
cacheFrom: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
cacheTo: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
secrets: {
mysecret: "value",
},
addHosts: ["metadata.google.internal:169.254.169.254"],
ssh: {
default: ["/home/runner/.ssh/id_ed25519"],
},
network: "host",
platforms: ["linux/amd64"],
registries: [{
address: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
}],
});
Migrating from Docker provider v4
The new Docker Build provider largely exposes the same fields as v4, but the fields are pluralized or renamed to better align with the Docker CLI.
Reference the below TypeScript code to compare a v4 definition of a docker.Image
with the new dockerbuild.Image
. In particular, note the top-level, pluralized names, e.g., registries
, platforms
, etc.
// v4 Image
const v4 = new docker.Image("v4-image", {
imageName: "myregistry.com/user/repo:latest",
skipPush: false,
build: {
dockerfile: "./Dockerfile",
context: "../app",
target: "mytarget",
args: {
MY_BUILD_ARG: "foo",
},
cacheFrom: {
images: ["myregistry.com/user/repo:cache"],
},
addHosts: ["metadata.google.internal:169.254.169.254"],
network: "host",
platform: "linux/amd64",
},
buildOnPreview: true,
registry: {
server: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
},
});
// v4 Image after migrating to docker-build.Image
const v4Migrated = new dockerbuild.Image("v4-to-buildx", {
tags: ["myregistry.com/user/repo:latest"],
push: true,
dockerfile: {
location: "./Dockerfile",
},
context: {
location: "../app",
},
target: "mytarget",
buildArgs: {
MY_BUILD_ARG: "foo",
},
cacheFrom: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
cacheTo: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
addHosts: ["metadata.google.internal:169.254.169.254"],
network: "host",
platforms: ["linux/amd64"],
registries: [{
address: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
}],
});
Conclusion
Launching the new Docker Build provider marks a significant milestone in modernizing container management for developers. The new package, which contains the docker_build.Image
resource reflects our commitment to innovation and user-centric development. By incorporating the latest BuildKit technology, the Docker Build provider supports the newest features, enhances multi-architecture builds, and improves caching strategies. Moreover, its integration with Docker Build Cloud underscores our dedication to providing a cutting-edge toolset for developers. This launch is a testament to our ongoing efforts to modernize and improve how developers build images.
Check out our documentation for more details on utilizing the Docker Build provider.
If you are still wondering, “Which provider should I use?” the answer depends on whether you are doing anything related to docker build
.
Provider | Use cases |
---|---|
@pulumi/docker-build | Use this provider to build and push Docker images. Ideal for scenarios requiring automation of image creation with Docker, including setting up CI/CD pipelines for Docker image generation. |
@pulumi/docker | Use this provider for everything else, including managing local Docker containers, networks, and volumes on the host machine. |