Pulumi Blog: Containers

105 Ways to Run Containers: The Cloud Container Iceberg

Adam Gordon Bell — Tue, 03 Dec 2024 10:29:54 -0500

If a computer can be connected to the internet, someone has tried to run a container on it. From quantum computers to smart toasters, from phones in AWS racks to CI pipelines, there’s many ways to deploy containers. While most people only know about the mainstream cloud providers at the surface, there’s actually a vast world of increasingly unusual and specialized options beneath.

This guide serves two purposes: to showcase the surprising breadth of container options available today, and to help you understand the full spectrum of choices – from practical to very experimental. Whether you’re looking for production-ready solutions or just curious about what’s out there in the dark depths, you’ll find something interesting here.

Constraints

But first, we need ground rules for our container adventures. All listed items should:

Technically possible: Must work today with existing tech, even if the setup makes you question your life choices.
Runs real containers: OCI containers only. No JavaScript “containers” pretending to be the real thing.
Actually doable: You can set this up right now with a laptop and a credit card. No theoretical schemes allowed.

Above the iceberg – The usual suspects

These are the familiar faces of container orchestration—the ones you’d introduce to your parents. Reliable, well-documented, and about as exciting as watching paint dry.

Amazon Elastic Kubernetes Service (EKS)
Amazon Elastic Container Service (ECS)
AWS Fargate
Azure Kubernetes Service (AKS)
Google Kubernetes Engine (GKE)
Google Cloud Run
Amazon EC2
AWS App Runner
Azure App Service for Containers
Google Compute Engine with Containers

On the iceberg – The slightly less obvious

As we dip just below the surface, we find services like a rare B-side acoustic release from a big band. It’s not as mainstream but still a respectable choice for running your containers.

AWS Elastic Beanstalk – Deploy containers, debug YAML
Amazon Lightsail Containers – AWS with the guard rails on
Azure Container Apps – Serverless, but complicated
Google App Engine Flexible Environment – PaaS for Containerized Apps
Azure Container Instances (ACI) – Kubernetes without the chaos

The kubernetes multiverse – Because one Universe of k8s wasn’t enough

Diving deeper, we discover there’s an entire multiverse of regional and specialty K8s providers, helpfully cataloged by the CNCF. Think parallel universes, but instead of evil twins, you get slightly different container runtime implementations.

The Global Titans – The international tech giants couldn’t resist joining the K8s party: Huawei Cloud Container Engine, Tencent, and Baidu bring their own flavor of cloud-native orchestration to the mix. Add in Samsung, China Mobile KCS, JD Cloud, Inspur, and QingCloud, and you have enough options to start a diplomatic incident.
The European Union of Containers – Europeans take their container privacy seriously: IONOS and plusserver bring German engineering to K8s, OVH adds French flair, Exoscale offers Swiss precision, and STACKIT, SysEleven, WIIT, Gcore, and Previder ensure your containers comply with every possible EU regulation. Conoa Proact throws in some Swedish design principles for good measure.
The Asia-Pacific Pod Party – The APAC region isn’t missing out: NHN brings Korean optimization to your clusters, while ARDC Nectar and Catalyst let you run containers upside down in Australia and New Zealand. TKS Cloud Service ensures your containers can run anywhere from Tokyo to Tasmania.
The Specialists’ Society – For when regular K8s isn’t quirky enough: OKESTRO VIOLA orchestrates your containers with musical flair, while Petasus, De Novo, Aruba, and Taikun provide enough specialty K8s options to ensure every cluster is a unique snowflake.

Below the water – Kubernetes everywhere – Let a thousand K8s bloom

Let’s plunge deeper. Because one Kubernetes cluster is never enough, why not sprinkle them everywhere? On-premises, in the cloud, under your desk—there’s a K8s solution for all that.

Amazon EKS Anywhere – AWS container but in your own data center
Amazon ECS Anywhere – Like EKS Anywhere but ECS
Azure Arc – Hybrid cloud for masochists
Google Anthos – Hybrid cloud something something
AWS Outposts – AWS, now with hardware shipping delays
Azure Red Hat OpenShift – Azure but with a fedora
VMware Tanzu – Because VMware heard you like Kubernetes
Red Hat OpenShift – Fedoras on AWS
IBM Cloud Kubernetes Service – Kubernetes with a touch of enterprise nostalgia
Oracle Container Engine for Kubernetes (OKE) – For the brave souls trusting Oracle
Alibaba Cloud Elastic Container Service (ACK) – K8s with Chinese characteristics
DigitalOcean Kubernetes (DOKS) – K8s in the ocean, hopefully no sharks
Linode Kubernetes Engine (LKE) – Lightweight K8s for the minimalist
Vultr Kubernetes Engine (VKE) – Yet another place to run your pods
Scaleway Kapsule – French elegance meets Kubernetes
Hetzner Cloud Managed Kubernetes – German engineering for your clusters
Nutanix Karbon - Because regular Kubernetes wasn’t converged enough
Metal Stack Cloud Kubernetes - For when virtual machines are too virtual

You might also like:

Middle of the iceberg – Serverless and function containers – Now you see me, now you don’t

As we descend to the depths, we find containers with commitment issues. They appear, do their job, and disappear.

AWS Batch – Containers that only work the night shift
AWS Lambda – Lambda’s containers for when a 250 MB zip file wasn’t enough baggage.
Azure Functions – Roll your own functions in containers and hope for the best.
Google Cloud Functions – Containers pretending to be functions, but now on Google Cloud.
Alibaba Function Compute – You get the idea.

Bottom of the iceberg – Specialized and hybrid platforms – The swiss army knife of container hosting

Down here, containers run in ways that make people question your life choices. These platforms let you host containers in environments many fear to tread.

AWS Proton – Automate your deployments and run a container
Azure Virtual Machines with Docker – When you miss managing servers but love containers
IBM Cloud Code Engine – Run anything, anywhere, if you can figure it out
Alibaba Elastic Container Instance (ECI) – Containers that disappear faster than you can say “Jack Ma”

Below the iceberg – Containers on the edge of reason

In these lightless depths, you want your containers to live life on the edge—literally. Because nothing says “enterprise-ready” like running production workloads on a smart toaster.

AWS IoT Greengrass – Run AWS Lambda and Docker containers on everything from factory equipment to smart fridges. Nothing says “digital transformation” like your smart fridge running machine learning models.
Azure IoT Edge – Like Greengrass, but only when your toaster prefers Windows updates.
AWS Snowball Edge – For when ‘cloud migration’ means paying FedEx to move your containers. Yes, it’s literally a box they mail you that can run containers.
AWS Snowcone – An even smaller box that runs just enough containers to make you appreciate your data center. Perfect for when you need AWS in places where AWS doesn’t want to build a region.
BalenaCloud IoT Device Management – A platform that lets you remotely deploy and manage containers on fleets of Raspberry Pis. Because nothing says “enterprise IoT” like running your production stack on $35 computers.
Fly.io – Global Edge Container Deployment, because apparently just picking one region was too simple.
Edge Node Service (Tencent Cloud) – Edge deployment behind great firewall.
StackPath Edge Computing – More edge containers.
Koyeb – Serverless Edge containers.
Platform.sh – Multi-Cloud Edge containers.
StarlingX - Edge computing platform for when 5G and IoT buzzwords aren’t enough
ZStack Edge - Edge computing with Chinese characteristics, now at the edge of the network, too

The abyss – Where containers fear to tread

And now we reach the darkest depths. For the bravest souls who look at conventional container platforms and think, “Not edgy enough.” Here lie the most unusual ways to run containers in production.

Amazon GameLift – Run your containerized services disguised as game servers. High scores may apply.
AWS Device Farm – Deploy containers on hundreds of actual phones in AWS racks. Perfect for when “mobile-first” goes too far.
AWS AppStream 2.0 – Stream Windows to run Docker to stream containers. It’s virtualization all the way down.
Google Cloud Dataflow – Hide containerized services in pipeline steps named “daily_data_normalize_37”. Hope your SRE enjoys debugging ETL.
Google Cloud Composer – Your containerized microservice masquerades as an Airflow DAG. Because nothing says reliability like cron.
Google Colab & Jupyter Notebooks – Run containers in data_preprocessing_step.ipynb. Watch ML teams debug production.
Azure Quantum – Package quantum containers with az quantum job submit. Now, your uptime is truly uncertain.
Azure Cognitive Services in Containers – Offline AI containers that promise not to become self-aware. Terms and conditions may apply.
Azure DevTest Labs – Run containerized test environments that self-destruct like Mission Impossible tapes. Perfect for when temporary becomes permanent.

Beyond the abyss

There you have it, the container iceberg. And I didn’t even have room to add the extra cursed ways you can run containers. Here is some of what didn’t make it:

Cloud IDEs and Workspaces – Want to turn your development environment into a sneaky container platform? Take your pick: GitHub Codespaces, GitPod for the GitLab enthusiasts, or Eclipse Che for the masochists who miss Eclipse crashing locally. JetBrains Space or Replit? All this lets you run a container until you close your browser tab.
GitHub Actions et al. – Ever wanted to run a production service disguised as a CI job? Just label your production API start-up script as “integration_test_step_42”, add a sleep 21600 and a call to retrigger itself, and hope nobody notices your “test” has been running for months. Apply this same trick to AWS CodeBuild, GitLab CI/CD, CircleCI, Azure Pipelines, Travis CI, Bitbucket Pipelines, Drone CI, Jenkins Cloud, TeamCity Cloud, and Google Cloud Build for Bonus points.

Managing your container zoo with pulumi

Whether you’re deploying containers to mainstream cloud providers or attempting to run them on quantum computers (really, please let us know how that goes), Pulumi helps you manage it all with real code instead of config file soup. Our Kubernetes support means you can wrangle your containers wherever they might roam – from AWS to Azure, GCP to Kubernetes, or even that smart toaster you’re eyeing for production deployment.

With Pulumi, you get:

Cloud-native container orchestration across all major platforms and Kubernetes distributions
Infrastructure testing to ensure your containers deploy correctly (even in the weird places)
Provider templates for when you invent yet another way to run containers
Stack references for managing multiple environments without losing your mind
Automation API for programmatically herding your container fleets

Ready to bring some sanity to your container deployment strategy? Get started with Pulumi:

Try Pulumi for Free

The final word

At this point, 105 options in, you might wonder: “Has cloud container hosting gone too far?” Well, I’ve a solution to propose:

At Pulumi, we’ve got providers for all the usual suspects and, surprisingly, many more in this container circus. And if you’ve figured out how to run containers on a quantum computer inside a data pipeline in New Zealand, you can write a provider for that, too.

So, use Pulumi and we will figure out how RedHat-Tanzu-Greengrass-Icecore-Device-Farm works and provide you a sane interface to it. You’ve got other things to worry about.

Unified and Programmatic Approach to Infrastructure Management at BMW Using Pulumi

Sara Huddleston — Thu, 03 Oct 2024 09:52:10 +0000

In the ever-evolving world of automotive technology, BMW has been at the forefront of innovation, seamlessly integrating software into the heart of their vehicles. As cars become increasingly complex, with a growing emphasis on connectivity, over-the-air upgrades, and brand-specific user experiences, the need for a robust and scalable software development approach has become paramount.

Enter the BMW Software Factory, a platform that aims to empower the company’s developers and provide them with a superior development experience. At the core of this initiative is the adoption of Pulumi, a modern infrastructure as code (IaC) solution that has transformed the way BMW manages its software ecosystem.

On this article:

The Challenges of a Sprawling Software Landscape

BMW’s software journey has been a testament to the exponential growth of automotive technology. What started with simple engine controllers has evolved into a complex network of electronic control units (ECUs) scattered throughout the vehicle. As the software footprint continues to expand, BMW recognized the need for a unified and efficient approach to software development and deployment.

The company’s initial efforts involved introducing a platform called “Code Craft,” which provided a comprehensive stack of services to support the software development lifecycle. This stack included a GitHub Enterprise application for source code management, a Gerrit system for Android-based developments, a continuous integration (CI) pipeline, artifact stores, build caching, and various other tools and services.

However, as the demand for software-driven features grew, the complexity of managing this sprawling ecosystem became increasingly challenging. BMW found itself grappling with the need to scale its infrastructure, navigate network limitations across multiple data centers, and adapt to the ever-changing landscape of cloud computing.

The Evolution of BMW’s Software Development Toolchain

BMW’s journey to streamline its software development process has been gradual and iterative. The company’s initial approach involved using Ansible for deployment and a custom-built deployment scripting solution for its OpenShift cluster.

As the complexity of the platform increased, BMW turned to Helm and Kubernetes to manage its containerized services. However, as the company ventured into the public cloud, the limitations of these tools became apparent. The team recognized the need for a more comprehensive and scalable solution to manage their infrastructure as code.

At this critical juncture, BMW discovered Pulumi. This modern IaC solution offered a unique advantage – the ability to leverage a full-fledged programming language, Python, to define and manage their infrastructure. This shift proved to be a game-changer, allowing BMW to leverage its expertise in Python and benefit from the rich ecosystem of libraries and tools available in the Python community.

Embracing Pulumi: Streamlining Infrastructure Management

BMW’s adoption of Pulumi was a strategic move that aimed to address the growing complexity of its software ecosystem. By transitioning from a patchwork of tools to a unified IaC solution, the company was able to streamline its infrastructure management and improve developer productivity.

Shared Modules: Promoting Reusability and Best Practices

One of BMW’s key initiatives was developing a shared modules library, which allowed the team to abstract the complexity of various infrastructure components, such as databases, and provide a consistent and user-friendly interface for their developers.

By leveraging Pulumi’s Python bindings, BMW was able to create reusable modules that encapsulated best practices and sensible defaults, making it easier for developers to provision and manage infrastructure resources. BMW used Pydantic, a data validation library, to define schema-based configurations for its infrastructure. These were then integrated into its IDEs, providing developers with auto-completion and validation support.
This approach not only accelerated the development process but also ensured that the infrastructure deployed across the organization adhered to consistent security and compliance standards.

Transformation Features: Protecting Legacy Services

As part of their software factory, BMW also faced the challenge of integrating legacy services that did not natively support modern authentication and authorization mechanisms, such as OpenID Connect (OIDC).
To address this, BMW leveraged Pulumi’s transformation features to seamlessly inject an OAuth2 proxy into their deployments, providing a secure and consistent way to protect these services without requiring extensive modifications to the underlying applications.
By encapsulating this functionality within a shared module, BMW was able to apply the OAuth2 proxy to multiple services, ensuring a consistent and secure access control layer across their software ecosystem.

Policy Enforcement: Ensuring Compliance and Security

One key benefit of Pulumi’s IaC approach is the ability to define and enforce policies across the organization, ensuring that infrastructure deployments adhere to security and compliance standards. BMW has leveraged Pulumi’s policy-as-code capabilities to implement mandatory checks, such as ensuring that all S3 buckets are encrypted at rest, preventing the deployment of non-compliant resources.

By integrating these policy checks into their deployment workflows, BMW has shifted security and compliance concerns to the left, addressing issues early in the development process and reducing the risk of costly post-deployment remediations.

The Benefits of Pulumi: Accelerating Development and Improving Maintainability

BMW’s adoption of Pulumi has yielded significant benefits, transforming the way the company approaches software development and infrastructure management.

Accelerated Development with Shared Modules

The implementation of shared modules has been a game-changer for BMW. It allows developers to leverage pre-built and tested infrastructure components without having to reinvent the wheel. This has resulted in a significant acceleration of the development process, as teams can focus on building their applications rather than grappling with the complexities of infrastructure provisioning.

Improved Maintainability and Consistency

By centralizing infrastructure management within the shared modules, BMW has ensured that best practices and security standards are consistently applied across the organization. This has not only improved the overall maintainability of the software ecosystem but has also reduced the risk of security and compliance violations.

Leveraging Python’s Ecosystem

BMW’s decision to leverage Pulumi’s Python bindings has been a strategic advantage, as the company was able to tap into the rich ecosystem of Python libraries and tools. This has enabled the team to seamlessly integrate Pulumi with their existing Python-based toolchain, including dependency management, testing frameworks, and code quality tools, further enhancing the development experience.

Streamlined Cloud Migration

As BMW continues to expand its use of public cloud services, Pulumi has played a crucial role in simplifying the migration process. By providing a consistent IaC approach across on-premises and cloud environments, Pulumi has enabled BMW to manage its infrastructure in a unified manner, reducing the complexity and overhead associated with multi-cloud deployments.

The Future of BMW’s Software Factory: Embracing the Cloud

Looking ahead, BMW’s Software Factory is poised to take the next step in its evolution, with plans to transition away from the self-hosted backend and embrace cloud-native services. This strategic move aims to further improve developer productivity and reduce the internal effort required to maintain the underlying infrastructure.

By leveraging the expertise and reliability of cloud service providers, BMW can focus on delivering innovative software features to their customers while the cloud providers handle the operational aspects of running the necessary services.

As BMW continues to push the boundaries of automotive software, Pulumi’s role in its Software Factory will only become more crucial. By providing a scalable and flexible IaC solution, Pulumi empowers BMW’s developers to innovate with confidence and be secure in the knowledge that their infrastructure is managed consistently and in alignment with the company’s security and compliance standards.

Conclusion: Unlocking the Future of Automotive Software

BMW’s journey with Pulumi in the Software Factory showcases the power of modern IaC solutions in navigating the complexities of the automotive software landscape. By embracing a unified and programmatic approach to infrastructure management, BMW has accelerated development, improved maintainability, and ensured compliance across its sprawling software ecosystem.

As the automotive industry continues to evolve, with cars becoming increasingly software-driven, the lessons learned by BMW can serve as a blueprint for other organizations looking to streamline their software development and deployment processes. By leveraging the capabilities of Pulumi and other cutting-edge technologies, the future of automotive software is poised to be more efficient, secure, and responsive to the ever-changing needs of both manufacturers and consumers.

To learn more about Pulumi and how it can transform your software development and infrastructure management:

Get started with Pulumi Tutorials
Attend an upcoming workshop
Try out the Pulumi Neo code assistant to accelerate your infrastructure as code journey

Introducing The New Docker Build Provider

Bryce Lampe — Thu, 25 Apr 2024 09:00:00 -0700

Deploying and managing containerized workloads is one of Pulumi’s fastest-growing areas. Standing up managed container services and Kubernetes clusters is a common area for automation, and many of our customers use Pulumi to automate building and publishing images to their registry of choice.

Given this critical use case, we are thrilled to introduce the latest addition to the Pulumi ecosystem: the new Docker Build provider, designed to streamline and modernize Docker image builds from your Pulumi programs.

This new addition expands upon the solid foundation of our highly utilized Docker provider. It elevates the docker.Image resource, one of the most heavily used and most powerful resources in the Pulumi ecosystem, into its very own dedicated package. By fully embracing the power of BuildKit, the Docker Build provider brings many new features to support the best-in-class Docker capabilities developers love.

The new provider exposes Docker’s next-gen buildx interface, so now even your most complex Docker images can be handled by Pulumi. This is a remarkably flexible and versatile tool for building images and includes key features such as,

Multi-platform image support: Build images that run seamlessly across different hardware architectures.
Advanced caching mechanisms: Speed up builds and reduce resource consumption by utilizing cache backends like S3, GitHub Actions, local disk, and more.
Support for build secrets: Unlike building images with the docker CLI, which requires juggling environment variables or files on disk, Pulumi’s first-class support for secrets means you can now safely and efficiently incorporate sensitive information into your builds. With Pulumi ESC (Environments, Secrets, and Configuration), it’s easy to share build secrets with other developers and teams.
Support for multiple export types: Export your images to registries, disk, or blob storage to increase the power and flexibility of your workflows.
Support for Docker Build Cloud: Use Docker Build Cloud to offload your builds and caches to the cloud, enhancing productivity and performance.

A brief history

Pulumi first introduced the Docker provider in 2018, and it has been widely adopted since. Over the years, Pulumi has invested in the Docker provider, notably introducing version 4.0 last year and additional improvements like build-on-preview behavior.

At the same time, the Docker build ecosystem has seen tremendous advancements in how images can be built and distributed. These changes were initially experimental, but they became official when BuildKit graduated to become the default builder in Docker version 23.

We have heard many requests from users to expose more BuildKit functionality in the Docker provider. After some consideration, we decided that a new, standalone provider focused exclusively on building images would provide the best overall user experience. In addition, it allows Pulumi to stay current with the evolving Docker build landscape.

Getting started

Add the docker-build package to your Pulumi program to take full advantage of modern image builds.

import * as docker_build from "@pulumi/docker-build";

import pulumi_docker_build as docker_build

using DockerBuild = Pulumi.DockerBuild;

import "github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild"

resources:
 my-image:
 type: docker-build:Image

import com.pulumi.dockerbuild.Image

Suppose your Pulumi program has a subfolder called ./app that contains a file named Dockerfile (its contents can be as simple as FROM alpine). In that case, the example below shows:

how to build a multi-platform image (linux/amd64 and linux/arm64),
publish the image to a remote AWS ECR registry, and;
use an inline cache to speed up subsequent builds.

import * as pulumi from "@pulumi/pulumi"
import * as aws from "@pulumi/aws";
import * as docker_build from "@pulumi/docker-build";

// Create an ECR repository for pushing.
const ecrRepository = new aws.ecr.Repository("ecr-repository", {});

// Grab auth credentials for ECR.
const authToken = aws.ecr.getAuthorizationTokenOutput({
 registryId: ecrRepository.registryId,
});

// Build and push an image to ECR with inline caching.
const myImage = new docker_build.Image("my-image", {
 // Tag our image with our ECR repository's address.
 tags: [pulumi.interpolate`${ecrRepository.repositoryUrl}:latest`],
 context: {
 location: "./app",
 },
 // Use the pushed image as a cache source.
 cacheFrom: [{
 registry: {
 ref: pulumi.interpolate`${ecrRepository.repositoryUrl}:latest`,
 },
 }],
 // Include an inline cache with our pushed image.
 cacheTo: [{
 inline: {},
 }],
 // Build a multi-platform image manifest for ARM and AMD.
 platforms: [
 "linux/amd64",
 "linux/arm64",
 ],
 // Push the final result to ECR.
 push: true,
 // Provide our ECR credentials.
 registries: [{
 address: ecrRepository.repositoryUrl,
 password: authToken.password,
 username: authToken.userName,
 }],
});

// Export a ref for the pushed images so we can deploy it.
export const ref = myImage.ref;

import pulumi
import pulumi_aws as aws
import pulumi_docker_build as docker_build

# Create an ECR repository for pushing.
ecr_repository = aws.ecr.Repository("ecr-repository")

# Grab auth credentials for ECR.
auth_token = aws.ecr.get_authorization_token_output(registry_id=ecr_repository.registry_id)

# Build and push an image to ECR with inline caching.
my_image = docker_build.Image("my-image",
 # Tag our image with our ECR repository's address.
 tags=[ecr_repository.repository_url.apply(lambda repository_url: f"{repository_url}:latest")],
 context=docker_build.BuildContextArgs(
 location="./app",
 ),
 # Use the pushed image as a cache source.
 cache_from=[docker_build.CacheFromArgs(
 registry=docker_build.CacheFromRegistryArgs(
 ref=ecr_repository.repository_url.apply(lambda repository_url: f"{repository_url}:latest"),
 ),
 )],
 # Include an inline cache with our pushed image.
 cache_to=[docker_build.CacheToArgs(
 inline=docker_build.CacheToInlineArgs(),
 )],
 # Build a multi-platform image manifest for ARM and AMD.
 platforms=[
 docker_build.Platform.LINUX_AMD64,
 docker_build.Platform.LINUX_ARM64,
 ],
 # Push the final result to ECR.
 push=True,
 # Provide our ECR credentials.
 registries=[docker_build.RegistryArgs(
 address=ecr_repository.repository_url,
 password=auth_token.password,
 username=auth_token.user_name,
 )],
)

# Export a ref for the pushed images so we can deploy it.
pulumi.export("ref", my_image.ref)

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;
using DockerBuild = Pulumi.DockerBuild;

return await Deployment.RunAsync(() =>
{
 // Create an ECR repository for pushing.
 var ecrRepository = new Aws.Ecr.Repository("ecr-repository");

 // Grab auth credentials for ECR.
 var authToken = Aws.Ecr.GetAuthorizationToken.Invoke(new()
 {
 RegistryId = ecrRepository.RegistryId,
 });

 // Build and push an image to ECR with inline caching.
 var myImage = new DockerBuild.Image("my-image", new()
 {
 // Tag our image with our ECR repository's address.
 Tags = new[]
 {
 ecrRepository.RepositoryUrl.Apply(repositoryUrl => $"{repositoryUrl}:latest"),
 },
 Context = new DockerBuild.Inputs.BuildContextArgs
 {
 Location = "./app",
 },
 // Use the pushed image as a cache source.
 CacheFrom = new[]
 {
 new DockerBuild.Inputs.CacheFromArgs
 {
 Registry = new DockerBuild.Inputs.CacheFromRegistryArgs
 {
 Ref = ecrRepository.RepositoryUrl.Apply(repositoryUrl => $"{repositoryUrl}:latest"),
 },
 },
 },
 // Include an inline cache with our pushed image.
 CacheTo = new[]
 {
 new DockerBuild.Inputs.CacheToArgs
 {
 Inline = null,
 },
 },
 // Build a multi-platform image manifest for ARM and AMD.
 Platforms = new[]
 {
 DockerBuild.Platform.Linux_amd64,
 DockerBuild.Platform.Linux_arm64,
 },
 // Push the final result to ECR.
 Push = true,
 // Provide our ECR credentials.
 Registries = new[]
 {
 new DockerBuild.Inputs.RegistryArgs
 {
 Address = ecrRepository.RepositoryUrl,
 Password = authToken.Apply(getAuthorizationTokenResult => getAuthorizationTokenResult.Password),
 Username = authToken.Apply(getAuthorizationTokenResult => getAuthorizationTokenResult.UserName),
 },
 },
 });

 // Export a ref for the pushed images so we can deploy it.
 return new Dictionary<string, object?>
 {
 ["ref"] = myImage.Ref,
 };
});

package main

import (
 "fmt"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ecr"
 "github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an ECR repository for pushing.
 ecrRepository, err := ecr.NewRepository(ctx, "ecr-repository", nil)
 if err != nil {
 return err
 }

 // Grab auth credentials for ECR.
 authToken := ecr.GetAuthorizationTokenOutput(ctx, ecr.GetAuthorizationTokenOutputArgs{
 RegistryId: ecrRepository.RegistryId,
 }, nil)

 // Build and push an image to ECR with inline caching.
 myImage, err := dockerbuild.NewImage(ctx, "my-image", &dockerbuild.ImageArgs{
 // Tag our image with our ECR repository's address.
 Tags: pulumi.StringArray{
 ecrRepository.RepositoryUrl.ApplyT(func(repositoryUrl string) (string, error) {
 return fmt.Sprintf("%v:latest", repositoryUrl), nil
 }).(pulumi.StringOutput),
 },
 Context: &dockerbuild.BuildContextArgs{
 Location: pulumi.String("./app"),
 },
 // Use the pushed image as a cache source.
 CacheFrom: dockerbuild.CacheFromArray{
 &dockerbuild.CacheFromArgs{
 Registry: &dockerbuild.CacheFromRegistryArgs{
 Ref: ecrRepository.RepositoryUrl.ApplyT(func(repositoryUrl string) (string, error) {
 return fmt.Sprintf("%v:latest", repositoryUrl), nil
 }).(pulumi.StringOutput),
 },
 },
 },
 // Include an inline cache with our pushed image.
 CacheTo: dockerbuild.CacheToArray{
 &dockerbuild.CacheToArgs{
 Inline: nil,
 },
 },
 // Build a multi-platform image manifest for ARM and AMD.
 Platforms: dockerbuild.PlatformArray{
 dockerbuild.Platform_Linux_amd64,
 dockerbuild.Platform_Linux_arm64,
 },
 // Push the final result to ECR.
 Push: pulumi.Bool(true),
 // Provide our ECR credentials.
 Registries: dockerbuild.RegistryArray{
 &dockerbuild.RegistryArgs{
 Address: ecrRepository.RepositoryUrl,
 Password: authToken.ApplyT(func(authToken ecr.GetAuthorizationTokenResult) (*string, error) {
 return &authToken.Password, nil
 }).(pulumi.StringPtrOutput),
 Username: authToken.ApplyT(func(authToken ecr.GetAuthorizationTokenResult) (*string, error) {
 return &authToken.UserName, nil
 }).(pulumi.StringPtrOutput),
 },
 },
 })
 if err != nil {
 return err
 }

 // Export a ref for the pushed images so we can deploy it.
 ctx.Export("ref", myImage.Ref)
 return nil
 })
}

description: Push to AWS ECR with caching
name: ecr
outputs:
 ref: ${my-image.ref}
resources:
 # Create an ECR repository for pushing.
 ecr-repository:
 type: aws:ecr:Repository

 # Build and push an image to ECR with inline caching.
 my-image:
 type: docker-build:Image
 properties:
 # Tag our image with our ECR repository's address.
 tags:
 - ${ecr-repository.repositoryUrl}:latest
 context:
 location: ./app
 # Use the pushed image as a cache source.
 cacheFrom:
 - registry:
 ref: ${ecr-repository.repositoryUrl}:latest
 # Include an inline cache with our pushed image.
 cacheTo:
 - inline: {}
 # Build a multi-platform image manifest for ARM and AMD.
 platforms:
 - linux/amd64
 - linux/arm64
 # Push the final result to ECR.
 push: true
 # Provide our ECR credentials.
 registries:
 - address: ${ecr-repository.repositoryUrl}
 password: ${auth-token.password}
 username: ${auth-token.userName}

runtime: yaml
variables:
 auth-token:
 # Grab auth credentials for ECR.
 fn::aws:ecr:getAuthorizationToken:
 registryId: ${ecr-repository.registryId}

package myapp;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.ecr.Repository;
import com.pulumi.aws.ecr.EcrFunctions;
import com.pulumi.aws.ecr.inputs.GetAuthorizationTokenArgs;
import com.pulumi.dockerbuild.Image;
import com.pulumi.dockerbuild.ImageArgs;
import com.pulumi.dockerbuild.inputs.CacheFromArgs;
import com.pulumi.dockerbuild.inputs.CacheFromRegistryArgs;
import com.pulumi.dockerbuild.inputs.CacheToArgs;
import com.pulumi.dockerbuild.inputs.CacheToInlineArgs;
import com.pulumi.dockerbuild.inputs.BuildContextArgs;
import com.pulumi.dockerbuild.inputs.RegistryArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 // Create an ECR repository for pushing.
 var ecrRepository = new Repository("ecrRepository");

 // Grab auth credentials for ECR.
 final var authToken = EcrFunctions.getAuthorizationToken(GetAuthorizationTokenArgs.builder()
 .registryId(ecrRepository.registryId())
 .build());

 // Build and push an image to ECR with inline caching.
 var myImage = new Image("myImage", ImageArgs.builder()
 // Tag our image with our ECR repository's address.
 .tags(ecrRepository.repositoryUrl().applyValue(repositoryUrl -> String.format("%s:latest", repositoryUrl)))
 .context(BuildContextArgs.builder()
 .location("./app")
 .build())
 // Use the pushed image as a cache source.
 .cacheFrom(CacheFromArgs.builder()
 .registry(CacheFromRegistryArgs.builder()
 .ref(ecrRepository.repositoryUrl().applyValue(repositoryUrl -> String.format("%s:latest", repositoryUrl)))
 .build())
 .build())
 // Include an inline cache with our pushed image.
 .cacheTo(CacheToArgs.builder()
 .inline()
 .build())
 // Build a multi-platform image manifest for ARM and AMD.
 .platforms(
 "linux/amd64",
 "linux/arm64")
 // Push the final result to ECR.
 .push(true)
 // Provide our ECR credentials.
 .registries(RegistryArgs.builder()
 .address(ecrRepository.repositoryUrl())
 .password(authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult).applyValue(authToken -> authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult.password())))
 .username(authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult).applyValue(authToken -> authToken.applyValue(getAuthorizationTokenResult -> getAuthorizationTokenResult.userName())))
 .build())
 .build());

 ctx.export("ref", myImage.ref());
 }
}

The Pulumi output, ref, provides a convenient way to reference the pushed image in downstream Pulumi resources like ECS TaskDefinitions or Kubernetes Deployments.

Build container images with Pulumi

Use the Docker Build provider to build multi-platform images, cache layers across builds, and push to your registry of choice as part of your Pulumi program.

Get started

Migrating from docker.Image to docker_build.Image

The new docker_build.Image replaces the old docker.Image resource. It adds new features and addresses many bugs with the previous resource by fully aligning with Docker’s underlying buildx commands. From now on, we will continue to invest in Docker Build and the docker_build.Image resource.

We do not expect to make further changes to docker.Image. It will remain available for the foreseeable future, so you can continue using it if it meets your needs. However, we recommend you migrate your images to docker_build.Image to get the best possible support, features, and performance. We expect to deprecate the docker.Image resource in the future.

The migration process is straightforward and detailed in the API documentation. Docker Build provides a superset functionality over the previous docker.Image resource; thus, you can migrate existing resources without issue. The new Docker Build Image options will look very familiar if you use the Docker command-line tool.

The new Docker Build provider matches the Docker CLI behavior. Thus, it does not push images by default. If you want to push to a registry, include push: true in the code just as you would include --push on the command line.

The new Docker Build provider builds images by default during previews to reduce the risk of merging broken images as part of CI pipelines. You can change the default behavior by specifying a buildOnPreview flag.

Jump over to the applicable migration guide to learn how to update your existing Pulumi program:

Migrating from Docker provider v3
Migrating from Docker provider v4

Migrating from Docker provider v3

Version 3.x of the Docker provider is still widely used because it exposes BuildKit functionality through raw command-line arguments. However, the new Docker Build provider exposes those arguments as top-level fields on the resource.

Reference the below TypeScript code to compare a v3 definition of a docker.Image with the new dockerbuild.Image. In particular, note the extraOptions in v3 are now top-level fields, e.g., cacheFrom, cacheTo, platforms, etc.

// v3 Image
const v3 = new docker.Image("v3-image", {
 imageName: "myregistry.com/user/repo:latest",
 localImageName: "local-tag",
 skipPush: false,
 build: {
 dockerfile: "./Dockerfile",
 context: "../app",
 target: "mytarget",
 args: {
 MY_BUILD_ARG: "foo",
 },
 env: {
 DOCKER_BUILDKIT: "1",
 },
 extraOptions: [
 "--cache-from",
 "type=registry,myregistry.com/user/repo:cache",
 "--cache-to",
 "type=registry,myregistry.com/user/repo:cache",
 "--add-host",
 "metadata.google.internal:169.254.169.254",
 "--secret",
 "id=mysecret,src=https://www.pulumi.com/local/secret",
 "--ssh",
 "default=/home/runner/.ssh/id_ed25519",
 "--network",
 "host",
 "--platform",
 "linux/amd64",
 ],
 },
 registry: {
 server: "myregistry.com",
 username: "username",
 password: pulumi.secret("password"),
 },
});

// v3 Image after migrating to docker-build.Image
const v3Migrated = new dockerbuild.Image("v3-to-buildx", {
 tags: ["myregistry.com/user/repo:latest", "local-tag"],
 push: true,
 dockerfile: {
 location: "./Dockerfile",
 },
 context: {
 location: "../app",
 },
 target: "mytarget",
 buildArgs: {
 MY_BUILD_ARG: "foo",
 },
 cacheFrom: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
 cacheTo: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
 secrets: {
 mysecret: "value",
 },
 addHosts: ["metadata.google.internal:169.254.169.254"],
 ssh: {
 default: ["/home/runner/.ssh/id_ed25519"],
 },
 network: "host",
 platforms: ["linux/amd64"],
 registries: [{
 address: "myregistry.com",
 username: "username",
 password: pulumi.secret("password"),
 }],
});

Migrating from Docker provider v4

The new Docker Build provider largely exposes the same fields as v4, but the fields are pluralized or renamed to better align with the Docker CLI.

Reference the below TypeScript code to compare a v4 definition of a docker.Image with the new dockerbuild.Image. In particular, note the top-level, pluralized names, e.g., registries, platforms, etc.

// v4 Image
const v4 = new docker.Image("v4-image", {
 imageName: "myregistry.com/user/repo:latest",
 skipPush: false,
 build: {
 dockerfile: "./Dockerfile",
 context: "../app",
 target: "mytarget",
 args: {
 MY_BUILD_ARG: "foo",
 },
 cacheFrom: {
 images: ["myregistry.com/user/repo:cache"],
 },
 addHosts: ["metadata.google.internal:169.254.169.254"],
 network: "host",
 platform: "linux/amd64",
 },
 buildOnPreview: true,
 registry: {
 server: "myregistry.com",
 username: "username",
 password: pulumi.secret("password"),
 },
});

// v4 Image after migrating to docker-build.Image
const v4Migrated = new dockerbuild.Image("v4-to-buildx", {
 tags: ["myregistry.com/user/repo:latest"],
 push: true,
 dockerfile: {
 location: "./Dockerfile",
 },
 context: {
 location: "../app",
 },
 target: "mytarget",
 buildArgs: {
 MY_BUILD_ARG: "foo",
 },
 cacheFrom: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
 cacheTo: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
 addHosts: ["metadata.google.internal:169.254.169.254"],
 network: "host",
 platforms: ["linux/amd64"],
 registries: [{
 address: "myregistry.com",
 username: "username",
 password: pulumi.secret("password"),
 }],
});

Conclusion

Launching the new Docker Build provider marks a significant milestone in modernizing container management for developers. The new package, which contains the docker_build.Image resource reflects our commitment to innovation and user-centric development. By incorporating the latest BuildKit technology, the Docker Build provider supports the newest features, enhances multi-architecture builds, and improves caching strategies. Moreover, its integration with Docker Build Cloud underscores our dedication to providing a cutting-edge toolset for developers. This launch is a testament to our ongoing efforts to modernize and improve how developers build images.

Check out the Pulumi Docker Build documentation for more details on utilizing the Docker Build provider.

If you are still wondering, “Which provider should I use?” the answer depends on whether you are doing anything related to docker build.

Provider	Use cases
`@pulumi/docker-build`	Use this provider to build and push Docker images. Ideal for scenarios requiring automation of image creation with Docker, including setting up CI/CD pipelines for Docker image generation.
`@pulumi/docker`	Use this provider for everything else, including managing local Docker containers, networks, and volumes on the host machine.

Fast Docker Image Builds with Pulumi

Monica Rodriguez — Wed, 08 Mar 2023 14:00:00 -0800

How do I speed up Docker image builds with Pulumi? Use BuildKit (the default since Docker 23), enable a registry or layer cache so repeated builds reuse work, write a multi-stage Dockerfile so production images skip build-time dependencies, and reach for the dedicated Docker Build provider when you need buildx features like multi-platform images, build secrets, or Docker Build Cloud. With these techniques together, repeat builds in a Pulumi program commonly drop from minutes to seconds.

TL;DR

BuildKit is the default in current Docker releases and in the Pulumi Docker providers — you generally do not need to opt in.
Cache your layers somewhere persistent. A registry cache (cacheFrom / cacheTo) is what makes CI builds fast across runners and contributors.
Multi-stage Dockerfiles are the single biggest win for image size and rebuild speed when you have a compile or bundle step.
Use pulumi/docker-build for new projects. It exposes the full buildx surface (multi-platform, secrets, multiple cache backends, Docker Build Cloud).
Use pulumi/docker v4+ if you already depend on it — its docker.Image resource also rebuilds only on context change and supports BuildKit and registry caching.

What actually moves the needle

The numbers below are representative of what teams report from a typical Node.js or Python web app (single Dockerfile, ~300 MB final image). Your mileage varies with project size, network, and runner specs — treat this as a directional comparison, not a benchmark suite.

Configuration	Cold build	Repeat build (no source change)	Repeat build (small source change)
Legacy builder, no caching	minutes	minutes	minutes
BuildKit, local layer cache only	minutes	seconds (cache hit)	tens of seconds
BuildKit + multi-stage Dockerfile	minutes	seconds (cache hit)	tens of seconds
BuildKit + multi-stage + registry cache (`cacheFrom`/`cacheTo`)	minutes (first run)	seconds (cache hit)	seconds–tens of seconds, even on a fresh CI runner
`docker-build` provider with buildx + Docker Build Cloud	under 2 min	seconds (cache hit)	seconds

The headline pattern: each row past the first is dominated by what the build can skip, not by raw CPU. Pulumi’s job is to wire those caches up declaratively so they keep working across every pulumi up.

How does Pulumi build Docker images?

Pulumi has two providers for image builds, and which one you should use depends on how new your project is.

The Docker provider (pulumi/docker) is the original. Its docker.Image resource can build a local context and push to any registry as part of a Pulumi deployment, alongside resources for containers, networks, and volumes. Since v4 it rebuilds only when the build context changes and uses BuildKit by default.
The Docker Build provider (pulumi/docker-build) is a newer, dedicated provider focused exclusively on image builds. It exposes Docker’s buildx interface, so you get multi-platform builds, multiple cache backends, build secrets, and Docker Build Cloud as first-class features.

For new projects, prefer docker-build. For existing programs that already use docker.Image, you do not have to migrate — docker.Image continues to receive the speed and caching improvements covered below.

Why are my Docker builds slow?

Three things make Docker builds slow in practice, and Pulumi addresses each one declaratively.

No cache, or the wrong cache. A fresh CI runner has no local Docker layers. Without a remote cache, every build reinstalls every dependency.
Cache busting on noise. If your Dockerfile copies package.json and the rest of your source in the same step, every source change invalidates the dependency-install layer. Order layers from least-changing to most-changing.
Building dev tooling into the production image. Compilers, linters, and test frameworks have no business in a runtime image. Multi-stage builds let you discard them.

How do I enable BuildKit and registry caching in Pulumi?

BuildKit has been the default builder since Docker 23 and is the default in both Pulumi providers. The piece you usually have to configure is where the cache lives — local disk on a developer laptop is fine, but for CI you almost always want a registry cache so layers persist across runners.

With the Docker Build provider (recommended)

import * as docker_build from "@pulumi/docker-build";

const image = new docker_build.Image("my-image", {
 tags: ["docker.io/pulumibot/demo-image:latest"],
 context: { location: "./app" },
 dockerfile: { location: "./app/Dockerfile" },
 platforms: [
 "linux/amd64",
 "linux/arm64",
 ],
 cacheFrom: [{ registry: { ref: "docker.io/pulumibot/demo-image:cache" } }],
 cacheTo: [{ registry: { ref: "docker.io/pulumibot/demo-image:cache", mode: "max" } }],
 push: true,
});

cacheFrom and cacheTo together are the important pair: the first run populates the cache, every later run on any machine pulls it. mode: "max" exports cache for every layer, not just the final ones.

With the Docker provider (`docker.Image`)

import * as docker from "@pulumi/docker";

const image = new docker.Image("my-image", {
 imageName: "docker.io/pulumibot/demo-image:latest",
 build: {
 context: "app",
 dockerfile: "Dockerfile",
 platform: "linux/amd64",
 cacheFrom: {
 images: ["docker.io/pulumibot/demo-image:cache-base"],
 },
 },
});

The docker.Image resource accepts a list of cache images via cacheFrom.images. Tag a previous build with a stable cache tag (for example, :cache-base) and reference it here so subsequent builds reuse its layers.

How do I write a Dockerfile that builds fast?

The Dockerfile matters at least as much as the Pulumi code wrapping it. Two patterns do most of the work.

Order layers least-volatile first. Copy lockfiles and run dependency installs before copying source code. That way changing a .ts file does not bust the npm ci layer.

# Dependencies — only invalidates when lockfiles change
COPY package.json package-lock.json ./
RUN npm ci

# Source — changes constantly, but the dependency layer above survives
COPY . .
RUN npm run build

Use multi-stage builds to drop tooling. Compile in a fat image, copy the artifact into a slim one. The runtime image stays small and the build stage gets cached independently.

FROM node:20 AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-slim AS runtime
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]

When you combine multi-stage with a registry cache (cacheTo mode max), CI runs on fresh machines reuse both stages — the row in the table above where small source changes finish in 10–30 seconds.

How do I build multi-platform images with Pulumi?

Use the docker-build provider and pass a list to platforms. BuildKit will build all variants in parallel and push a manifest list, so a single tag serves whatever architecture the puller needs.

const image = new docker_build.Image("api", {
 tags: ["ghcr.io/example/api:latest"],
 context: { location: "./api" },
 platforms: [
 "linux/amd64",
 "linux/arm64",
 ],
 push: true,
});

If you are still on docker.Image, the build.platform field accepts a single target (for example, "linux/amd64"); for true multi-arch manifests, prefer docker-build.

How do I see Docker build output in Pulumi?

Filtered Docker build and push logs render directly in pulumi up output for both providers, so you can debug a failing layer without leaving your Pulumi run. With the Docker Build provider, structured progress shows per-stage timing — useful when you are trying to figure out which layer is the slow one.

How do I migrate an older `docker.Image` program?

If you are upgrading from a pre-v4 program, pulumi-docker v4 reshaped the docker.Image resource and renamed several supporting types, but does not manage backend state, so updating the package version and adjusting types is generally enough. The full migration guide lives in the v4.0.0 release notes.

If you are starting fresh or want the full BuildKit feature set, install pulumi-docker-build instead and follow the Docker Build provider getting started guide.

Getting started

Docker Build provider — recommended for new projects.
Docker provider — the original, including docker.Image, containers, and networks.
Introducing the Docker Build provider — deeper dive on buildx, secrets, and Docker Build Cloud.

The cheapest performance win in most Pulumi programs is wiring up a registry cache and a multi-stage Dockerfile. Start there, then reach for buildx features as the project grows.

AWS Enterprise Container Management with Pulumi

Isaac Harris — Thu, 14 Apr 2022 00:00:00 +0000

Managing containers and Kubernetes clusters are consistently popular topic areas on the Pulumi blog and in our docs. Our customers regularly cite that Pulumi simplifies container management scenarios, making it the primary reason for choosing Pulumi to define, deploy and manage all of their cloud resources. This includes teams that are just starting their cloud journey and spinning up their first project, as well as teams that want to modernize their apps and services with cloud-native architectures or even scale from one to many clouds.

We’re excited to be launch partners for the new Enterprise Container Management category of the AWS Container Competency program because it perfectly encapsulates Pulumi’s capabilities across the entire lifecycle of container-based architectures and applications. These scenarios are critical to the success of every cloud engineering and DevOps team and include provisioning, governance, security, and observability across Amazon Elastic Kubernetes Service (EKS) and Amazon Elastic Container Service (ECS). Here is a handy guide to some of the features that make Pulumi a great Enterprise Container Management Solution and how Pulumi makes each container scenario easier for your team.

Provisioning Container Infrastructure

Pulumi manages multi-region, multi-account, multi-cluster Kubernetes deployments with ease on any cloud. With Pulumi Crosswalk for Kubernetes we’ve provided a set of Day 0 and Day 1 playbooks that take the guesswork out of the provisioning process from creating a control plane to deploying your apps and updating workers.

Managing AWS Container Services

Pulumi gives you the flexibility to pick the container services that meet the needs of your workloads and the requirements of your organization. Choosing the right services can be a challenge, so we’ve assembled a short overview to help you get started managing AWS containers. Once you’ve chosen your scheduler, Pulumi has a host of examples to guide you through deploying Amazon ECS and Amazon EKS. There is also a Pulumi EKS component that provides multi-language convenience functions and boilerplate to simplify EKS deployments.

Support for the Entire Kubernetes API

Once you’ve provisioned a cluster, the Pulumi Kubernetes Provider enables you to provision any resource available in the Kubernetes API. This provides native support for features like Helm as well as the ability to configure Namespacing providing powerful multi-tenancy capabilities for your clusters.

Cluster Observability

Logging and monitoring are critical capabilities for keeping tabs on the health and security of your clusters. Fortunately, Pulumi has a host of integrations that simplify these scenarios including open-source tools like metrics-server and Prometheus as well as support for industry-leading platforms like Amazon CloudWatch and Datadog.

Authentication and Authorization

Adhering to the principle of least privilege for users and roles is an important step in securing your clusters and Pulumi has many capabilities to help you manage the complexities of authentication and authorization. For example, with Pulumi you can create and manage AWS IAM roles and you can integrate Pulumi Enterprise with your centralized identity and access management platform of choice via SAML 2.0 and OIDC.

Centralized Governance and Compliance Controls

Many customers are using Pulumi and Kubernetes to stand up shared services platforms (SSP) to empower their developers to self-service new infrastructure environments. To keep these environments compliant with internal policies, Pulumi Business Critical Edition includes CrossGuard policy-as-code capabilities built-in. This helps operators to ensure that configuration mistakes won’t reach production with policies that are enforced organization-wide.

Support for Hybrid Deployments

Many users need to manage container-based workloads on AWS as well as on-prem and on other clouds. Fortunately, Pulumi has built-in support for Amazon ECS Anywhere, so teams can use the familiar ECS control plane regardless of where their workloads need to run. Pulumi also supports the EKS Distro which brings the familiar managed Kubernetes capabilities of EKS to on-prem clusters.

Automated Deployment of Infrastructure and Applications

One of the benefits of Pulumi is that it enables infrastructure and application development to leverage the same tooling and processes familiar to software engineers. Adding infrastructure to your CI/CD workflow is easy with Pulumi because it supports a wide variety of test frameworks, simplifies the process of integration testing using ephemeral environments, and includes a CI/CD Integration Assistant to guide you through the process of connecting Pulumi to popular platforms such as GitHub Actions, AWS Code Services and many more.

Once you have your cluster provisioned, you can also deploy workloads into your cluster with GitOps style workflows using the Pulumi Kubernetes Operator.

Conclusion

This is just a sample of the many ways that Pulumi makes it easier than ever to manage containers and Kubernetes clusters in enterprise scenarios. Give it a try for yourself and let us know what you think in our Community Slack.

Azure Container Services Compared: ACI vs App Service vs AKS

Lee Briggs — Tue, 12 Oct 2021 08:00:00 -0700

Containers have emerged as one of the de facto standards for running software. When adopted with the right mindset, they can drastically improve the development lifecycle and help to close the loop between local development and running your applications in the cloud.

If you’re at the stage of trying to run your application in Microsoft Azure, the choices can be overwhelming. The Azure Container product page lists 7 different products on their landing page, and for new users it can often be difficult to decide which of the myriad products is right for their use case. What can make it even more confusing is that often these container services can be interoperable, meaning you can use one container product from another!

In this post, we’re going to examine each of the main container services offered in Azure and then examine what they’re good for and what they might not be so good for. Let’s take a look!

Azure Container Instances

Azure Container Instances (ACI) bills itself as an offering that lets you “easily run containers on Azure without managing servers.”

Launched in July 2017, it was arguably the first “serverless” container offering from the major cloud providers.

What’s good about it?

ACI is a Platform-as-a-Service (PaaS) type of product that is designed to remove the burden of launching and running containers.

Simplicity

Azure Container Instances has been designed from the ground up to be straightforward to use. It’s designed to get your container launched as quickly as possible, even including integrations with many of the tools you’re probably already familiar with if you’re using containers. One example is the integration with the Docker CLI.

In many cases, the only configuration options you need to pass to your ACI provisioning mechanism is the image you’re running and some sizing decisions, like how many CPUs and how much memory you need. There is some flexibility in this decision process, but having fewer required options means you’re not bogged down with complex architectural decisions when trying to get your container running.

This lends itself particularly well to web server projects, especially at smaller scales. You can launch applications very quickly and easily by specifying the image you’ve built, and your application can be running in very short order.

Start-Up Times

ACI Containers start fast—often in a matter of seconds. This speed really helps the developer lifecycle and helps to create productivity when you’re developing your application.

Pricing

ACI will only bill you for the time your applications are running. It offers no start-up costs (which goes great with the fast start times!) and per-second billing, meaning you only pay for every second the application is actually up. The pricing is also extremely competitive against other cloud providers.

The Other Stuff

From a security perspective, ACI separates your container from others running in the same environment, meaning your application will have the same level of security as if it was running on a different virtual machine. This separation really helps to provide peace of mind and is yet another benefit of the serverless model—less infrastructure means less worries.

As expected from Microsoft, ACI has excellent support for Windows containers and .NET platform applications, which is lacking around other cloud providers. If you need to run Windows containers, this support is a huge benefit.

What’s not great about it?

Scaling

ACI supports customizing the amount of resources you can assign to a container, but it will only run one copy. There’s no horizontal scaling available, so if you need more resources, you only have one option: scaling vertically. It is possible to run N number of ACI instances and configuring load balancers across them, but it’s a very manual process and can be awkward to manage and scale successfully.

If you’re running a webserver application, horizontal scaling behind a load balancer might be a better option, so this lack of scaling flexibility and manual processing can be a real shortcoming.

In addition to this opinionated scaling mechanism, there’s also no auto scaling option for ACI. If you’re finding that you’re hitting the limits of your ACI deployment, you’ll need to manually scale the service.

Configuration

ACI has some configuration options missing, which you’d expect would be there from the beginning. One glaring omission is port mapping (for example mapping public port 80 to container port 8080). This omission can mean modifying your container just to get it running in ACI.

ACI’s opinionated architecture seems to have been designed for ease-of-use, which often means more complex workloads just aren’t possible. If you have some unique configuration options, like needing access to privileged container options or complex network requirements, ACI might not be able to support you.

What we’d use it for

The lack of flexibility in ACI means that some workloads just aren’t able to run there. If you have full control over the container images you’re building and if you just want to get off the ground quickly with minimal fuss and configuration, ACI is a great choice.

Applications with smaller and more predictable usage patterns are a great use case for ACI. The lack of horizontal scaling could be seen as a negative, but from another perspective, it means the use case is straightforward and simple. If you just want to get your application into the hands of your users and think you can accurately predict your usage patterns, choose ACI.

Azure App Service

Azure App Service is a more fully-featured offering for hosting applications, focusing more on WebApps and API services. It started life as several differing offerings (Azure Websites, Azure Mobile Service, and Azure BizTalk) before being bundled into a single service in March 2015. Its origins as multiple solutions means it has developed tight integrations with other Azure solutions and offers many value-add features depending on your use case.

Because of the myriad features Azure App Service offers, running containers is only a small part of what you can do with it.

With that in mind, let’s take a look at where it shines with regard to running containers.

What’s it good for?

Azure App Services is designed to be a one-stop shop for your web application needs, so it’s highly opinionated and pushes you in a certain direction.

Integrations

Azure App Service has many integrations with Azure. As an example, if you need to add a CDN to your web service, it’s extremely straightforward to do so with Azure CDN.

This tight integration with Azure’s many other services makes it likely to meet the needs of your application and extend it as needed, introducing common architecture components like message queues and databases. This is particularly useful when it comes to the integration with Azure’s networking model. You can run an Azure App Service inside an Azure Virtual Network which is extremely useful in many scenarios where network separation is needed, like enterprise applications.

Scaling & Availability

Azure App Service has a myriad of available scaling capabilities built out of the box. Not only can you scale the actual container you’ve deployed (while also simultaneously placing pricing caps on your service), but you can also scale related resources if necessary, which fits well with the tight integration we mentioned previously.

App service scaling for your container can either scale up (i.e., increase the amount of resources associated with your container) or scale out (ie: have more containers behind your load balancer).

This flexibility in scaling means you can be safe to assume your application will meet spiking traffic requirements and service your users’ needs.

What’s not great about it?

Opinionated

Azure App Service is highly opinionated. Its tight integrations with other Azure services mean that you’re expected to follow established patterns for achieving your goals.

This opinionated implementation means that the initial learning curve can be steep. The origins of app services means the product has grown in complexity along with its opinionated implementation, and as a result it can be overwhelming when getting started.

In addition to the learning curve, App Service is primarily focused towards web app workloads, and support for running other applications is limited at best.

What we’d use it for

The scaling options available to App Service means that it’s extremely well positioned to handle your large scale, fluctuating web applications. If you expect to have traffic spikes during peak hours, you can configure App Service to scale relatively easily in a manner that suits your budget and needs.

Azure Kubernetes Service

Azure Kubernetes Service (AKS) is Azure’s managed Kubernetes offering, which has been available since October 2017.

Azure was very quick to market with its Kubernetes offering and hired several prominent Kubernetes engineers like Brendan Burns to head up the engineering departments building the service.

Kubernetes is a very fully featured product, and AKS is a very conformant implementation of Kubernetes. Therefore, a lot of our good and bad considerations next will be an evaluation of Kubernetes, as well as AKS.

What’s it good for?

Extensibility

Kubernetes is incredibly feature rich. Out of the box with AKS, you’ll get application self-healing capabilities via the mechanism of reconcile loops built into Kubernetes, the ability to run different types of workload via native Kubernetes resources such as StatefulSets, Deployments and Jobs, and a large amount of extensibility via Kubernetes’ Custom Resource Definitions capability.

Familiarity

Kubernetes also offers a familiar API regardless of the cloud provider you’re working with. If you’re new to AKS, but have used a managed Kubernetes service from another cloud provider, you might find it relatively easy to get your container up and running in AKS.

What’s not great about it?

Complexity

Kubernetes has a steep learning curve, and AKS is no exception. A lot of the knowledge needed to run and operate Kubernetes is handled by AKS itself, but you still have to have an in-depth understanding of how Kubernetes works to get your container running.

In addition, the specification for a simple Kubernetes job can often be hundreds of lines of code. You have to learn not only how to run the container but also understand how Kubernetes exposes services to the world.

What we’d use it for

The flexibility and extensibility offered by Kubernetes as a platform means it’s possible to run many types of workloads. Large companies have battle-tested Kubernetes (and by extension, AKS), running everything from web applications to machine learning.

If you’re able to invest time into learning and understanding the ecosystem and can dedicate people to maintaining and managing your Kubernetes implementation, AKS is a fantastic option for almost any workload you might consider.

Everything Else

Azure Red Hat OpenShift

Azure Red Hat OpenShift is an enterprise grade implementation of Kubernetes developed by Red Hat. OpenShift builds on Kubernetes by adding support, enterprise-grade security, and Platform-as-a-Service integrations.

If the idea of using Kubernetes for your container solutions seems appealing but you need service level agreements (SLAs) around support and integrations, Azure’s Red Hat OpenShift solution is a great option.

Azure Service Fabric

Azure Service Fabric is a container service focused on distributed applications, like microservice architectures or complex distributed compute environments.

If you’re planning on running a complex architecture, Azure Service Fabric can be a great fit. It provides more orchestration capabilities than Azure App Service. Service Fabric can also extend outside Azure’s cloud into your on-premise data center if necessary.

If your application has needs related to low latency, especially across distributed regions, Service Fabric is a great fit.

However, Azure Service Fabric can be extremely difficult to use. It is only targeted towards distributed, complex systems - if you’re not planning on building one of those, it’s probably not the correct choice.

Azure Batch

Azure Batch is a service focused on High Performance Compute (HPC) workloads. These workloads can often scale horizontally to thousands of processes and can be difficult to orchestrate.

Azure Batch makes this process simpler by providing a managed service to handle the difficult processes.

If you’re planning on passing large amounts of data to your application via a queuing mechanism and having that data processed very quickly, Azure Batch is definitely a service you should consider.

Azure Functions

Azure Functions is Azure’s Function-as-a-Service implementation. It supports launching “function apps” inside containers as a deployment mechanism.

Function-as-a-Service products are often designed for short-lived, event-driven workloads. If you need to run a script to react to a worker queue or other event, Azure Functions are a great option. It’s important to remember that Functions are often billed based on how long they run, so short-lived workloads are often better suited.

You might also like:

Wrap-Up

There are a number of options for running containers in Azure, and choosing the right one for your workload might be difficult. We’ve tried to give an overview of the current state of the world, but Microsoft continues to add options and capabilities to all these products.

If you’ve settled on a container solution, and are ready to get started, Pulumi’s Azure Native provider supports all of the methods of running containers and can quickly get you started. Check out our Azure getting started guide, or join one of our upcoming Azure Workshops!

How to Use ECS Anywhere with Pulumi [Step-by-Step Guide]

Piers Karsenbarg — Thu, 27 May 2021 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

When Amazon’s Elastic Container Service (ECS) first launched in 2014, it enabled an easy and convenient way of deploying and scheduling containers in the AWS ecosystem. Back then, you would run a set of EC2 instances, and ECS would deploy containers to instances based on the size, resources, and placement requirements you specified.

In 2018, AWS released Fargate, bringing a serverless feel to the platform. You could have ECS run your container without having to manage any infrastructure.

The next iteration is ECS Anywhere, which allows you to deploy your containers to almost any server. You can deploy to an EC2 instance inside AWS, a virtual machine (VM) in another cloud provider, or even a bare metal server in your private data center. Best of all, you can control the deployments in the same way you would with EC2 or Fargate.

Why would I want to use this?

As we’ve seen over the past few years, the ECS system makes it a convenient service to use. There are several reasons why you might want to use ECS as a control plane for your container-based applications but not run the actual applications within your AWS infrastructure. These include:

Latency requirements: You have systems on another network, and you need your containers to be running on that same network to give you the low latency required.
Regulatory requirements: There may be legal, security, or privacy needs that can only be satisfied by running your application in your own data center.
Operational requirements: you don’t want to manage a complex container cluster yourself on another network and are already running workloads on ECS.

Behind the scenes

The crucial part of understanding how ECS Anywhere works is that you’re using ECS to manage the deployment of the containers to your VMs (which we’ll call nodes going forward). The actual interaction between your application and whatever network traffic or instructions it receives is separate. You’ll see this shown in the example below where we deploy two VMs and a load balancer. Once the application is up and running, ECS takes a back seat and waits for the instructions to deploy a new version or stop the application from running altogether.

There are three parts to getting this to work:

1. Setting up the AWS infrastructure

AWS’ Systems Manager has a feature called “Hybrid Activations.” We’ll use it to link the external nodes to the ECS cluster using an activation code and an activation id. We’ll also create various IAM roles necessary to activate the nodes and assign them to the ECS cluster. In our example, we’ll also set up IAM roles so that our application can send logs to CloudWatch as well.

2. Deploying the nodes

During the node setup process, we’re going to download and run a custom script from AWS that performs several tasks:

It installs the docker engine.
It installs the SSM agent and registers it with the Systems Manager’s Fleet manager (which keeps track of the nodes and assigns an IAM role)
It installs the ECS agent, which connects to the ECS cluster and listens for instructions.

3. Deploying the application

There are very few differences between deploying an application to ECS internally on AWS infrastructure here. We set up a Task Definition and a Service within the ECS infrastructure, and the only difference is that we’ll use the new launch type of EXTERNAL instead of EC2 or FARGATE, which tells the cluster to push the instructions for running the application to our nodes.

Deploying a Node.js application using ECS Anywhere

In this example, we will be deploying a basic Node.js application within a container to a pair of virtual machines (VMs) behind a load balancer. You’ll need multiple VMs because ECS won’t deploy the multiple task definitions with the same open network port to the same node (this also happens within the EC2 flavor of ECS).

Let’s be constant

Before setting up our cloud resources, we begin by setting some variables that we’ll use later on throughout our Pulumi program.

We’ll first set up our configuration settings in our command line:

pulumi config set aws:region us-east-1
pulumi config set numberNodes 2

The first tells Pulumi which region to deploy the resources to (and we’ll also use it when setting up our nodes and CloudWatch logging). The second tells us how many nodes to create, and we’ll need at least two to allow ECS to perform a rolling deployment.

Since we’re using a third-party compute provider to prove that ECS can deploy the containers to non-AWS infrastructure, we need to provide an API token. In this example, we’ll add it to the config, but you should read our Setup Guide for more information on how to deploy.

To add your token to the config (as a secret), you should run the following command:

pulumi config set digitalocean:token XXXXXXXXXXXX --secret

Exchange XXXXXXXXXXXX for a personal access token located in the Control Panel.

We’ll use these to set variables in our code:

import * as pulumi from "@pulumi/pulumi";

const awsConfig = new pulumi.Config("aws");
const awsRegion = awsConfig.get("region");

const projectConfig = new pulumi.Config();
const numberNodes = projectConfig.getNumber("numberNodes") || 2;

(notice that for the number of nodes, we’re setting a default value in case this configuration setting does not exist)

Setting up IAM roles

We’re going to need three IAM roles:

The first is for the AWS Systems manager. We’re going to attach two managed policies to this: one to enable its core functionality (AmazonSSMManagedInstanceCore) and the default EC2 role for ECS (AmazonEC2ContainerServiceforEC2Role).
The second is the ECS execution role, and to that, we’re going to attach the managed policy to allow other AWS services to run ECS tasks (AmazonECSTaskExecutionRolePolicy)
The final one is the Task role, and we’re going to add a custom policy to that role to allow us to push logs from the nodes to CloudWatch.

const ssmRole = new aws.iam.Role("ssmRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.SsmPrincipal
 ),
});

new aws.iam.RolePolicyAttachment("rpa-ssmrole-ssminstancecore", {
 policyArn: aws.iam.ManagedPolicy.AmazonSSMManagedInstanceCore,
 role: ssmRole,
});

new aws.iam.RolePolicyAttachment("rpa-ssmrole-ec2containerservice", {
 policyArn: aws.iam.ManagedPolicy.AmazonEC2ContainerServiceforEC2Role,
 role: ssmRole,
});

const executionRole = new aws.iam.Role("taskExecutionRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.EcsTasksPrincipal
 ),
});

new aws.iam.RolePolicyAttachment("rpa-ecsanywhere-ecstaskexecution", {
 role: executionRole,
 policyArn: aws.iam.ManagedPolicy.AmazonECSTaskExecutionRolePolicy,
});

const taskRole = new aws.iam.Role("taskRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.EcsTasksPrincipal
 ),
});

new aws.iam.RolePolicy("taskRolePolicy", {
 role: taskRole.id,
 policy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: [
 "ssmmessages:CreateControlChannel",
 "ssmmessages:CreateDataChannel",
 "ssmmessages:OpenControlChannel",
 "ssmmessages:OpenDataChannel",
 ],
 Resource: "*",
 },
 {
 Effect: "Allow",
 Action: ["logs:DescribeLogGroups"],
 Resource: "*",
 },
 {
 Effect: "Allow",
 Action: [
 "logs:CreateLogStream",
 "logs:CreateLogGroup",
 "logs:DescribeLogStreams",
 "logs:PutLogEvents",
 ],
 Resource: "*",
 },
 ],
 }
});

Deploying the ECS cluster and node infrastructure

Next, we’re going to set up the System Manager activation, which provides an Id and a code that lets our nodes connect to the cluster. We’ll set them up at the same time as the CloudWatch log group that we’ll push application logs to later on:

const ssmActivation = new aws.ssm.Activation("ecsanywhere-ssmactivation", {
 iamRole: ssmRole.name,
 registrationLimit: numberNodes,
});

const cluster = new aws.ecs.Cluster("cluster");

export const clusterName = cluster.name;

const logGroup = new aws.cloudwatch.LogGroup("logGroup");

We’re using the ssmRole we created earlier to allow the Systems Manager to do this. You’ll also notice we’re exporting the clusterName here. We will use this to tidy up later on when we tear down our infrastructure.

When we create the nodes, we will run an AWS bash script provided as part of the setup and pass it into the resource as userData. In this script, we pass in the cluster name, the activation id, the activation code, and the AWS region where the cluster will be deployed.

const userData = pulumi
 .all([ssmActivation.activationCode, ssmActivation.id, cluster.name])
 .apply(
 ([activationCode, activationId, clusterName]) => `#!/bin/bash
# Download the ecs-anywhere install Script
curl -o "ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && sudo chmod +x ecs-anywhere-install.sh

# (Optional) Check integrity of the shell script
curl -o "ecs-anywhere-install.sh.sha256" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh.sha256" && sha256sum -c ecs-anywhere-install.sh.sha256

# Run the install script
sudo ./ecs-anywhere-install.sh \
 --cluster ${clusterName} \
 --activation-id ${activationId} \
 --activation-code ${activationCode} \
 --region ${awsRegion}
`
 );

This next section of code deploys the VMs.

We’re going to create a tag that attaches the nodes to a load balancer. Next, we’ll create the VMs and assign the tag. Finally, we’ll deploy the load balancer and assign the same tag to route traffic to the nodes. We’ll also take the IP address of the load balancer and mark it as an output so we have something we can either use in a browser or call with curl to ensure that our deployment has worked.

In the end, we’re going to output the load balancer IP address so we have something we can either use in a browser or call with curl to ensure that our deployment has worked.

const loadBalancerTag = new digitalocean.Tag("lb");

for(let i = 1; i <= numberNodes; i++) {
 new digitalocean.Droplet(`droplet-${i}`, {
 region: digitalocean.Regions.NYC1,
 size: "s-1vcpu-2gb",
 image: "ubuntu-20-04-x64",
 userData: userData,
 tags: [loadBalancerTag.id]
 });
}

const lb = new digitalocean.LoadBalancer("lb", {
 region: digitalocean.Regions.NYC1,
 forwardingRules: [{
 entryPort: 80,
 entryProtocol: digitalocean.Protocols.HTTP,
 targetPort: 80,
 targetProtocol: digitalocean.Protocols.HTTP
 }],
 healthcheck: {
 port: 80,
 protocol: digitalocean.Protocols.HTTP,
 path: "/"
 },
 dropletTag: loadBalancerTag.name
});

export const ip = lb.ip;

The last part of the example will be familiar to anyone using ECS as a container scheduler: setting up the container repository, describing the task definition, and creating the service. The difference is that we use new compatibility in the task definition and a new launch type in the service - EXTERNAL. This tells ECS to push the container and start them on the external nodes.

const repo = new awsx.ecr.Repository("app");

const image = repo.buildAndPushImage("./app");

const taskDefinition = pulumi.all([image, logGroup.name, logGroup.namePrefix]).apply(
 ([img, logGroupName, nameprefix]) =>
 new aws.ecs.TaskDefinition("taskdefinition", {
 family: "ecs-anywhere",
 requiresCompatibilities: ["EXTERNAL"],
 taskRoleArn: taskRole.arn,
 executionRoleArn: executionRole.arn,
 containerDefinitions: JSON.stringify([
 {
 name: "app",
 image: img,
 cpu: 256,
 memory: 256,
 essential: true,
 portMappings: [
 {
 containerPort: 80,
 hostPort: 80
 },
 ],
 logConfiguration: {
 logDriver: "awslogs",
 options: {
 "awslogs-group": logGroupName,
 "awslogs-region": awsRegion,
 "awslogs-stream-prefixs": nameprefix
 }
 }
 },
 ]),
 })
);

const service = new aws.ecs.Service("service", {
 launchType: "EXTERNAL",
 taskDefinition: taskDefinition.arn,
 cluster: cluster.id,
 desiredCount: numberNodes - 1,
 }
);

Now you can run pulumi up, and once it’s all deployed, run curl ${pulumi stack output ip}.

If you want to update the Dockerfile or the Node.js application, run pulumi up again. You’ll see the new container deployed. Then you can run curl ${pulumi stack output ip} again to see the new response.

Finally, when you want to destroy the infrastructure, there is an additional step before running the usual pulumi destroy. This is because the nodes are registered to AWS Systems Manager and the ECS cluster as part of the node setup and happen outside of the Pulumi stack.

Run the following in your command line (you’ll need to install jq for this to work):

aws ssm describe-instance-information | jq ".InstanceInformationList | .[] | .InstanceId" | grep "mi-" | xargs -L 1 aws ssm deregister-managed-instance --instance-id

# Cleanup ECS resources
aws ecs list-container-instances --cluster ${pulumi stack output clusterName} | jq ".containerInstanceArns | .[]" | xargs -L 1 aws ecs deregister-container-instance --cluster ${pulumi stack output clusterName} --force --container-instance

# Refresh Pulumi stack
pulumi refresh -y

And then you can tear down all your infrastructure with the usual pulumi destroy.

You can see this code in its entirety in our examples repository on GitHub.

Summary

ECS Anywhere is an excellent addition to Amazon Web Services’ suite of container orchestration services. I can see many companies using it to run containers in their private infrastructure but don’t necessarily have the means, know-how, or drive to set up and run their Kubernetes cluster or another such tool.

We’ve shown you how to set up your ECS cluster to use external nodes, build and deploy the cluster and application and see for yourself how easy it is to deploy subsequent changes. Pulumi makes it simple to create container images and provision and manage cloud infrastructure on any cloud using familiar programming languages, including TypeScript, Python, Go, and .NET. Cloud engineers can manage container images, ECR registries, and multi-cloud deployments within the same infrastructure definition.

How to Build a Container Registry

Sophia Parafina — Tue, 16 Feb 2021 00:00:00 +0000

Whether you are working with Kubernetes or serverless, your application uses containers. If you use the Docker desktop client, images are pushed to Docker Hub by default. Pulling images from Docker Hub is convenient, but there are many reasons to store images in your own registry. For example, Docker Hub doesn’t guarantee to produce the same image on repeated pulls, i.e., your base image might have changed. It’s also possible to inadvertently expose secrets in an intermediate image used to build the image stored on Docker Hub. There is also the possibility of vulnerabilities in even official images. This article shows how to create a repository and how to build and push images to that repository

Container registry offerings

AWS provides the Elastic Container Registry (ECR), Azure has Container Registry, and Google has it’s Container Registry. Each provider has associated services unique to their offering, but all support Docker or OCI compliant images.

Build it

Let’s examine how to create a registry with the provider of your choice. In these examples, we create a registry, build a Docker image, and push the image to the registry. The application used for the image is NGINX.

Choose your cloud provider to learn how to build a registry.

In this example, we create an ECR repository configured to scan an image’s Operating System components. Scanning for vulnerabilities in an application is currently out of scope. We also set a policy for the repository that controls the actions allowed and a lifecycle policy that expires an image after a set time.

With Pulumi, it’s possible to build an image locally using Docker and push it to your repository. To push the image, we obtain the credentials required to push from the registry. Finally, we export the credentials and the URL for the registry. Read more about ECR in the API Reference.

import * as docker from "@pulumi/docker";
import * as aws from "@pulumi/aws";

// Create a repository and configure to scan the image on push
const repo = new aws.ecr.Repository("myrepository", {
 imageScanningConfiguration: {
 scanOnPush: true
 },
 imageTagMutability: "MUTABLE",
});

// Set a use policy for the repository
const repositoryPolicy = new aws.ecr.RepositoryPolicy("myrepositorypolicy", {
 repository: repo.id,
 policy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [{
 Sid: "new policy",
 Effect: "Allow",
 Principal: "*",
 Action: [
 "ecr:GetDownloadUrlForLayer",
 "ecr:BatchGetImage",
 "ecr:BatchCheckLayerAvailability",
 "ecr:PutImage",
 "ecr:InitiateLayerUpload",
 "ecr:UploadLayerPart",
 "ecr:CompleteLayerUpload",
 "ecr:DescribeRepositories",
 "ecr:GetRepositoryPolicy",
 "ecr:ListImages",
 "ecr:DeleteRepository",
 "ecr:BatchDeleteImage",
 "ecr:SetRepositoryPolicy",
 "ecr:DeleteRepositoryPolicy"
 ]
 }]
 })
});

// Set a policy to control the lifecycle of an image
const lifecyclePolicy = new aws.ecr.LifecyclePolicy("mylifecyclepolicy", {
 repository: repo.id,
 policy: JSON.stringify({
 rules: [{
 rulePriority: 1,
 description: "Expire images older than 14 days",
 selection: {
 tagStatus: "untagged",
 countType: "sinceImagePushed",
 countUnit: "days",
 countNumber: 14
 },
 action: {
 type: "expire"
 }
 }]
 })
});

// Get the repository credentials we use to push to the repository
const repoCreds = repo.registryId.apply(async (registryId) => {
 const credentials = await aws.ecr.getCredentials({
 registryId: registryId,
 });
 const decodedCredentials = Buffer.from(credentials.authorizationToken, "base64").toString();
 const [username, password] = decodedCredentials.split(":");
 return { server: credentials.proxyEndpoint, username, password };
});

// Create a new image and push to the repository
const image = new docker.Image("myapp", {
 imageName: repo.repositoryUrl,
 build: "./app",
 registry: repoCreds,
})

// Export credentials and URL to the repository
export const credentials = repoCreds;
export const repoEndpoint = repo.repositoryUrl;

In this example, we create an Azure Resource Group to contain the resources for the registry, such as the App Service that hosts the registry.

We instantiate a registry with the containerservice module and use the Image module in the Docker package to build and push the image to the registry. We export the registry URL and the username and password in case we should want to push or pull and image using the Docker CLI.

import * as azure from "@pulumi/azure";
import * as docker from "@pulumi/docker";
import * as pulumi from "@pulumi/pulumi";

// Create an Azure Resource Group
const resourceGroup = new azure.core.ResourceGroup("examples");

// Create a dedicated App Service Plan for Linux App Services
const plan = new azure.appservice.Plan("linux-apps", {
 resourceGroupName: resourceGroup.name,
 kind: "Linux",
 reserved: true,
 sku: {
 tier: "Basic",
 size: "B1",
 },
});

const repo = new azure.containerservice.Registry("myrepository", {
 resourceGroupName: resourceGroup.name,
 sku: "Basic",
 adminEnabled: true,
});

const myImage = new docker.Image("myapp", {
 imageName: pulumi.interpolate`${repo.loginServer}/${"myapp"}:v1.0.0`,
 build: {
 context: `./${"app"}`,
 },
 registry: {
 server: repo.loginServer,
 username: repo.adminUsername,
 password: repo.adminPassword,
 },
});

export const server = repo.loginServer;
export const username = repo.adminUsername;
export const password = repo.adminPassword;

In this example, we’ll use the configuration and credentials from the gcloud CLI to build an image and push it into the GCP registry. Make sure the GCP project is set and you are logged into GCP and Docker is configured to use the GCR,

$ gcloud init
$ gcloud auth login
$ gcloud auth configure-docker

We use the Image module in the Docker package to build and push the image to the registry. We export the registry URL and the username and password in case we should want to push or pull an image using the Docker CLI.

import * as docker from "@pulumi/docker";
import * as gcp from "@pulumi/gcp";
import * as pulumi from "@pulumi/pulumi";

// Build and push image to gcr repository

const imageName = "myapp";

const myImage = new docker.Image(imageName, {
 imageName: pulumi.interpolate`gcr.io/${gcp.config.project}/${imageName}:latest`,
 build: {
 context: "./app",
 },
});

// Export the repository end point
export const repoEndpoint = pulumi.interpolate`gcr.io/${gcp.config.project}`;

Learn more

Container registries are just one of the many resources used for deploying modern applications. Implementations among cloud service providers differ by the functionality they offer and how they are deployed. The commonality among them is that they provide a secure place to store and retrieve Docker or OCI compliant container images. Explore how to create and manage resources for the cloud service provider of your choice with Pulumi. Great places to start are:

Build and publish container images to any cloud with Infrastructure as Code

Joe Duffy — Tue, 08 Dec 2020 00:00:00 +0000

Going from a containerized application to a service running in the cloud requires a few steps beyond an application’s normal build-and-test cycle. Namely, it means building and publishing a container image in a registry and then consuming that image from your target environment, whether that’s Kubernetes, Amazon ECS, or another container orchestrator. It’s not enough to just write a Dockerfile — you will need to pick a container registry, decide whether that registry should be public or private, authenticate against it, and ideally automate deploying subsequent updates. Infrastructure as code to the rescue! In this article, we’ll see how to build, publish, and consume a simple container image across any cloud, using just a few lines of code.

Approach

The general approach will be to create a new infrastructure as code project that

Prepares a container registry, either public or private
Builds and publishes your container image to that registry
Optionally, consumes the resulting image URL from a containerized task definition in Kubernetes, Amazon ECS, or any other container orchestrator

All told, this will be just a few dozen lines of code. This article demonstrates doing this in AWS, Azure, Google Cloud, DigitalOcean, and Docker Hub, and offers code examples in each supported Pulumi language, namely Python, JavaScript, TypeScript, Go, and C#. These steps will work for any application that has a Dockerfile and is buildable by Docker. In principle, similar steps could be applied if you prefer to build your container image using different means, such as Buildpack.

For purposes of illustration, we’ll create a simple Nginx web server whose Dockerfile contains:

FROM nginx
RUN echo "<h1>Hello, World!</h1>" > \
 /usr/share/nginx/html/index.html

Now, let’s dive in!

Prepare a Container Registry

The first step is to prepare a new container registry. A registry holds one or more repositories, each of which can store and serve many different container images with different tags and versions. Afterward we’ll show how to build and publish to this registry.

The specific details of how to prepare your registry differ by cloud provider, often significantly, particularly when it comes to authenticating. Pick your cloud provider to see the details:

Amazon Elastic Container Registry (ECR)

Amazon Elastic Container Registry (ECR) provides managed Docker container hosting that makes it easy to run containerized applications in your AWS account using Amazon Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS). Each account has a default registry per region, and each registry may have any number of repositories, each for a different Docker image. Each repository can store many versions of that particular image.

Create a New Project

To start, create a new project and ensure it is configured to use your AWS account, and then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var aws = require("@pulumi/aws");
var docker = require("@pulumi/docker");

// [Placeholder 1: Create a private ECR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as aws from "@pulumi/aws";
import * as docker from "@pulumi/docker";

// [Placeholder 1: Create a private ECR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import base64
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker

# [Placeholder 1: Create a private ECR registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/ecr"
 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private ECR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Aws.Ecr;
using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private ECR repository.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Provision an ECR Repository

Next, declare a new ECR repository resource:

// Create a private ECR repository.
var repo = new aws.ecr.Repository("my-repo");

// Create a private ECR repository.
const repo = new aws.ecr.Repository("my-repo");

# Create a private ECR repository.
repo = aws.ecr.Repository('my-repo')

// Create a private ECR repository.
repo, err := ecr.NewRepository(ctx, "my-repo", nil)
if err != nil {
 return err
}

// Create a private ECR repository.
var repo = new Repository("my-repo");

Authenticate with Temporary ECR Access Token

Next, we will need to generate authentication information to access the repository to build and publish our image. ECR supports this in multiple different ways; however, here, we will demonstrate generating a temporary access token:

// Get registry info (creds and endpoint).
var imageName = repo.repositoryUrl;
var registryInfo = repo.registryId.apply(id => {
 return aws.ecr.getCredentials({ registryId: id }).then(credentials => {
 var decodedCredentials = Buffer.from(credentials.authorizationToken, "base64").toString();
 var [username, password] = decodedCredentials.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: credentials.proxyEndpoint,
 username: username,
 password: password,
 };
 });
});

// Get registry info (creds and endpoint).
const imageName = repo.repositoryUrl;
const registryInfo = repo.registryId.apply(async id => {
 const credentials = await aws.ecr.getCredentials({ registryId: id });
 const decodedCredentials = Buffer.from(credentials.authorizationToken, "base64").toString();
 const [username, password] = decodedCredentials.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: credentials.proxyEndpoint,
 username: username,
 password: password,
 };
});

# Get registry info (creds and endpoint).
def getRegistryInfo(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])
image_name = repo.repository_url
registry_info = repo.registry_id.apply(getRegistryInfo)

// Get registry info (creds and endpoint).
imageName := repo.RepositoryUrl
registryInfo := repo.RegistryId.ApplyT(func(id string) (docker.ImageRegistry, error) {
 creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{RegistryId: id})
 if err != nil {
 return docker.ImageRegistry{}, err
 }
 decoded, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
 if err != nil {
 return docker.ImageRegistry{}, err
 }
 parts := strings.Split(string(decoded), ":")
 if len(parts) != 2 {
 return docker.ImageRegistry{}, errors.New("Invalid credentials")
 }
 return docker.ImageRegistry{
 Server: creds.ProxyEndpoint,
 Username: parts[0],
 Password: parts[1],
 }, nil
}).(docker.ImageRegistryOutput)

// Get registry info (creds and endpoint).
var imageName = repo.RepositoryUrl;
var registryInfo = repo.RegistryId.Apply(async (id) =>
{
 var creds = await GetCredentials.InvokeAsync(new GetCredentialsArgs { RegistryId = id });
 var decodedData = Convert.FromBase64String(creds.AuthorizationToken);
 var decoded = ASCIIEncoding.ASCII.GetString(decodedData);

 var parts = decoded.Split(':');
 if (parts.Length != 2)
 {
 throw new Exception("Invalid credentials");
 }

 return new ImageRegistry
 {
 Server = creds.ProxyEndpoint,
 Username = parts[0],
 Password = parts[1],
 };
});

Alternatively, Authenticate with ECR Credential Helper

An alternative approach ECR supports is to use the Docker credential helper from Amazon, which integrates with local IAM settings, adds smart caching, and removes the need for Docker to login. If you elect to go this route, leave out the username/password parts of the docker.ImageRegistry:

// Get registry info (creds and endpoint).
var imageName = repo.repositoryUrl;
var registryInfo = undefined; // use ECR credentials helper.

// Get registry info (creds and endpoint).
const imageName = repo.repositoryUrl;
const registryInfo = undefined; // use ECR credentials helper.

# Get registry info (creds and endpoint).
image_name = repo.repository_url
registry_info = None # use ECR credentials helper.

// Get registry info (creds and endpoint).
imageName := repo.RepositoryUrl
registryInfo := docker.ImageRegistryArgs{} // use ECR credentials helper.

// Get registry info (creds and endpoint).
var imageName = repo.RepositoryUrl;
// Omit registryInfo and use ECR credentials helper.

Learn More About ECR

ECR offers many additional options not shown here. This includes configuring advanced IAM permissions, enabling image vulnerability scanning, managing the lifetime of images so that older, unused images are deleted based on configured policies, encrypting images, and more. For details on these advanced capabilities and more, refer to Pulumi’s ECR user guide, Pulumi’s ECR API reference, or Amazon’s product documentation.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

Azure Container Registry (ACR)

Azure Container Registry (ACR) allows you to build, store, secure, scan, replicate, and manage container images and artifacts using a fully managed and geo-replicated instance of OCI distribution. This makes it easy to deploy containerized workloads to native Azure services like Azure Kubernetes Service (AKS), Red Hat OpenShift, and App Service. Each registry is capable of storing and serving multiple versions of multiple container images.

Create a New Project

To start, create a new project and ensure it is configured to use your Azure account, and then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var azure = require("@pulumi/azure");
var docker = require("@pulumi/docker");

// [Placeholder 1: Create a private ACR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as azure from "@pulumi/azure";
import * as docker from "@pulumi/docker";

// [Placeholder 1: Create a private ACR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import pulumi
import pulumi_azure as azure
import pulumi_docker as docker

# [Placeholder 1: Create a private ACR registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "github.com/pulumi/pulumi-azure/sdk/v2/go/azure/containerservice"
 "github.com/pulumi/pulumi-azure/sdk/v2/go/azure/core"
 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private ECR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Azure.Core;
using Pulumi.Azure.ContainerService;
using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private ECR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Provision a New ACR Registry

Next, declare a new ACR registry resource. To create a new registry, we need an Azure resource group; here we will create a new one, but feel free to use an existing one by passing its name in the resource group’s place:

// Create a private ACR registry.
var rg = new azure.core.ResourceGroup("myrg");
var registry = new azure.containerservice.Registry("myregistry", {
 resourceGroupName: rg.name,
 adminEnabled: true,
 sku: "Basic",
});

// Create a private ACR registry.
const rg = new azure.core.ResourceGroup("myrg");
const registry = new azure.containerservice.Registry("myregistry", {
 resourceGroupName: rg.name,
 adminEnabled: true,
 sku: "Basic",
});

# Create a private ACR registry.
rg = azure.core.ResourceGroup('myrg')
registry = azure.containerservice.Registry('myregistry',
 resource_group_name=rg.name,
 admin_enabled=True,
 sku='Basic'
)

// Create a private ACR registry.
rg, err := core.NewResourceGroup(ctx, "myrg", nil)
if err != nil {
 return err
}
registry, err := containerservice.NewRegistry(ctx, "my-repo", &containerservice.RegistryArgs{
 ResourceGroupName: rg.Name,
 AdminEnabled: pulumi.Bool(true),
 Sku: pulumi.String("Basic"),
})
if err != nil {
 return err
}

// Create a private ACR registry.
var rg = new ResourceGroup("myrg");
var registry = new Registry("myregistry", new RegistryArgs
{
 ResourceGroupName = rg.Name,
 AdminEnabled = true,
 Sku = "basic"
});

Authenticate with Admin Account

Now we need to gather up the image name and registry authentication information, which has been auto-generated by Azure, in preparation for building and publishing to it. For this simple example, we have enabled admin access, which we will use to authenticate:

// Get registry info (creds and endpoint).
var imageName = registry.loginServer.apply(s => `${s}/myapp`);
var registryInfo = {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
};

// Get registry info (creds and endpoint).
const imageName = registry.loginServer.apply(s => `${s}/myapp`);
const registryInfo = {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
};

# Get registry info (creds and endpoint).
image_name = registry.login_server.apply(lambda s: f'{s}/myapp')
registry_info = docker.ImageRegistry(
 server=registry.login_server,
 username=registry.admin_username,
 password=registry.admin_password
)

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registry.LoginServer)
registryInfo := docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: registry.AdminUsername,
 Password: registry.AdminPassword,
}

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registry.LoginServer}/myapp");
var registryInfo = new ImageRegistry
{
 Server = registry.LoginServer,
 Username = registry.AdminUsername,
 Password = registry.AdminPassword,
};

Alternatively, Authenticate with ActiveDirectory Service Principal

Although enabling admin access is “easy”, it isn’t the most secure approach. An alternative is to set “admin enabled” to false when declaring our registry, provision or assign an ActiveDirectory service principal with an acrpush role scoped to our registry, and use that for authentication instead:

// New imports:
var azuread = require("@pulumi/azuread");
var random = require("@pulumi/random");

// [As before...]

// Get registry info (creds and endpoint).
var sp = new azuread.ServicePrincipal("mysp", {
 applicationId: new azuread.Application("myspapp").applicationId,
});
var spPassword = new azuread.ServicePrincipalPassword("mysp-pass", {
 servicePrincipalId: sp.id,
 value: new random.RandomPassword("mypass", {
 length: 32,
 }, { additionalSecretOutputs: [ "result" ] }).result,
 endDateRelative: "8760h",
});
var spAuth = new azure.authorization.Assignment("myauth", {
 scope: registry.id,
 roleDefinitionName: "acrpush",
 principalId: sp.id,
});
var registryInfo = {
 server: registry.loginServer,
 username: sp.applicationId,
 password: spAuth.id.apply(_ => spPassword.value),
};

// New imports:
import * as azuread from "@pulumi/azuread";
import * as random from "@pulumi/random";

// [As before...]

// Get registry info (creds and endpoint).
const sp = new azuread.ServicePrincipal("mysp", {
 applicationId: new azuread.Application("myspapp").applicationId,
});
const spPassword = new azuread.ServicePrincipalPassword("mysp-pass", {
 servicePrincipalId: sp.id,
 value: new random.RandomPassword("mypass", {
 length: 32,
 }, { additionalSecretOutputs: [ "result" ] }).result,
 endDateRelative: "8760h",
});
const spAuth = new azure.authorization.Assignment("myauth", {
 scope: registry.id,
 roleDefinitionName: "acrpush",
 principalId: sp.id,
});
const registryInfo = {
 server: registry.loginServer,
 username: sp.applicationId,
 password: spAuth.id.apply(_ => spPassword.value),
};

# New imports:
import pulumi_azuread as azuread
import pulumi_random as random

# [As before...]

# Get registry info (creds and endpoint).
image_name = registry.login_server.apply(lambda s: f'{s}/myapp')
sp = azuread.ServicePrincipal('mysp',
 application_id=azuread.Application('myspapp').application_id,
)
sp_password = azuread.ServicePrincipalPassword('mysp-pass',
 service_principal_id=sp.id,
 value=random.RandomPassword('mypass',
 length=32,
 opts=pulumi.ResourceOptions(additional_secret_outputs=['result'])
 ).result,
 end_date_relative='8760h',
)
sp_auth = azure.authorization.Assignment('myauth',
 scope=registry.id,
 role_definition_name='acrpush',
 principal_id=sp.id,
)
registry_info = docker.ImageRegistry(
 server=registry.login_server,
 username=sp.application_id,
 password=sp_auth.id.apply(lambda _: sp_password.value),
)

// New imports:
import (
 "github.com/pulumi/pulumi-azure/sdk/v2/go/azure/authorization"
 "github.com/pulumi/pulumi-azuread/sdk/v2/go/azuread"
 "github.com/pulumi/pulumi-random/sdk/v2/go/random"
)

// [As before...]

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registry.LoginServer)
spApp, err := azuread.NewApplication(ctx, "myspapp", nil)
if err != nil {
 return err
}
sp, err := azuread.NewServicePrincipal(ctx, "mysp", &azuread.ServicePrincipalArgs{
 ApplicationId: spApp.ApplicationId,
})
if err != nil {
 return err
}
password, err := random.NewRandomPassword(ctx, "mypass",
 &random.RandomPasswordArgs{
 Length: pulumi.Int32(32),
 },
 pulumi.AdditionalSecretOutputs([]string{"result"}),
)
if err != nil {
 return err
}
spPassword, err := azuread.NewServicePrincipalPassword(ctx, "mysp-pass", &azuread.ServicePrincipalPasswordArgs{
 ServicePrincipalId: sp.ID(),
 Value: password.Result,
 EndDateRelative: pulumi.String("8760h"),
})
if err != nil {
 return err
}
apAuth, err := authorization.NewAssignment(ctx, "myauth", &authorization.AssignmentArgs{
 Scope: registry.ID(),
 RoleDefinitionName: pulumi.String("acrpush"),
 PrincipalId: sp.ID(),
})
if err != nil {
 return err
}
registryInfo := docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: sp.ApplicationId,
 Password: spPassword.Value,
}

// New imports:
using Pulumi.Azure.Authorization;
using AAD = Pulumi.AzureAD;
using Pulumi.Random;

// [As before...]

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registry.LoginServer}/myapp");
var sp = new AAD.ServicePrincipal("mysp", new AAD.ServicePrincipalArgs
{
 ApplicationId = new AAD.Application("myspapp").ApplicationId,
});
var spPassword = new AAD.ServicePrincipalPassword("mysp-pass", new AAD.ServicePrincipalPasswordArgs
{
 ServicePrincipalId = sp.Id,
 Value = new RandomPassword("mypass",
 new RandomPasswordArgs
 {
 Length = 32,
 },
 new CustomResourceOptions { AdditionalSecretOutputs = { "result" } }
 ).Result,
 EndDateRelative = "8760h",
});
var spAuth = new Assignment("myauth", new AssignmentArgs
{
 Scope = registry.Id,
 RoleDefinitionName = "acrpush",
 PrincipalId = sp.Id,
});
var registryInfo = new Docker.ImageRegistry
{
 Server = registry.LoginServer,
 Username = sp.ApplicationId,
 Password = spAuth.Id.Apply(_ => spPassword.Value),
};

Learn More About ACR

ACR offers advanced functionality not shown here, including configuring retention policies, storage account details, and geo-replication options. For more details on these and more, see Pulumi’s Azure-Native or Azure API documentation, or Azure’s product page.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

Google Container Registry (GCR)

Google Container Registry (GCR) enables you to store, manage, and secure your Docker container images. It includes built-in security vulnerability scanning and fast, high-availability access by storing images in regional private repositories across the world. This makes it easy to deploy custom application images to Google Kubernetes Engine (GKE), App Engine, or Cloud Run. Each account has a default registry per project per region, backed by Google Cloud Storage (GCS), and each registry is capable of storing many images and many versions.

Create a New Project

To start, create a new project and ensure it is configured to use your Google Cloud account, then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var gcp = require("@pulumi/gcp");
var docker = require("@pulumi/docker");

// [Placeholder 1: Create a private GCR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as gcp from "@pulumi/gcp";
import * as docker from "@pulumi/docker";

// [Placeholder 1: Create a private GCR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import pulumi
import pulumi_gcp as gcp
import pulumi_docker as docker

# [Placeholder 1: Create a private GCR registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi-gcp/sdk/v2/go/gcp/container"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private GCR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Docker;
using Pulumi.Gcp.Container;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private GCR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Ensure Your GCR Registry is Ready

Google Cloud automatically provisions a managed project-wide registry as needed. To ensure our Google Cloud registry is ready, we allocate a registry object, and then fetch the autogenerated repository URL that we can use for images:

// Create a private GCR registry.
var registry = new gcp.container.Registry("my-registry");
var registryUrl = registry.id.apply(_ =>
 gcp.container.getRegistryRepository().then(reg => reg.repositoryUrl));

// Create a private GCR registry.
const registry = new gcp.container.Registry("my-registry");
const registryUrl = registry.id.apply(_ =>
 gcp.container.getRegistryRepository().then(reg => reg.repositoryUrl));

# Create a private GCR repository.
registry = gcp.container.Registry('my-registry')
registry_url = registry.id.apply(lambda _: gcp.container.get_registry_repository().repository_url)

// Create a private GCR registry.
registry, err := container.NewRegistry(ctx, "my-registry", nil)
if err != nil {
 return err
}
registryUrl := registry.ID().ApplyString(func(_ string) (string, error) {
 rep, err := container.GetRegistryRepository(ctx, nil)
 if err != nil {
 return "", err
 }
 return rep.RepositoryUrl, nil
})

// Create a private GCR registry.
var registry = new Registry("my-registry");
var registryUrl = registry.Id.Apply(async _ => {
 return (await GetRegistryRepository.InvokeAsync()).RepositoryUrl;
});

Below we will append the container repository name to this registry URL.

Authenticate with the Google Cloud SDK

Google Cloud offers several mechanisms to authenticate to your registry; however, the most secure and preferred option is to use the Google Cloud CLI, gcloud, as a Docker credential helper. After setting up gcloud on your client, no explicit authentication is required in your code — instead, the ambient gcloud authentication settings will be used instead. As a result, we can leave the authentication information blank:

// Get registry info (creds and endpoint).
var imageName = registryUrl.apply(url => `${url}/myapp`);
var registryInfo = undefined; // use gcloud for authentication.

// Get registry info (creds and endpoint).
const imageName = registryUrl.apply(url => `${url}/myapp`);
const registryInfo = undefined; // use gcloud for authentication.

# Get registry info (creds and endpoint).
image_name = registry_url.apply(lambda url: f'{url}/myapp')
registry_info = None # use gcloud for authentication.

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registryUrl)
registryInfo := docker.ImageRegistryArgs{} // use gcloud for authentication.

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registryUrl}/myapp");
// Omit registryInfo and use gcloud for authentication.

Learn More About GCR

GCR uses Google Cloud Storage to store images, which may be configured separately, including configuring IAM and lifecycle policies. For more details on this, please refer to Pulumi’s Google Cloud API or Google’s own Container Registry documentation.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

DigitalOcean Container Registry

DigitalOcean’s Container Registry is an easy way to store and manage private container images for your applications for use with DigitalOcean’s managed Kubernetes service.

Create a New Project

To start, create a new project and ensure it is configured to use your DigitalOcean account and then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var digitalocean = require("@pulumi/digitalocean");
var docker = require("@pulumi/docker");
var pulumi = require("@pulumi/pulumi");

// [Placeholder 1: Create a private DigitalOcean container registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as digitalocean from "@pulumi/digitalocean";
import * as docker from "@pulumi/docker";
import * as pulumi from "@pulumi/pulumi";

// [Placeholder 1: Create a private DigitalOcean container registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import pulumi
import pulumi_digitalocean as digitalocean
import pulumi_docker as docker

# [Placeholder 1: Create a private DigitalOcean container registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "encoding/json"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-digitalocean/sdk/v2/go/digitalocean"
 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private DigitalOcean container registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.DigitalOcean;
using Pulumi.Docker;
using Newtonsoft.Json;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private DigitalOcean container registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Create a DigitalOcean Container Registry

Next, declare a new DigitalOcean container registry resource:

// Create a private DigitalOcean Container Registry.
var registry = new digitalocean.ContainerRegistry("my-reg", {
 subscriptionTierSlug: "starter",
});

// Create a private DigitalOcean Container Registry.
const registry = new digitalocean.ContainerRegistry("my-reg", {
 subscriptionTierSlug: "starter",
});

# Create a private DigitalOcean Container Registry.
repo = digitalocean.ContainerRegistry('my-reg',
 subscription_tier_slug='starter',
)

// Create a private DigitalOcean Container Registry.
registry, err := digitalocean.NewContainerRegistry(ctx, "my-reg",
 &digitalocean.ContainerRegistryArgs{
 // SubscriptionTierSlug: pulumi.String("starter"),
 })
if err != nil {
 return err
}

// Create a private DigitalOcean Container Registry.
var registry = new ContainerRegistry("my-reg", new ContainerRegistryArgs
{
 SubscriptionTierSlug = "starter"
});

Authenticate with Temporary Credentials

DigitalOcean supports generating temporary read or read/write credentials that Docker can use to authenticate with your new private container registry, which we’ll now use. This resource returns a new credentials file similar to what the docker login command generates, and we can parse it and base64 decode its contents to discover the temporary username and password. Here, we gather the username and password plus the registry’s URL endpoint in preparation for building and publishing the container image to the newly provisioned registry:

// Get registry info (creds and endpoint) so we can build/publish to it.
var imageName = registry.endpoint.apply(s => `${s}/myapp`);
var creds = new digitalocean.ContainerRegistryDockerCredentials("my-reg-creds", {
 registryName: registry.name,
 write: true,
});
var registryInfo = pulumi.all(
 [creds.dockerCredentials, registry.serverUrl]
).apply(([authJson, serverUrl]) => {
 // We are given a Docker creds file; parse it to find the temp username/password.
 var auths = JSON.parse(authJson);
 var authToken = auths["auths"][serverUrl]["auth"];
 var decoded = Buffer.from(authToken, "base64").toString();
 var [username, password] = decoded.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: serverUrl,
 username: username,
 password: password,
 };
});

// Get registry info (creds and endpoint) so we can build/publish to it.
const imageName = registry.endpoint.apply(s => `${s}/myapp`);
const creds = new digitalocean.ContainerRegistryDockerCredentials("my-reg-creds", {
 registryName: registry.name,
 write: true,
});
const registryInfo = pulumi.all(
 [creds.dockerCredentials, registry.serverUrl]
).apply(([authJson, serverUrl]) => {
 // We are given a Docker creds file; parse it to find the temp username/password.
 const auths = JSON.parse(authJson);
 const authToken = auths["auths"][serverUrl]["auth"];
 const decoded = Buffer.from(authToken, "base64").toString();
 const [username, password] = decoded.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: serverUrl,
 username: username,
 password: password,
 };
});

# Get registry info (creds and endpoint).
image_name = registry.endpoint.apply(lambda s: f'{s}/myapp')
def getRegistryInfo(info):
 # We are given a Docker creds file; parse it to find the temp username/password.
 auth_json = info[0]
 auths = json.loads(auth_json)
 server_url = info[1]
 auth_token = auths['auths'][server_url]['auth']
 decoded = base64.b64decode(auth_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception('Invalid credentials')
 return docker.ImageRegistry(server_url, parts[0], parts[1])
creds = digitalocean.ContainerRegistryDockerCredentials('reg-creds',
 registry_name=registry.name,
 write=True,
)
registry_info = pulumi.Output.all(
 creds.docker_credentials, registry.server_url).apply(getRegistryInfo)

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registry.Endpoint)
creds, err := digitalocean.NewContainerRegistryDockerCredentials(ctx, "my-reg-creds",
 &digitalocean.ContainerRegistryDockerCredentialsArgs{
 RegistryName: registry.Name,
 Write: pulumi.Bool(true),
 },
)
if err != nil {
 return err
}

registryInfo := pulumi.All(creds.DockerCredentials, registry.ServerUrl).ApplyT(
 func(args []interface{}) (docker.ImageRegistry, error) {
 // We are given a Docker creds file; parse it to find the temp username/password.
 authJson := args[0].(string)
 serverUrl := args[1].(string)
 var auths map[string]interface{}
 if err := json.Unmarshal([]byte(authJson), &auths); err != nil {
 return docker.ImageRegistry{}, err
 }
 authMap := auths["auths"].(map[string]interface{})
 authToken := authMap[serverUrl].(map[string]interface{})["auth"].(string)
 decoded, err := base64.StdEncoding.DecodeString(authToken)
 if err != nil {
 return docker.ImageRegistry{}, err
 }
 parts := strings.Split(string(decoded), ":")
 if len(parts) != 2 {
 return docker.ImageRegistry{}, errors.New("Invalid credentials")
 }
 return docker.ImageRegistry{
 Server: serverUrl,
 Username: parts[0],
 Password: parts[1],
 }, nil
 },
).(docker.ImageRegistryOutput)

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registry.Endpoint}/myapp");
var registryCreds = new ContainerRegistryDockerCredentials("my-reg-creds",
 new ContainerRegistryDockerCredentialsArgs
 {
 RegistryName = registry.Name,
 Write = true,
 });
var registryInfo = Output.All(
 registryCreds.DockerCredentials, registry.ServerUrl).
 Apply(args =>
 {
 var authJson = args[0];
 var serverUrl = args[1];
 dynamic auths = JsonConvert.DeserializeObject(authJson);
 var authToken = auths["auths"][serverUrl]["auth"];
 var decoded = ASCIIEncoding.ASCII.GetString(authToken);

 var parts = decoded.Split(':');
 if (parts.Length != 2)
 {
 throw new Exception("Invalid credentials");
 }

 return new ImageRegistry
 {
 Server = serverUrl,
 Username = parts[0],
 Password = parts[1],
 };
 });

Learn More About DigitalOcean Container Registry

Each DigitalOcean account may have just a single registry, so you are likely to want to provision that in a separate stack for most “real world” examples. Also, this project uses the lowest subscription tier, "starter", which is inexpensive but quite limited (you may only store a single repository of images). If you would like to use an advanced tier or configure any other options, please refer to Pulumi’s DigitalOcean API or DigitalOcean’s product documentation.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

Docker Hub

Docker offers the Docker Hub as an easy way to store and consume public or private container images from a centralized location to any cloud. To start building and publishing to Docker Hub with infrastructure as code, create a new project and scaffold it with the imports and overall program structure that we will fill in one piece at a time:

var docker = require("@pulumi/docker");
var pulumi = require("@pulumi/pulumi");

// [Placeholder 1: Get registry info (creds and endpoint).]

// [Placeholder 2: Build and publish the container image.]

import * as docker from "@pulumi/docker";
import * as pulumi from "@pulumi/pulumi";

// [Placeholder 1: Get registry info (creds and endpoint).]

// [Placeholder 2: Build and publish the container image.]

import pulumi
import pulumi_docker as docker

# [Placeholder 1: Get registry info (creds and endpoint).]

# [Placeholder 2: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "encoding/json"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Get registry info (creds and endpoint).]

 // [Placeholder 2: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Get registry info (creds and endpoint).]

 // [Placeholder 2: Build and publish the container image.]
 }
}

For purposes of this example, you will need to manually provision a repository, and select whether to make it public or private, via the Docker Hub’s UI. Refer to the Docker Hub documentation for details on how to do this. Once you have an account and have set up your registry, create an access token. We will now use your username and this access token to configure authentication so that you can build and publish to your registry.

Because the repository was provisioned outside of the purview of Pulumi, we can jump straight to authenticating. To do this, we will simply use the Pulumi configuration system to store and retrieve authentication information, using the token created above:

// Fetch the Docker Hub auth info from config.
var config = new pulumi.Config();
var username = config.require("dockerUsername");
var password = config.requireSecret("dockerPassword");

// Populate the registry info (creds and endpoint).
var imageName = `${username}/myapp`;
var registryInfo = {
 server: "docker.io",
 username: username,
 password: password,
};

// Fetch the Docker Hub auth info from config.
const config = new pulumi.Config();
const username = config.require("dockerUsername");
const password = config.requireSecret("dockerPassword");

// Populate the registry info (creds and endpoint).
const imageName = `${username}/myapp`;
const registryInfo = {
 server: "docker.io",
 username: username,
 password: password,
};

# Fetch the Docker Hub auth info from config.
config = pulumi.Config()
username = config.require('dockerUsername')
accessToken = config.require_secret('dockerAccessToken')

# Populate the registry info (creds and endpoint).
image_name=f'{username}/myapp',
def get_registry_info(token):
 return docker.ImageRegistry(
 server='docker.io',
 username=username,
 password=token,
 )
registry_info=accessToken.apply(get_registry_info)

// Fetch the Docker Hub auth info from config.
username := config.Require(ctx, "dockerUsername")
password := config.RequireSecret(ctx, "dockerPassword")

// Populate the registry info (creds and endpoint).
imageName := pulumi.String(username + "/myapp")
registryInfo := docker.ImageRegistryArgs{
 Server: pulumi.String("docker.io"),
 Username: pulumi.String(username),
 Password: password.(pulumi.StringOutput),
}

// Fetch the Docker Hub auth info from config.
var config = new Pulumi.Config();
var username = config.Require("dockerUsername");
var password = config.RequireSecret("dockerPassword");

// Populate the registry info (creds and endpoint).
var imageName = $"{username}/myapp";
var registryInfo = new ImageRegistry
{
 Server = "docker.io",
 Username = username,
 Password = password,
};

After writing this code, add your Docker username and password to your Pulumi project:

$ pulumi config set dockerUsername [YOUR USERNAME]
$ pulumi config set --secret dockerAccessToken [YOUR TOKEN]

There are of course other ways to configure this authentication information, however, the Pulumi secrets system ensures that your token is encrypted and safe to use.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

At this stage, we can run pulumi up to check that the program works and to provision any necessary cloud resources.

Build and Publish Your Container

Now we are ready to build and publish your container image to the chosen registry. The Docker provider’s Image component internally uses the Docker engine to perform the necessary steps to carry this out, including building, tagging, capturing, and streaming container build logs as progress is made and pushing the final result.

Simply pass the path to your application’s Dockerfile as the build context, the registry’s URL as the image’s name, and the registry configuration object built up earlier to facilitate authentication:

// [Existing imports ...]
var docker = require("@pulumi/docker");

// [Registry configuration as shown above ...]

// Build and publish the container image.
var image = new docker.Image("my-image", {
 build: "app",
 imageName: imageName,
 registry: registry,
});

// Export the base and specific version image name.
module.exports = {
 baseImageName: image.baseImageName,
 fullImageName: image.imageName,
};

// [Existing imports ...]
import * as docker from "@pulumi/docker";

// [Registry configuration as shown above ...]

// Build and publish the container image.
const image = new docker.Image("my-image", {
 build: "app",
 imageName,
 registry,
});

// Export the base and specific version image name.
export const baseImageName = image.baseImageName;
export const fullImageName = image.imageName;

# [Existing imports ...]
import pulumi_docker as docker

# [Registry configuration as shown above ...]

# Build and publish the container image.
image = docker.Image('my-image',
 build='app',
 image_name=image_name,
 registry=registry_info,
)

# Export the base and specific version image name.
pulumi.export('baseImageName', image.base_image_name)
pulumi.export('fullImageName', image.image_name)

// [Existing imports ...]

import "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Registry configuration as shown above ...]

 // Build and publish the container image.
 image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
 Build: &docker.DockerBuildArgs{Context: pulumi.String("app")},
 ImageName: imageName,
 Registry: registryInfo,
 })

 // Export the base and specific version image name.
 ctx.Export("baseImageName", image.BaseImageName)
 ctx.Export("fullImageName", image.ImageName)
 return nil
 })
}

// [Existing imports ...]

using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Registry configuration as shown above ...]

 // Build and publish the container image.
 var image = new Image("my-image", new ImageArgs
 {
 Build = new DockerBuild { Context = "app" },
 ImageName = imageName,
 Registry = registryInfo,
 });

 // Export the base and specific version image name.
 return new Dictionary<string, object>
 {
 { "baseImageName", image.BaseImageName },
 { "fullImageName", image.ImageName },
 };
 });
}

Now let’s run pulumi up. If you haven’t already done so, this will provision the cloud resources, as well as build/publish the container image. As the build runs, you will see Docker build output streamed to your terminal:

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack project-dev created
 ... cloud-specific resources omitted ...
 + └─ docker:image:Image my-image created

Outputs:
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 fullImageName: "...cloud-specific url.../my-repo-dc811b0:78d0fce7c2450c15a6153b6b11208fcb6b9edea7bb7ef3b7b6194f3fc101a170"

Also note the base and versioned image URLs are exported as stack outputs. These are optional but have three benefits:

As we see here, the CLI will print them after each deployment.
The CLI can fetch them on-demand, e.g., ‘pulumi stack output fullImageName’, making it easy to script access to these container images.
The Pulumi StackReference component can be used to depend on this stack from another, allowing us to build higher levels of infrastructure that consume these images.

As an example of (2) in action, let’s run the image locally using the exported image name:

$ pulumi stack output fullImageName
...cloud-specific url.../my-repo-dc811b0:78d0fce7c2450c15a6153b6b11208fcb6b9edea7bb7ef3b7b6194f3fc101a170

Note that this requires that our client can pull from our provisioned registry, which may require cloud-specific authentication first.

To redeploy changes to the Dockerfile or anything that image depends on, including application code, rerun pulumi up. It will detect the differences, rebuild and re-push the minimal layer changes necessary to update and retag the repository image, and then export the resulting image name.

To see this in action, change the Dockerfile’s contents to:

FROM nginx
RUN echo "<h1>Hello, World -- from Pulumi!</h1>" > \
 /usr/share/nginx/html/index.html

And then rerun Pulumi – notice how it detects the change, updates the image, and the versioned image hash changes (indicated by the tilde ~):

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack project-dev created
 ~ └─ docker:image:Image my-image updated

Outputs:
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 ~ fullImageName: "...cloud-specific url.../my-repo-dc811b0:78d0fce7c2450c15a6153b6b11208fcb6b9edea7bb7ef3b7b6194f3fc101a170"

The Pulumi Docker Image component supports a number of additional options to control its behavior, including passing build arguments, environment variables, extra options for the Docker build, as well as various controls for image caching. For full details on each of these, refer to the image API documentation.

Consume the Container Image

The same container image URLs exported above can be used as inputs to other resources, including infrastructure that will run your container inside of a container orchestration system such as Kubernetes, Amazon ECS, and so on.

This article assumes you already have a containerized environment to deploy to, like a Kubernetes cluster, and have configured your project accordingly. If not, you can provision one using Pulumi first. Pulumi supports many clouds and infrastructure resources, but here are a few starting points to get up and running with:

This example demonstrates deploying our Nginx web server as a load balanced service within Kubernetes. To do so, we’ll declare our Kubernetes configuration, right inside of our existing program defined above, and export its resulting IP address:

// [Previous imports...]
var k8s = require("@pulumi/kubernetes");

// [Registry, build, etc., code from above...]

// Create a load balanced Kubernetes service using this image, and export its IP.
var appLabels = { app: "myapp" };
var appDep = new k8s.apps.v1.Deployment("app-dep", {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 3,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: "myapp",
 image: image.imageName,
 }],
 },
 },
 },
});
var appSvc = new k8s.core.v1.Service("app-svc", {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 selector: appLabels,
 },
});

module.exports = {
 // [Previous exports...]
 appIp: appSvc.status.loadbalancer.ingress[0].ip,
};

// [Previous imports...]
import * as k8s from "@pulumi/kubernetes";

// [Registry, build, etc, code from above...]

// Create a load balanced Kubernetes service using this image, and export its IP.
const appLabels = { app: "myapp" };
const appDep = new k8s.apps.v1.Deployment("app-dep", {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 3,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: "myapp",
 image: image.imageName,
 }],
 },
 },
 },
});
const appSvc = new k8s.core.v1.Service("app-svc", {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 selector: appLabels,
 },
});
export const appIp = appSvc.status.loadBalancer.ingress[0].ip;

# [Previous imports...]
import pulumi_kubernetes as k8s

# [Registry, build, etc, code from above...]

# Create a load balanced Kubernetes service using this image, and export its IP.
app_labels = { 'app': 'myapp' }
app_dep = k8s.apps.v1.Deployment('app-dep',
 spec={
 'selector': { 'matchLabels': app_labels },
 'replicas': 3,
 'template': {
 'metadata': { 'labels': app_labels },
 'spec': {
 'containers': [{
 'name': 'myapp',
 'image': image.image_name,
 }],
 },
 },
 },
)
app_svc = k8s.core.v1.Service('app-svc',
 metadata={ 'labels': app_labels },
 spec={
 'type': 'LoadBalancer',
 'ports': [{ 'port': 80, 'targetPort': 80, 'protocol': 'TCP' }],
 'selector': app_labels,
 }
)
pulumi.export('appIp', app_svc.status.apply(lambda s: s.loadbalancer.ingress[0].ip))

package main

import (
 // [Previous imports...]
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Registry, build, etc, code from above...]

 // Create a load balanced Kubernetes service using this image, and export its IP.
 appLabels := pulumi.StringMap{"app": pulumi.String("myapp")}
 _, deperr := appsv1.NewDeployment(ctx, "app-dep", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{Labels: appLabels},
 Spec: appsv1.DeploymentSpecArgs{
 Selector: &metav1.LabelSelectorArgs{MatchLabels: appLabels},
 Replicas: pulumi.Int(3),
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{Labels: appLabels},
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("myapp"),
 Image: image.ImageName,
 },
 },
 },
 },
 },
 })
 if deperr != nil {
 return deperr
 }
 appSvc, svcerr := corev1.NewService(ctx, "app-svc", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{Labels: appLabels},
 Spec: &corev1.ServiceSpecArgs{
 Type: pulumi.String("LoadBalancer"),
 Ports: corev1.ServicePortArray{
 corev1.ServicePortArgs{Port: pulumi.Int(80)},
 },
 Selector: appLabels,
 },
 })
 if svcerr != nil {
 return svcerr
 }
 ctx.Export("appIp", appSvc.Status.ApplyT(func(status *corev1.ServiceStatus) *string {
 return status.LoadBalancer.Ingress[0].Ip
 }))

 return nil
 })
}

// [Previous imports...]
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Apps.V1;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Registry, build, etc, code from above...]

 // Create a load balanced Kubernetes service using this image, and export its IP.
 var appLabels = new InputMap<string>
 {
 { "app", "myapp" }
 };
 var appDep = new Pulumi.Kubernetes.Apps.V1.Deployment("app-dep", new DeploymentArgs
 {
 Spec = new DeploymentSpecArgs
 {
 Selector = new LabelSelectorArgs
 {
 MatchLabels = appLabels
 },
 Replicas = 3,
 Template = new PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels = appLabels,
 },
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "myapp",
 Image = image.ImageName,
 }
 }
 }
 }
 }
 });
 var appSvc = new Pulumi.Kubernetes.Core.V1.Service("app-svc", new Pulumi.Kubernetes.Types.Inputs.Core.V1.ServiceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels = appLabels
 },
 Spec = new ServiceSpecArgs
 {
 Type = "LoadBalancer",
 Ports =
 {
 new ServicePortArgs
 {
 Port = 80,
 TargetPort = 80
 }
 },
 Selector = appLabels
 }
 });

 return new Dictionary<string, object>
 {
 // [Previous exports...]
 { "appIp", appSvc.Status.Apply(status => status.LoadBalancer.Ingress[0].Ip) },
 };
 }
}

Notice here that we are referring to the resulting image name from our service specification. All we need to do run a ‘pulumi up’ and Pulumi will now build, publish, and consume our application’s container image from within our Kubernetes cluster:

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack project-dev created
 ... prior output omitted ...
 + ├─ kubernetes:apps/v1:Deployment app-dep created
 + └─ kubernetes:core/v1:Service app-svc created

Outputs:
 appIp : "34.82.32.166"
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 fullImageName: "...cloud-specific url.../my-repo-dc811b0:fcbbd958636e4f8ac02568db7dbf8b89a043e210e917432cf12649aaedfa266c"

If this is the first time you’ve run pulumi up, all of the prior resources will be provisioned first, and in any case, the Kubernetes resources will follow.

Now that our deployment has completed, we can curl the endpoint:

$ curl $(pulumi stack output appIp)
<h1>Hello, World -- from Pulumi!</h1>

This image name is the versioned one, meaning that any time a new version is created, Pulumi will detect that it needs to update the Kubernetes deployment which references it, triggering a rollout of the new deployed image. As such, we just need to run pulumi up anytime the application or its Dockerfile changes and Pulumi will detect and trigger a deployment to Kubernetes.

To see this in action, let’s change our Dockerfile’s contents once more:

FROM nginx
RUN echo "<h1>Hello, World -- from Pulumi and Kubernetes!</h1>" > \
 /usr/share/nginx/html/index.html

And then rerun Pulumi – notice that both the image and the Kubernetes deployment has changed this time:

$ pulumi up
Updating (dev)

 Type Name Status Info
 + pulumi:pulumi:Stack project-dev created
 ~ ├─ docker:image:Image my-image updated
 ~ └─ kubernetes:core/v1:Deployment app-dep updated [diff: ~spec]

Outputs:
 appIp : "34.82.32.166"
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 fullImageName: "...cloud-specific url.../my-repo-dc811b0:36919b56f6b2dedfa879815443c9964d17a89506caed25025328b1d0204a083f"

Finally, if we re-curl the endpoint, we will now see that the updated container is now live!

$ curl $(pulumi stack output appIp)
<h1>Hello, World -- from Pulumi and Kubernetes!</h1>

And there we go: we have gone from a Dockerfile to a published container image in a container registry of our choosing, which has been used to spin up a load balanced Kubernetes service, and which is now ready for fully automated continuous deployments.

Wrapping Up

In this article, we’ve seen how easy it is to build, publish, and use container images in many popular public and private container registry options. We have seen how to provision new registries using infrastructure as code and how easy it is to trigger deployments of application updates by running a single pulumi up command.

All of these steps were manually run from a CLI; however, a natural next step is to wire the entire process up to a CI/CD system such as GitHub Actions, GitLab Pipelines, Jenkins, Spinnaker, or one of the many available options, so that you can deploy continuously as you merge code. For even more advanced scenarios, we might want to build a custom program such as a CLI that uses the Automation API to perform these actions behind a simpler, purpose-built interface.

Although we’ve shown a very simple set of infrastructure resources, it would be natural to extend these examples by provisioning other ancillary services that your application needs, including databases, pub/sub topics, queues, metrics and dashboards, and more. Using an infrastructure as code approach to building, publishing, and consuming your container images means you can incrementally add on such infrastructure to the base code shown above and reference them from your containers easily with automatic dependency tracking.

The complete examples in this article are available on GitHub. Pulumi is open source and free to get started with — give it a try, and get up and running with infrastructure as code for your containers today!

Running Container Images in AWS Lambda

Mikhail Shilkov — Tue, 01 Dec 2020 00:00:00 +0000

TL;DR — To run a container image in AWS Lambda, build an OCI image that implements the Lambda Runtime API (most teams start from an AWS-provided base image), push it to Amazon ECR, and create a Lambda function with packageType: "Image" pointing at the image URI. Lambda containers support images up to 10 GB, up to 10 GB of memory, up to 10 GB of /tmp ephemeral storage, and a 15-minute execution ceiling. Pulumi automates the build, push, and function wiring in a single program. Pick Lambda containers when your workload is event-driven and bursty but your dependencies (binaries, ML models, system libraries) outgrow the 250 MB ZIP limit; pick AWS Fargate or ECS when you need long-running tasks, persistent connections, or multi-container pods.

How do I run a container image in AWS Lambda?

You package your function as a Linux container image, push it to Amazon ECR, and create a Lambda function whose packageType is Image. The image must implement the Lambda Runtime API, which is an HTTP protocol Lambda uses to deliver invocation events and collect responses. The simplest path is to start FROM an AWS-provided base image (public.ecr.aws/lambda/nodejs:22, public.ecr.aws/lambda/python:3.13, public.ecr.aws/lambda/java:21, and so on) because they ship with the Runtime Interface Client preinstalled. If you want to use a different OS or runtime, add the open-source Runtime Interface Client yourself.

With Pulumi, one program covers the entire path: ECR repository, Docker build and push, IAM role, Lambda function, and event source. It works in TypeScript, Python, Go, .NET, Java, or YAML. The rest of this post walks through that program with a real video-thumbnailer example.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 context: "./app",
 platform: "linux/amd64",
});

const role = new aws.iam.Role("thumbnailerRole", {
 assumeRolePolicy: aws.iam.getPolicyDocument({
 statements: [{
 actions: ["sts:AssumeRole"],
 principals: [{ type: "Service", identifiers: ["lambda.amazonaws.com"] }],
 }],
 }).then(doc => doc.json),
});

new aws.iam.RolePolicyAttachment("lambdaExecute", {
 role: role.name,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

const thumbnailer = new aws.lambda.Function("thumbnailer", {
 packageType: "Image",
 imageUri: image.imageUri,
 role: role.arn,
 timeout: 900,
 memorySize: 2048,
});

import pulumi_aws as aws
import pulumi_awsx as awsx

repo = awsx.ecr.Repository("repo", force_delete=True)

image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 context="./app",
 platform="linux/amd64",
)

role = aws.iam.Role(
 "thumbnailerRole",
 assume_role_policy=aws.iam.get_policy_document(statements=[{
 "actions": ["sts:AssumeRole"],
 "principals": [{"type": "Service", "identifiers": ["lambda.amazonaws.com"]}],
 }]).json,
)

aws.iam.RolePolicyAttachment(
 "lambdaExecute",
 role=role.name,
 policy_arn=aws.iam.ManagedPolicy.AWS_LAMBDA_EXECUTE,
)

thumbnailer = aws.lambda_.Function(
 "thumbnailer",
 package_type="Image",
 image_uri=image.image_uri,
 role=role.arn,
 timeout=900,
 memory_size=2048,
)

pulumi up builds the image from ./app, pushes it to a fresh ECR repository, and deploys the Lambda — in one step, with one command, with no shell scripts.

What is a Lambda container image?

A Lambda container image is an OCI (or Docker v2) image whose entrypoint speaks the Lambda Runtime API. The image is stored in Amazon ECR, and Lambda pulls it on cold start. Functionally, a container-image function behaves the same as a ZIP-package function: it is invoked by the same triggers (S3, SQS, API Gateway, EventBridge, and so on), it bills the same way (per request and per GB-second), and it is bound by the same 15-minute execution ceiling. The only meaningful difference is how the code is packaged and what limits apply to that package.

Lambda container images were announced at re:Invent 2020 and have been generally available ever since. They are not a separate service — they are a deployment format for the same Lambda runtime.

What are the AWS Lambda container limits in 2026?

The hard limits below apply to every Lambda function regardless of packaging, except for the image-size column, which is specific to container images.

Limit	Container image	ZIP package
Maximum package size	10 GB (uncompressed)	250 MB (unzipped, including layers)
Maximum memory	10,240 MB	10,240 MB
Maximum ephemeral `/tmp`	10,240 MB	10,240 MB
Maximum execution timeout	15 minutes	15 minutes
Maximum payload (sync)	6 MB request / 6 MB response	6 MB request / 6 MB response
Architectures	`x86_64`, `arm64`	`x86_64`, `arm64`
SnapStart support	Not supported for container images	Java, Python, .NET

Sources: Lambda quotas, Lambda images, SnapStart documentation.

A note on cold starts: the public guidance from AWS is that container-image cold starts are typically comparable to ZIP cold starts when the image is well-optimized — AWS caches image layers and uses on-demand block-level loading so only the bytes touched at startup are downloaded. The biggest lever you have is image size: keep the image small, put rarely changed dependencies in lower layers, and prefer a minimal base. SnapStart, which is the most effective cold-start mitigation for Java, Python, and .NET ZIP functions, does not apply to container-image functions; if you need SnapStart, ship a ZIP.

How much do Lambda container images cost?

Container-image functions are billed identically to ZIP functions:

Requests: $0.20 per 1M requests (x86_64 and arm64 alike).
Duration: $0.0000166667 per GB-second on x86_64, $0.0000133334 on arm64 (Graviton2) — roughly 20% lower per GB-second, with up to 34% better price-performance per AWS.
Free tier: 1M requests and 400,000 GB-seconds per month, every month.
Storage: ECR charges $0.10 per GB-month for image storage; data transfer to Lambda within the same Region is free.

Always confirm rates against the live Lambda pricing and ECR pricing pages, since AWS adjusts them periodically.

How do Lambda containers compare to Fargate, ECS, and ZIP Lambdas?

The four options below all run containers, but they sit on a spectrum from “fully event-driven, scale to zero” to “long-running, always on.”

Dimension	Lambda (ZIP)	Lambda (container image)	AWS Fargate	Amazon ECS (EC2)
Packaging	ZIP up to 250 MB	OCI image up to 10 GB	OCI image (no Lambda runtime API needed)	OCI image
Cold start	Milliseconds to seconds; SnapStart available	Seconds (image-size dependent); no SnapStart	Tens of seconds for new tasks	Tens of seconds (capacity dependent)
Max execution time	15 minutes	15 minutes	Unbounded	Unbounded
Scale to zero	Yes	Yes	No (minimum task count)	No (minimum container instances)
Billing	Per-ms duration + per-request	Per-ms duration + per-request	Per-second of vCPU and memory	Per-EC2-instance-hour + per-task overhead
Networking	Optional VPC attachment	Optional VPC attachment	First-class VPC, ENIs, persistent connections	First-class VPC, ENIs, persistent connections
Native event triggers	100+ AWS services	100+ AWS services	EventBridge / Step Functions integrations	EventBridge / Step Functions integrations
Best for	Short, bursty, event-driven workloads	Same as ZIP, but with heavy dependencies, native binaries, large ML models	Long-running services, daemons, persistent workloads	Workloads needing custom EC2 sizing or GPU

The decision tree is short: if your workload runs for less than 15 minutes per invocation and you can fit it in 10 GB, Lambda is almost always cheaper and operationally simpler than Fargate. If you cannot fit in a 250 MB ZIP, use a Lambda container image. If your workload is long-running or needs persistent network connections, choose Fargate or ECS.

How do I deploy a Lambda container image with Pulumi?

Let’s walk through a complete program: a video thumbnailer that runs every time a .mp4 is uploaded to an S3 bucket. The function uses FFmpeg, a binary that is impractical to ship in a 250 MB ZIP, to extract a thumbnail and write it back to the bucket. The full source is in pulumi/examples.

Quick start

Bootstrap a project: pulumi new https://github.com/pulumi/examples/tree/master/aws-ts-lambda-thumbnailer
Edit ./app/Dockerfile and ./app/index.js with your function’s logic.
Deploy: pulumi up.

1. Define a Dockerfile

The container image is based on the AWS-provided Node.js 22 image, which preinstalls the Runtime Interface Client. It also installs the AWS CLI and FFmpeg, copies the handler, and points to index.handler as the entrypoint.

FROM public.ecr.aws/lambda/nodejs:22

RUN dnf install -y tar xz unzip \
 && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
 && unzip awscliv2.zip && ./aws/install \
 && curl -O https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz \
 && tar -xf ffmpeg-release-amd64-static.tar.xz \
 && mv ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ \
 && rm -rf awscliv2.zip aws ffmpeg-*-amd64-static*

COPY index.js ${LAMBDA_TASK_ROOT}
CMD ["index.handler"]

2. Create an S3 bucket for inputs and outputs

import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("bucket");

import pulumi_aws as aws

bucket = aws.s3.Bucket("bucket")

3. Build the image and push it to ECR

Pulumi Crosswalk for AWS wraps the ECR repository, the Docker build, and the push into two resources:

import * as awsx from "@pulumi/awsx";

const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 context: "./app",
 platform: "linux/amd64",
});

import pulumi_awsx as awsx

repo = awsx.ecr.Repository("repo", force_delete=True)

image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 context="./app",
 platform="linux/amd64",
)

The ./app directory holds the Dockerfile and the handler. platform: "linux/amd64" is explicit so the build is reproducible from Apple Silicon and Graviton developer machines. Switch to linux/arm64 if you want the cheaper Graviton runtime.

4. Set up the IAM role

const role = new aws.iam.Role("thumbnailerRole", {
 assumeRolePolicy: aws.iam.getPolicyDocument({
 statements: [{
 actions: ["sts:AssumeRole"],
 principals: [{ type: "Service", identifiers: ["lambda.amazonaws.com"] }],
 }],
 }).then(doc => doc.json),
});

new aws.iam.RolePolicyAttachment("lambdaExecute", {
 role: role.name,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

role = aws.iam.Role(
 "thumbnailerRole",
 assume_role_policy=aws.iam.get_policy_document(statements=[{
 "actions": ["sts:AssumeRole"],
 "principals": [{"type": "Service", "identifiers": ["lambda.amazonaws.com"]}],
 }]).json,
)

aws.iam.RolePolicyAttachment(
 "lambdaExecute",
 role=role.name,
 policy_arn=aws.iam.ManagedPolicy.AWS_LAMBDA_EXECUTE,
)

AWSLambdaExecute is the AWS-managed policy that grants the read/write S3 access and CloudWatch Logs access this function needs. For a real production workload, scope this down to a custom policy that names the bucket explicitly.

5. Configure the Lambda function

const thumbnailer = new aws.lambda.Function("thumbnailer", {
 packageType: "Image",
 imageUri: image.imageUri,
 role: role.arn,
 timeout: 900,
 memorySize: 2048,
 architectures: ["x86_64"],
});

thumbnailer = aws.lambda_.Function(
 "thumbnailer",
 package_type="Image",
 image_uri=image.image_uri,
 role=role.arn,
 timeout=900,
 memory_size=2048,
 architectures=["x86_64"],
)

packageType: "Image" is the only setting that distinguishes a container function from a ZIP function. The imageUri is the digest-pinned URI of the image we just pushed, so any change to the Dockerfile or handler triggers a redeploy on the next pulumi up.

6. Wire up the S3 trigger

new aws.s3.BucketNotification("onNewVideo", {
 bucket: bucket.id,
 lambdaFunctions: [{
 lambdaFunctionArn: thumbnailer.arn,
 events: ["s3:ObjectCreated:*"],
 filterSuffix: ".mp4",
 }],
});

new aws.lambda.Permission("allowBucket", {
 action: "lambda:InvokeFunction",
 function: thumbnailer.name,
 principal: "s3.amazonaws.com",
 sourceArn: bucket.arn,
});

aws.s3.BucketNotification(
 "onNewVideo",
 bucket=bucket.id,
 lambda_functions=[{
 "lambda_function_arn": thumbnailer.arn,
 "events": ["s3:ObjectCreated:*"],
 "filter_suffix": ".mp4",
 }],
)

aws.lambda_.Permission(
 "allowBucket",
 action="lambda:InvokeFunction",
 function=thumbnailer.name,
 principal="s3.amazonaws.com",
 source_arn=bucket.arn,
)

That is the entire program. pulumi up provisions everything in the right order, and a subsequent edit to the Dockerfile rebuilds and rolls out a new image automatically.

Deploy Lambda containers with Pulumi

Build the image, push it to Amazon ECR, wire up the IAM role, and deploy the function from one program in TypeScript, Python, Go, or the language you already use.

Get started

When should I choose Lambda containers over ZIP packages?

Reach for a container image when one or more of the following is true:

Your dependencies exceed 250 MB unzipped. Native binaries (FFmpeg, ImageMagick, headless Chromium), large ML model weights, and full Python data-science stacks all blow past the ZIP limit quickly.
You need a custom OS, runtime, or system library. Lambda’s managed runtimes are good but opinionated. A container lets you pin a specific glibc version, install OS packages, or run a language Lambda doesn’t natively support.
Your team’s CI/CD already builds containers. Reusing your existing image-build pipeline (and its scanning, signing, and SBOM tooling) is often easier than maintaining a parallel ZIP-packaging pipeline.
You want a single artifact format across Lambda, Fargate, and ECS. A container image that runs in Lambda for short bursts can also run in Fargate for long jobs — convenient for hybrid workloads.

Stick with ZIP when your code is small, you want SnapStart, or you want the fastest possible cold starts.

You might also like:

Conclusion

AWS Lambda’s container-image support brings the industry-standard packaging format to event-driven serverless functions, and Pulumi covers the full deploy (ECR repository, image build, IAM role, function, and event source) in one program in the language you already use. If you have outgrown the ZIP package, this is the pattern you want.

Watch a demo of the thumbnailer below.

Next steps:

Browse the full Lambda + container image example in pulumi/examples.
Read the Pulumi AWS Lambda guide for event sources, IAM patterns, and packaging tips.
See AWS Lambda SnapStart with Pulumi if you need faster cold starts on a ZIP-packaged Java, Python, or .NET function.
Cut x86 costs with Lambda functions powered by Graviton2.
Get started with Pulumi for AWS today.

Pulumi container images now available on Amazon ECR Public

Paul Stack — Tue, 01 Dec 2020 00:00:00 +0000

At re:Invent, the AWS team unveiled the new Amazon Elastic Container Registry Public (Amazon ECR Public), creating a new option for users in publishing and pulling public container images. Pulumi fully supports Amazon ECR Public in two ways:

Official Pulumi container images are available today on Amazon ECR Public.
Pulumi is the easiest way to package and publish your container images, and we’ll support publishing your container images to Amazon ECR Public very soon.

If you want to learn more about Pulumi and building resources in AWS, join one of our upcoming workshops.

Pulumi publishes official container images for running Pulumi along with the desired Pulumi programming language runtimes. These containers are used directly by developers as the foundation for process automation, such as GitHub Actions; in custom container environments with Pulumi’s Automation API; and in Kubernetes-based solutions, like the Pulumi Kubernetes Operator. These images are publicly available via Docker Hub and will continue to be available there.

With the unveiling of Amazon ECR Public, we want to make sure Pulumi’s official container images are always available to ensure that our users can ultimately choose the right container image that suits their needs. Going forward, Pulumi’s official container images will be available on Amazon ECR Public.

We’ve published our Pulumi container images in this new repository, and we plan to publish to both the Docker Hub and Amazon ECR Public repositories for all new Pulumi releases. You can browse Pulumi images in the Amazon ECR gallery here: https://gallery.ecr.aws/?searchTerm=pulumi.

Try it out!

Here’s how easy it is to grab the latest Pulumi container images from the Amazon ECR Public repository:

$ docker pull public.ecr.aws/pulumi/pulumi:latest

To grab the latest language runtime specific containers, run the following commands:

Python:

$ docker pull public.ecr.aws/pulumi/pulumi-python:latest

Go:

$ docker pull public.ecr.aws/pulumi/pulumi-go:latest

TypeScript / JavaScript:

$ docker pull public.ecr.aws/pulumi/pulumi-nodejs:latest

DotNet:

$ docker pull public.ecr.aws/pulumi/pulumi-dotnet:latest

Finally, you can grab the latest official image for the Pulumi Kubernetes Operator with this command:

$ docker pull public.ecr.aws/pulumi/pulumi-kubernetes-operator:latest

Switching the application stack from PERN to MERN

Vova Ivanov — Fri, 18 Sep 2020 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

In this blog post, we return to the PERN application we previously migrated to Kubernetes and replace the PostgreSQL database with MongoDB. Although it might seem like a difficult task initially, the straightforward design of Pulumi and Kubernetes allows us to easily transition the application form a PERN stack to a MERN one.

The main difference between PostgreSQL and MongoDB is data is stored. PostgreSQL is a relational database that allows users to create complex tables and relations between values. In contrast, MongoDB is a key-value database that stores data in simple collections instead of a relational model. Because the application takes the form of three distinct components–the client, server, and database–almost all of the changes will occur in the database component. In fact, we can reuse most of the existing code and will be able to change the database without additional hassle.

The first step to switching the stack is to clone the PERN Kubernetes example. We’ll use a sparse clone to copy only the aws-ts-k8s-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-k8s-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-typescript.

$ mkdir aws-ts-k8s-mern-voting-app && cd aws-ts-k8s-mern-voting-app
$ pulumi new aws-typescript

The project will require several configuration variables, which we set using pulumi config set. The variables are used to configure the MongoDB user account, initializing the database schema and table, and its region.

$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set aws:region <REGION>

The package.json file lists the libraries used by the project. We will add the following to the dependencies section:

 "@pulumi/eks": "^0.20.0",
 "@pulumi/cloud-aws": "^0.19.0"

As most of our project components are nearly identical to those in the PERN Kubernetes application, we can copy many of them into our directory.

$ cd ..
$ cp -r aws-ts-k8s-voting-app/clientside/ aws-ts-k8s-mern-voting-app/clientside
$ cp -r aws-ts-k8s-voting-app/serverside/ aws-ts-k8s-mern-voting-app/serverside
$ cp -r aws-ts-k8s-voting-app/databaseside/ aws-ts-k8s-mern-voting-app/databaseside

$ cd aws-ts-k8s-voting-app

Hosting MongoDB will require a rework of the database Docker container and startup scripts. Like before, the outer databaseside folder holds our Dockerfile, while the inner database folder contains code.

FROM ubuntu:18.04

WORKDIR /

EXPOSE 5432

RUN apt update && \
 apt install -y mongodb

ADD database /database

CMD [ "/database/startDatabase.sh" ]

Our setup doesn’t require any files, and so the only thing we need to put in the database folder is a startDatabase.sh script that launches the database, configures users, and creates a table.

#!/bin/bash
set -exu
FILE=/persistentVolume/mongod.lock

if test -f "$FILE"; then
 echo "/persistentVolume already contains MongoDB, no need to initialize database."
 mongod --fork --dbpath /persistentVolume/ --bind_ip_all --logpath /persistentVolume/mongoLogs
else
 echo "/persistentVolume is empty, and we need to initialize the MongoDB database."
 mongod --fork --dbpath /persistentVolume/ --bind_ip_all --logpath /persistentVolume/mongoLogs
 echo "use $DATABASE_NAME
 db.choices.insert({ _id: 0, text: \"Tabs\", vote_count: 0 })
 db.choices.insert({ _id: 1, text: \"Spaces\", vote_count: 0 })
 db.createUser(
 {
 user: \"$USER_NAME\",
 pwd: \"$USER_PASSWORD\",
 roles: [ { role: \"readWrite\", db: \"$DATABASE_NAME\" } ]
 }
 )" | mongo
fi

while true; do
 sleep 3600
done

With the database in place, all that is left is to make some small MongoDB-specific changes to how the client and server components send queries.

The server component will be changed to use a model.js file, which acts as a blueprint for the database’s choices.

const mongoose = require("mongoose");

let Choice = new mongoose.Schema({
 _id: Number,
 text: {
 type: String
 },
 vote_count: {
 type: Number
 },
});

module.exports = mongoose.model("choice", Choice);

Likewise, the db.js file will be modified to create a specialized to our database.

const mongoose = require("mongoose");

const url = "mongodb://"
 + process.env["MONGODB_ADDRESS"]
 + ":" + process.env["MONGODB_PORT"]
 + "/" + process.env["DATABASE_NAME"]

mongoose.connect(url, {
 useNewUrlParser: true,
 user: process.env["USER_NAME"],
 pass: process.env["USER_PASSWORD"]
});

const connection = mongoose.connection;
module.exports = connection;

And finally, the index.js file holding the main code of our server will be modified to send queries to the MongoDB database.

const express = require("express");
const app = express();
const cors = require("cors");
const connection = require("./db");

app.use(cors());
app.use(express.json());

connection.once("open", function() {
 console.log("Connection opened to database!");
})

let Choice = require("./model");

app.get("/voting", async (request, response) => {
 console.log("Get all request");
 Choice.find(function(error, result) {
 if (error) {
 console.log(error);
 } else {
 response.json(result);
 }
 });
});

app.post("/voting/:id", async (request, response) => {
 const { id } = request.params;
 console.log("Casting vote for: " + id);

 Choice.findById(id, function(error, result) {
 if (!result) {
 response.status(404).send("Choice not found");
 } else {
 result.vote_count = result.vote_count + 1;
 result.save().then(result => {
 response.json("Vote successfully cast");
 }).catch(error => {
 response.status(400).send("Failed to cast vote");
 });
 }
 });
});

app.listen(5000, async () => {
 console.log("server has started on port 5000");
});

The VotingComponent.js file in the client will be slightly modified to reference choices by their MongoDB _id parameter.

import React, { Fragment, useEffect, useState } from "react";
const fullServerUrl = "http://" + window.SERVER_URL + ":5000/voting";

const ListChoices = () => {
 const [choices, setChoices] = useState([]);

 const getChoices = async () => {...};

 useEffect(() => {...}, []);

 return (
 <Fragment>
 <h1 className="text-center mt-5">Pulumi Voting App</h1>
 {" "}
 <table className="table mt-5 text-center">
 <thead>
 <tr>
 <th>Choice</th>
 <th>Votes</th>
 </tr>
 </thead>
 <tbody>
 {choices.map(choice => (
 <tr key={choice._id}>
 <td>{choice.text}</td>
 <td>{choice.vote_count}</td>
 <td>
 <button
 className="btn btn-success"
 onClick={() => castVote(choice._id)}
 >
 Vote
 </button>
 </td>
 </tr>
 ))}
 </tbody>
 </table>
 </Fragment>
 );
};

async function castVote(id) {...}

export default ListChoices;

Now that, server, client, and database have been fully converted to use MongoDB, we can make the last remaining change in the infrastructure by updating the database deployment and service.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const sqlUserName = config.require("sql-user-name");
const sqlUserPassword = config.requireSecret("sql-user-password");
const region = aws.config.region;

const eksCluster = new eks.Cluster(...);

const ebsVolume = new aws.ebs.Volume(...);

const databaseAppName = "database-side-service";
const databaseAppLabels = { appClass: databaseAppName };
const databaseDeployment = new k8s.apps.v1.Deployment(databaseAppName, {
 metadata: { name: databaseAppName, labels: databaseAppLabels },
 spec: {
 replicas: 1,
 selector: { matchLabels: databaseAppLabels },
 template: {
 metadata: { labels: databaseAppLabels },
 spec: {
 containers: [{
 name: databaseAppName,
 image: awsx.ecr.buildAndPushImage("database-side-service", "./databaseside").image(),
 ports: [{ name: "http", containerPort: 27017 }],
 env: [
 { name: "DATABASE_NAME", value: databaseName },
 { name: "USER_NAME", value: sqlUserName },
 { name: "USER_PASSWORD", value: sqlUserPassword },

 ],
 volumeMounts: [{
 name: "persistent-volume",
 mountPath: "/persistentVolume"
 }],
 resources: {
 limits: {
 memory: "1Gi",
 cpu: "1000m"
 }
 }
 }],
 volumes: [{
 name: "persistent-volume",
 awsElasticBlockStore: {
 volumeID: ebsVolume.id,
 },
 }],
 affinity: {
 nodeAffinity: {
 requiredDuringSchedulingIgnoredDuringExecution: {
 nodeSelectorTerms: [{
 matchExpressions: [{
 key: "failure-domain.beta.kubernetes.io/zone",
 operator: "In",
 values: [ebsVolume.availabilityZone],
 }]
 }]
 }
 }
 }
 }
 },
 strategy: {
 type: "Recreate"
 }
 }}, {
 deleteBeforeReplace: true,
 provider: eksCluster.provider,
});

const databasesideListener = new k8s.core.v1.Service("database-side-listener", {
 metadata: { labels: databaseDeployment.metadata.labels },
 spec: {
 type: "ClusterIP",
 ports: [{ port: 27017, targetPort: "http" }],
 selector: databaseAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

const mongodbAddress = databasesideListener.spec.clusterIP;

const serverAppName = "server";
const serverAppLabels = { appClass: serverAppName };
const serverDeployment = new k8s.apps.v1.Deployment("server-side-service", {
 metadata: { labels: serverAppLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: serverAppLabels },
 template: {
 metadata: { labels: serverAppLabels },
 spec: {
 containers: [{
 name: serverAppName,
 image: awsx.ecr.buildAndPushImage("server-side-service", "./serverside").image(),
 ports: [{ name: "http", containerPort: 5000 }],
 env: [
 { name: "USER_NAME", value: sqlUserName },
 { name: "USER_PASSWORD", value: sqlUserPassword },
 { name: "MONGODB_ADDRESS", value: mongodbAddress },
 { name: "MONGODB_PORT", value: String(27017) },
 { name: "DATABASE_NAME", value: databaseName },
 ],
 resources: {
 limits: {
 memory: "500Mi",
 cpu: "500m"
 }
 }
 }],
 }
 }
 }}, {
 provider: eksCluster.provider,
});

const serversideListener = new k8s.core.v1.Service(...);

const clientAppName = ...;
const clientAppLabels = { ... };
const clientDeployment = new k8s.apps.v1.Deployment(...);

const clientsideListener = new k8s.core.v1.Service(...);

To give us a simple way to connect to our application, we export the clientside listener’s address. We can now open a browser window with the URL and port number to view our application.

export const kubeConfig = eksCluster.kubeconfig;
export const URL = clientsideListener.status.loadBalancer.ingress[0].hostname;

In this example, we showed how to change a PERN stack-based application to use MERN, and demonstrated the ease with which a designer can swap out different components. The isolated nature of containerized applications allows for a great deal of modularity. It can significantly reduce the amount of time spent renovating previously written code to fit any added new components.

The MERN application code is on Github.

Migrating a cloud application to Kubernetes

Vova Ivanov — Mon, 14 Sep 2020 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

In this blog post, we will explore and demonstrate the advantages of Kubernetes by converting and deploying our PERN application to Amazon EKS. With the help of Pulumi, the process becomes greatly simplified and allows us to focus more on the big picture of designing our cloud architecture.

Kubernetes offers several distinct advantages over individually deploying Docker containers to the cloud. Firstly, it’s a robust system for minimizing downtime and rolling out updates. If a pod or node is shut down, a new one is automatically allocated to take its place. Secondly, Kubernetes can scale and distribute your infrastructure. By placing containers into pods, Kubernetes allows you to design the right cloud setup to fit your target effortlessly. And lastly, Kubernetes runs on many cloud providers, letting you choose your organization’s best cloud service provider. Since AWS, Azure, and GCP all support deploying Kubernetes, you have the option of switching if the need arises.

The first step to creating the app is to clone the Typescript PERN example. We’ll use a sparse clone to copy only the aws-ts-pern-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-pern-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-typescript.

$ mkdir aws-ts-k8s-voting-app && cd aws-ts-k8s-voting-app
$ pulumi new aws-typescript

The project will require several configuration variables, which we set using pulumi config set. The variables are used to configure the PostgreSQL admin account, a user account for initializing the schema and table, and its region.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set aws:region <REGION>

The package.json file lists the libraries used by the project. We will add the following to the dependencies section:

"@pulumi/eks": "^0.20.0",
"@pulumi/cloud-aws": "^0.19.0"

Some of the components of our Kubernetes project are nearly identical to those in the PERN application. We can copy many of these files from the example into our directory.

$ cd ..
$ cp -r aws-ts-pern-voting-app/clientside/ aws-ts-k8s-voting-app/clientside
$ cp -r aws-ts-pern-voting-app/serverside/ aws-ts-k8s-voting-app/serverside
$ cd aws-ts-k8s-voting-app

To prepare our application for Kubernetes, we create a PostgreSQL Docker container for the application. Although this is slightly more complex than directly asking AWS to provision an RDS instance, the underlying mechanics are the same for either approach.

The first step is to create two directories:

$ mkdir databaseside && cd databaseside
$ mkdir database

Like the PERN app, the outer databaseside folder holds our Dockerfile, while the inner database folder contains code.

FROM ubuntu:18.04

WORKDIR /

EXPOSE 5432

RUN apt update && \
 apt install -y postgresql

ADD database /database

CMD [ "/database/startDatabase.sh" ]

Our setup doesn’t require any files, and so the only thing we need to put in the database folder is a startDatabase.sh script that launches the database, configures users, and creates a table.

#!/bin/bash
set -exu
FILE=/persistentVolume/postgresqlDb/postgresql.conf
chown postgres:postgres /persistentVolume

if test -f "$FILE"; then
 echo "/persistentVolume already contains postgresqlDb, no need to initialize database."
else
 echo "/persistentVolume is empty, and we need to initialize the postgresql database."
 cd /persistentVolume
 su postgres -c "/usr/lib/postgresql/10/bin/initdb -D /persistentVolume/postgresqlDb"

 echo "host all all 0.0.0.0/0 md5" >> /persistentVolume/postgresqlDb/pg_hba.conf
 echo "host all all ::/0 md5" >> /persistentVolume/postgresqlDb/pg_hba.conf
 echo "listen_addresses='*'" >> /persistentVolume/postgresqlDb/postgresql.conf

 su postgres -c "/usr/lib/postgresql/10/bin/pg_ctl -D /persistentVolume/postgresqlDb --wait -l logfile start"

 set +x
 echo "psql -U postgres -c \"CREATE ROLE $ADMIN_NAME LOGIN SUPERUSER PASSWORD *********;\""
 psql -U postgres -c "CREATE ROLE $ADMIN_NAME LOGIN SUPERUSER PASSWORD '$ADMIN_PASSWORD';"
 echo "psql -U postgres -c \"CREATE USER $USER_NAME WITH PASSWORD *********;\""
 psql -U postgres -c "CREATE USER $USER_NAME WITH PASSWORD '$USER_PASSWORD';"
 echo "psql -U postgres -c \"ALTER ROLE postgres WITH NOLOGIN;\""
 psql -U postgres -c "ALTER ROLE postgres WITH NOLOGIN;"
 set -x

 psql -U $ADMIN_NAME -d postgres -c "CREATE DATABASE $DATABASE_NAME;"
 psql -U $ADMIN_NAME -d $DATABASE_NAME -c "
 CREATE SCHEMA voting_app;
 CREATE TABLE voting_app.choice(
 choice_id SERIAL PRIMARY KEY,
 text VARCHAR(255) NOT NULL,
 vote_count INTEGER NOT NULL
 );
 GRANT USAGE ON SCHEMA voting_app TO $USER_NAME;
 GRANT SELECT, UPDATE ON ALL TABLES IN SCHEMA voting_app TO $USER_NAME;
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Tabs', 0);
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Spaces', 0);
 "
 su postgres -c "/usr/lib/postgresql/10/bin/pg_ctl -D /persistentVolume/postgresqlDb --wait -l logfile stop"
fi

su postgres -c "/usr/lib/postgresql/10/bin/postgres -D /persistentVolume/postgresqlDb"

Because we don’t have any SSL certificates to authenticate and encrypt internet traffic, the way the server connects to the database needs to be modified to include an ssl: false pool parameter.

const Pool = require("pg").Pool;

const pool = new Pool({
 user: process.env["USER_NAME"],
 password: process.env["USER_PASSWORD"],
 host: process.env["POSTGRES_ADDRESS"],
 port: process.env["POSTGRES_PORT"],
 database: process.env["DATABASE_NAME"],
 ssl: false,
});

module.exports = pool;

With our server, client, and database in place, we can focus on the infrastructure. We start with importing libraries and describing the application’s configuration options.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const sqlAdminName = config.require("sql-admin-name");
const sqlAdminPassword = config.requireSecret("sql-admin-password");
const sqlUserName = config.require("sql-user-name");
const sqlUserPassword = config.requireSecret("sql-user-password");
const awsConfig = new pulumi.Config("aws");
const region = aws.config.region;

The central part of our application is an AWS Elastic Kubernetes Cluster. It holds our application and dictates how many nodes should be allocated for the processes running inside the cluster.

const eksCluster = new eks.Cluster("eksCluster", {
 name: "eksCluster",
 instanceType: "t2.medium",
 desiredCapacity: 3,
 minSize: 2,
 maxSize: 4,
 deployDashboard: false,
 enabledClusterLogTypes: [
 "api",
 "audit",
 "authenticator",
 ],
});

Kubernetes nodes use ephemeral storage. When Pods shut down or get restarted, everything is erased. To create a permanent storage system suitable for our database, we need a Persistent Volume instance.

const ebsVolume = new aws.ebs.Volume("storage-volume", {
 size: 1,
 type: "gp2",
 availabilityZone: region + "a",
 encrypted: true,
 }, {
 protect: false,
});

Now that the infrastructure for our database is ready, we can launch a PostgreSQL server on the cloud in the form of a Deployment. Deployments offer a multitude of customization options ranging from attaching volumes, setting the memory and cpu requirements, to specifying which deployments can and cannot be run on the same node.

const databaseAppName = "database-side-service";
const databaseAppLabels = { appClass: databaseAppName };
const databaseDeployment = new k8s.apps.v1.Deployment(databaseAppName, {
 metadata: { name: databaseAppName, labels: databaseAppLabels },
 spec: {
 replicas: 1,
 selector: { matchLabels: databaseAppLabels },
 template: {
 metadata: { labels: databaseAppLabels },
 spec: {
 containers: [{
 name: databaseAppName,
 image: awsx.ecr.buildAndPushImage("database-side-service", "./databaseside").image(),
 ports: [{ name: "http", containerPort: 5432 }],
 env: [
 { name: "ADMIN_NAME", value: sqlAdminName },
 { name: "ADMIN_PASSWORD", value: sqlAdminPassword },
 ],
 volumeMounts: [{
 name: "persistent-volume",
 mountPath: "/persistentVolume"
 }],
 resources: {
 limits: {
 memory: "1Gi",
 cpu: "1000m"
 }
 }
 }],
 volumes: [{
 name: "persistent-volume",
 awsElasticBlockStore: {
 volumeID: ebsVolume.id,
 },
 }],
 affinity: {
 nodeAffinity: {
 requiredDuringSchedulingIgnoredDuringExecution: {
 nodeSelectorTerms: [{
 matchExpressions: [{
 key: "failure-domain.beta.kubernetes.io/zone",
 operator: "In",
 values: [ebsVolume.availabilityZone],
 }]
 }]
 }
 }
 }
 }
 },
 strategy: {
 type: "Recreate"
 }
 }}, {
 deleteBeforeReplace: true,
 provider: eksCluster.provider,
});

In order to have a a way for the pods in cluster to communicate with our database, we create a Kubernetes service for it. It is best practice to keep databases and other similar components entirely within the cluster, and not allow them to be accessed directly the internet.

const databasesideListener = new k8s.core.v1.Service("database-side-listener", {
 metadata: { labels: databaseDeployment.metadata.labels },
 spec: {
 type: "ClusterIP",
 ports: [{ port: 5432, targetPort: "http" }],
 selector: databaseAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

const postgresqlAddress = databasesideListener.spec.clusterIP;
});

Now, all that remains is deploying the client and server containers to Kubernetes. Since the PERN application already deployed them as docker containers, it doesn’t take much effort to reconfigure it for Kubernetes. Unlike the database, these components can safely be open to the Internet using a service with a Load Balancer type.

We’ll first set up the server deployment:

const serverAppName = "server";
const serverAppLabels = { appClass: serverAppName };
const serverDeployment = new k8s.apps.v1.Deployment("server-side-service", {
 metadata: { labels: serverAppLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: serverAppLabels },
 template: {
 metadata: { labels: serverAppLabels },
 spec: {
 containers: [{
 name: serverAppName,
 image: awsx.ecr.buildAndPushImage("server-side-service", "./serverside").image(),
 ports: [{ name: "http", containerPort: 5000 }],
 env: [
 { name: "USER_NAME", value: sqlUserName },
 { name: "USER_PASSWORD", value: sqlUserPassword },
 { name: "POSTGRES_ADDRESS", value: postgresqlAddress },
 { name: "POSTGRES_PORT", value: String(5432) },
 { name: "DATABASE_NAME", value: postgresDatabase.name },
 ],
 resources: {
 limits: {
 memory: "500Mi",
 cpu: "500m"
 }
 }
 }],
 }
 }
 }}, {
 provider: eksCluster.provider,
});

const serversideListener = new k8s.core.v1.Service("server-side-listener", {
 metadata: { labels: serverDeployment.metadata.labels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 5000, targetPort: "http" }],
 selector: serverAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

And lastly, the client deployment.

const clientAppName = "client";
const clientAppLabels = { appClass: clientAppName };
const clientDeployment = new k8s.apps.v1.Deployment("client-side-service", {
 metadata: { labels: clientAppLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: clientAppLabels },
 template: {
 metadata: { labels: clientAppLabels },
 spec: {
 containers: [{
 name: clientAppName,
 image: awsx.ecr.buildAndPushImage("client-side-service", "./clientside").image(),
 ports: [{ name: "http", containerPort: 3000 }],
 env: [
 { name: "SERVER_HOSTNAME", value: serversideListener.status.loadBalancer.ingress[0].hostname },
 ],
 resources: {
 limits: {
 memory: "500Mi",
 cpu: "500m"
 }
 },
 }],
 }
 }
 }}, {
 provider: eksCluster.provider,
});

const clientsideListener = new k8s.core.v1.Service("client-side-listener", {
 metadata: { labels: clientDeployment.metadata.labels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 3000, targetPort: "http" }],
 selector: clientAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

To give us a simple way to connect to our application, we export the clientside listener’s address. We can now open a browser window with the URL and port number to view our application.

export const kubeConfig = eksCluster.kubeconfig;
export const URL = clientsideListener.status.loadBalancer.ingress[0].hostname;

In this example, we demonstrated the benefits Kubernetes offers and showed how to use Pulumi to deploy an application to Kubernetes hosted on AWS. Although it might seem unnecessary to use such a system for only a single voting app, the advantages of running your project in the form of containerized services become apparent as your application grows in size.

Next week, we’ll explore applications using the MERN stack: MongoDB, Express, React, and NodeJS.

The blog post’s code can be found on Github.

Deploying a PERN stack application to AWS

Vova Ivanov — Fri, 04 Sep 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

In this blog post, we will explore PERN stack applications and deploy one to AWS. PERN is an acronym for PostgreSQL, Express, React, and Node. A PERN stack application is a project that uses PostgreSQL, Express as an application framework, React as a user interface framework, and runs on Node. We will also use Pulumi Crosswalk to reduce the amount of code and provide a quick and straightforward path for deploying the application.

PERN projects have four distinct tiers:

a database that keeps track of our data,
a stateless server that receives commands to manage the database,
a clientside server that contains and renders the user interface code,
and the users who run the code in their browsers and send requests to the stateless server.

An advantage of infrastructure as code is that the application and infrastructure deployment can use the same language. As React and the other components use NodeJS, we’ll use it for our infrastructure by writing it in TypeScript. The first step is to create a new directory and initialize a Pulumi project with pulumi new aws-typescript.

$ mkdir aws-ts-pern-voting-app && cd aws-ts-pern-voting-app
$ pulumi new aws-typescript

This tutorial was written for the aws-ts-pern-voting-app example but will work with any other PERN stack application. The example uses two folders to hold the client and server tiers and a Dockerfile that builds images that run as containers in AWS.

To deploy our PERN stack, the project requires several configuration variables, which we set using pulumi config set. The variables are used to configure the PostgreSQL admin account, a user account for initializing the schema and table, and the database’s region.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set aws:region <REGION>

The package.json file lists the libraries used by the project. We will add the following to the dependencies section:

"@pulumi/cloud-aws": "^0.19.0",
"@pulumi/postgresql": "^2.3.0",
"pg": "^8.3.3"

Our project uses a Dynamic Provider to help create tables and Schemas. It offers the same features as our MySQL provider, but for PostgreSQL.

import * as pulumi from "@pulumi/pulumi";
import * as crypto from "crypto";

export interface SchemaInputs {
 creatorName: pulumi.Input<string>
 creatorPassword: pulumi.Input<string>
 serverAddress: pulumi.Input<string>
 databaseName: pulumi.Input<string>
 creationScript: pulumi.Input<string>
 deletionScript: pulumi.Input<string>
 postgresUserName: pulumi.Input<string>
}

export class SchemaProvider implements pulumi.dynamic.ResourceProvider {
 async create(args: SchemaInputs): Promise<pulumi.dynamic.CreateResult> {
 const { Pool } = require("pg");
 const pool = new Pool({
 user: args.creatorName,
 password: args.creatorPassword,
 host: args.serverAddress,
 port: 2000,
 database: args.databaseName
 });
 const scriptExecuted = await pool.query(args.creationScript);

 await pool.end();
 return {id: "postgresqlSchema-" + crypto.randomBytes(16).toString('hex'), outs: args};
 }

 async delete(id: string, args: SchemaInputs): Promise<void> {
 const { Pool } = require("pg");
 const pool = new Pool({
 user: args.creatorName,
 password: args.creatorPassword,
 host: args.serverAddress,
 port: 2000,
 database: args.databaseName
 });
 const scriptExecuted = await pool.query(args.deletionScript);
 await pool.end();
 }

 async diff(id: string, oldArgs: SchemaInputs, newArgs: SchemaInputs): Promise<pulumi.dynamic.DiffResult> {
 let changes: boolean = ((oldArgs.creatorName != newArgs.creatorName) ||
 (oldArgs.creatorPassword != newArgs.creatorPassword) || (oldArgs.serverAddress != newArgs.serverAddress) ||
 (oldArgs.databaseName != newArgs.databaseName) || (oldArgs.creationScript != newArgs.creationScript) ||
 (oldArgs.deletionScript != newArgs.deletionScript));

 let replaces: string[] = [];
 if (oldArgs.serverAddress != newArgs.serverAddress) { replaces.push("serverAddress") };
 if (oldArgs.databaseName != newArgs.databaseName) { replaces.push("databaseName") };
 if (oldArgs.creationScript != newArgs.creationScript) { replaces.push("creationScript") };

 return {
 changes: changes,
 replaces: replaces,
 stables: [],
 deleteBeforeReplace: true
 }
 }

 async update(id: string, oldArgs: SchemaInputs, newArgs: SchemaInputs): Promise<pulumi.dynamic.UpdateResult> {
 return { outs: newArgs };
 }
}

export class Schema extends pulumi.dynamic.Resource {
 public readonly creatorName!: pulumi.Output<string>;
 public readonly creatorPassword!: pulumi.Output<string>;
 public readonly serverAddress!: pulumi.Output<string>;
 public readonly databaseName!: pulumi.Output<string>;
 public readonly creationScript!: pulumi.Output<string>;
 public readonly deletionScript!: pulumi.Output<string>;
 constructor(name: string, args: SchemaInputs) {
 super(new SchemaProvider, name, args);
 }
}

With the Dynamic provider configured, we can focus on the main index.ts file. We start with importing libraries and describing the application’s configuration options.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as pulumi from "@pulumi/pulumi";
import * as postgresql from "@pulumi/postgresql";
import { Schema } from "./PostgreSqlDynamicProvider";
import { table } from "console";

const config = new pulumi.Config();
const sql_admin_name = config.require("sql-admin-name");
const sql_admin_password = config.requireSecret("sql-admin-password");
const sql_user_name = config.require("sql-user-name");
const sql_user_password = config.requireSecret("sql-user-password");
const awsConfig = new pulumi.Config("aws");
const availabilityZone = awsConfig.get("region");

Apart from language differences, the code for creating the VPC, subnets, and tables is almost identical to our previous projects. Since we will be using Crosswalk, we can leave out a few components that will be automatically created later.

const appVpc = new aws.ec2.Vpc("app-vpc", {
 cidrBlock: "172.31.0.0/16",
 enableDnsHostnames: true
});

const appGateway = new aws.ec2.InternetGateway("app-gateway", {
 vpcId: appVpc.id
});

const appRoutetable = new aws.ec2.RouteTable("app-routetable", {
 routes: [
 {
 cidrBlock: "0.0.0.0/0",
 gatewayId: appGateway.id,
 }
 ],
 vpcId: appVpc.id
});

const appRoutetableAssociation = new aws.ec2.MainRouteTableAssociation("app-routetable-association", {
 routeTableId: appRoutetable.id,
 vpcId: appVpc.id
});

const rdsSecurityGroup = new aws.ec2.SecurityGroup("rds-security-group", {
 vpcId: appVpc.id,
 description: "Enables HTTP access",
 ingress: [{
 protocol: 'tcp',
 fromPort: 0,
 toPort: 65535,
 cidrBlocks: ['0.0.0.0/0'],
 }],
 egress: [{
 protocol: '-1',
 fromPort: 0,
 toPort: 0,
 cidrBlocks: ['0.0.0.0/0'],
 }]
});

const firstRdsSubnet = new aws.ec2.Subnet("first-rds-subnet", {
 vpcId: appVpc.id,
 cidrBlock: "172.31.0.0/20",
 availabilityZone: availabilityZone + "a"
});

const secondRdsSubnet = new aws.ec2.Subnet("second-rds-subnet", {
 vpcId: appVpc.id,
 cidrBlock: "172.31.128.0/20",
 availabilityZone: availabilityZone + "b"
});

We create a SubnetGroup and an RDS instance. The latter uses PostgreSQL instead of MySQL, and is assigned 2000.

const rdsSubnetGroup = new aws.rds.SubnetGroup("rds-subnet-group", {
 subnetIds: [firstRdsSubnet.id, secondRdsSubnet.id]
});

const postgresqlRdsServer = new aws.rds.Instance("postgresql-rds-server", {
 engine: "postgres",
 username: sql_admin_name,
 password: sql_admin_password,
 instanceClass: "db.t2.micro",
 allocatedStorage: 20,
 skipFinalSnapshot: true,
 publiclyAccessible: true,
 port: 2000,
 dbSubnetGroupName: rdsSubnetGroup.name,
 vpcSecurityGroupIds: [rdsSecurityGroup.id]
});

Pulumi offers additional tools to make handling PostgreSQL easier.

const postgresqlProvider = new postgresql.Provider("postgresql-provider", {
 host: postgresqlRdsServer.address,
 port: postgresqlRdsServer.port,
 username: sql_admin_name,
 password: sql_admin_password,
 superuser: false
});

We initialize the example database and create a user to manage it.

const postgresDatabase = new postgresql.Database("postgresql-database", {
 name: "votes"}, {
 provider: postgresqlProvider
});

const postgresUser = new postgresql.Role("postgres-standard-role", {
 name: sql_user_name,
 password: sql_user_password,
 superuser: false,
 login: true,
 connectionLimit: -1}, {
 provider: postgresqlProvider
});

We use the Dynamic provider to create a schema and a table, grant permissions for our user to edit and select it, and populate the table with two initial voting options.

const creation_script = `
 CREATE SCHEMA voting_app;
 CREATE TABLE voting_app.choice(
 choice_id SERIAL PRIMARY KEY,
 text VARCHAR(255) NOT NULL,
 vote_count INTEGER NOT NULL
 );
 GRANT USAGE ON SCHEMA voting_app TO ${sql_user_name};
 GRANT SELECT, UPDATE ON ALL TABLES IN SCHEMA voting_app TO ${sql_user_name};
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Tabs', 0);
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Spaces', 0);
`;

const deletion_script = "DROP SCHEMA IF EXISTS voting_app CASCADE";

const postgresqlVotesTable = new Schema("postgresql-votes-schema", {
 creatorName: sql_admin_name,
 creatorPassword: sql_admin_password,
 serverAddress: postgresqlRdsServer.address,
 databaseName: postgresDatabase.name,
 creationScript: creation_script,
 deletionScript: deletion_script,
 postgresUserName: postgresUser.name
});

With the basic infrastructure and provider completed, we can write the application deployment code to ECS. We will use Pulumi Crosswalk, a collection of libraries that makes common infrastructure-as-code tasks in AWS easier and more secure by automatically using well-architected best practices.

We’ll first set up the server. The Network Listener is assigned the same port as the server, which is, in our case, port 5000. A set of environment variables representing our PostgreSQL connection credentials are passed directly to the awsx.ecs.FargateService. With AWS Crosswalk, what would have been over 150 lines of code is reduced to just under 20 lines of code.

const serversideListener = new awsx.elasticloadbalancingv2.NetworkListener("server-side-listener", { port: 5000 });
const serversideService = new awsx.ecs.FargateService("server-side-service", {
 taskDefinitionArgs: {
 containers: {
 serversideService: {
 image: awsx.ecs.Image.fromPath("server-side-service", "./serverside"),
 memory: 512,
 portMappings: [serversideListener],
 environment: [
 { name: "USER_NAME", value: sql_user_name },
 { name: "USER_PASSWORD", value: sql_user_password },
 { name: "RDS_ADDRESS", value: postgresqlRdsServer.address },
 { name: "RDS_PORT", value: String(2000) },
 { name: "DATABASE_NAME", value: postgresDatabase.name },
 ],
 },
 },
 },
});

The same is true for the client service, which is reduced to a short and easy to understand format. By default, React uses port 3000, but it can set to a different port. The SERVER_HOSTNAME environment variable is passed in when the container starts to generate a tiny configuration file at runtime called serverParams.js with the URL. This way, we do not have to rebuild the entire docker image should the server URL change.

const clientsideListener = new awsx.elasticloadbalancingv2.NetworkListener("client-side-listener", { port: 3000 });
const clientsideService = new awsx.ecs.FargateService("client-side-service", {
 taskDefinitionArgs: {
 containers: {
 clientsideService: {
 image: awsx.ecs.Image.fromPath("client-side-service", "./clientside"),
 memory: 512,
 portMappings: [clientsideListener],
 environment: [
 { name: "SERVER_HOSTNAME", value: serversideListener.endpoint.hostname },
 ],
 },
 },
 },
});

To make our PERN stack application available on the Internet, we export the clientside listener’s address. We can open a browser window with the URL and port to view our application.

export const URL = clientsideListener.endpoint.hostname;

In this example, I explained the basic principles behind PERN stack applications and showed how to create the infrastructure to deploy them on ECS. A flexible and optimized tool, Pulumi Crosswalk supports rapidly prototyping applications, scaling workloads, securing and integrating applications in existing infrastructure, and going to production in multiple complex environments.

Next week, I’ll demonstrate how to integrate applications with Kubernetes, and seamlessly deploy them to EKS using Pulumi.

The blog post’s code can be found on Github.

How to Deploy a Django Application to AWS

Vova Ivanov — Fri, 28 Aug 2020 00:00:00 +0000

In this blog post, we will finish swapping out the frontend and backend of our Python AWS application. Although Flask and Redis are different from Django and MySQL in many ways, the underlying infrastructure behind their deployment is nonetheless very similar, and can be effortlessly updated as we transition from one to the other.

We will be paying additional attention to security, and will be making use of Pulumi’s secret management tools to protect our passwords and private keys.

The first step is to create a new directory and initialize a Pulumi project with pulumi new aws-python.

$ mkdir aws-py-django-voting-app && cd aws-py-django-voting-app
$ pulumi new aws-python

Next, let’s create a folder to hold our application and start a Django project in it.

$ mkdir frontend && cd frontend
$ django-admin startproject mysite
$ cd mysite

This tutorial was written for the aws-py-django-voting-app example but will work with any other Django application. The most important file is ./mysite/settings.py, which we will modify to accept secrets and configuration parameters in the form of environment variables. A common mistake that programmers make is to submit files with important data to source code repositories. Even if it is a private repository, it is still not recommended to leave passwords and private keys in your files.

SECRET_KEY = os.environ['SECRET_KEY']

DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.mysql',
 'NAME': os.environ['DATABASE_NAME'],
 'USER': os.environ['USER_NAME'],
 'PASSWORD': os.environ['USER_PASSWORD'],
 'HOST': os.environ['DATABASE_ADDRESS'],
 'PORT': os.environ['DATABASE_PORT'],
 }
}

To perform database migrations as part of the deployment, we will create a setupDatabase.sh script inside the aws-py-django-voting-app/frontend/mysite folder. As a bonus, the script will create an admin account that we can use to log into our website. Make sure the file has exec permissions.

#!/bin/bash
set -exu
python3 /mysite/manage.py makemigrations
python3 /mysite/manage.py migrate
export DJANGO_SUPERUSER_PASSWORD=$DJANGO_PASSWORD
python3 /mysite/manage.py createsuperuser \
 --no-input \
 --username=$DJANGO_NAME \
 --email=$DJANGO_NAME@example.com

The next step is to go back into the aws-py-django-voting-app/frontend folder and containerize our application with Docker. First, we will list the libraries our application uses by creating a requirements.txt file with the following lines.

django==3.1
mysqlclient==1.4.0

And lastly, we can create the Dockerfile.

FROM ubuntu:18.04

WORKDIR /

EXPOSE 80

RUN apt-get update && \
 apt install -y gcc python3-dev python3-pip mysql-client-core-5.7 libmysqlclient-dev

ADD requirements.txt /

RUN pip3 install -r requirements.txt

ADD mysite /mysite

CMD [ "python3", "/mysite/manage.py", "runserver", "0.0.0.0:80" ]

Now that our Django application and Dockerfile are ready, we can return to the main aws-py-django-voting-app folder. The Pulumi project requires several configuration variables, which we set using pulumi config set. They are used to configure the MySQL admin account, a user account for initializing the table, and the Django website admin account. The private key that Django uses will also be passed in the same way.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set django-admin-name <NAME>
$ pulumi config set django-admin-password <PASSWORD> --secret
$ pulumi config set django_secret_key = <VALUE> --secret

The requirements.txt file lists the libraries used by the project. We will need to add the following:

pulumi-docker>=2.0.0,<3.0.0
pulumi-mysql>=2.0.0,<3.0.0
mysql-connector-python>=1.0.0,<10.0.0

The first few lines of our __main__.py file indicate the libraries to import and describe the application’s configuration options.

import json
import base64
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker
import pulumi_mysql as mysql

config = pulumi.Config()
sql_admin_name = config.require("sql-admin-name")
sql_admin_password = config.require_secret("sql-admin-password")
sql_user_name = config.require("sql-user-name")
sql_user_password = config.require_secret("sql-user-password")
availability_zone = pulumi.Config("aws").get("region")

django_admin_name = config.require("django-admin-name")
django_admin_password = config.require_secret("django-admin-password")

django_secret_key = config.require_secret("django-secret-key")

After setting up the imports and configurations, we create an ECS Cluster, VPC, Subnet, RDS instance, and several other items. All the components are identical to the ones in the first and second blog posts, and the full code can be seen in this example’s github repository.

app_cluster = aws.ecs.Cluster(...)

app_vpc = aws.ec2.Vpc(...)
app_vpc_subnet = aws.ec2.Subnet(...)

app_gateway = aws.ec2.InternetGateway(...)
app_routetable = aws.ec2.RouteTable(...)
app_security_group = aws.ec2.SecurityGroup(...)

app_exec_role = aws.iam.Role(...)
exec_policy_attachment = aws.iam.RolePolicyAttachment(...)
app_task_role = aws.iam.Role(...)
task_policy_attachment = aws.iam.RolePolicyAttachment(...)
task_policy_attachment = aws.iam.RolePolicyAttachment(...)

app_ecr_repo = aws.ecr.Repository(...)
app_lifecycle_policy = aws.ecr.LifecyclePolicy(...)

extra_rds_subnet = aws.ec2.Subnet(...)
app_database_subnetgroup = aws.rds.SubnetGroup(...)
mysql_rds_server = aws.rds.Instance(...)
mysql_provider = mysql.Provider(...)
mysql_database = mysql.Database(...)
mysql_user = mysql.User(...)
mysql_access_grant = mysql.Grant(...)

With the basic infrastructure and MySQL backend in place, all that’s left is to create the Django frontend.

A target group, balancer, and listener are created for the frontend.

django_targetgroup = aws.lb.TargetGroup("django-targetgroup",
 port=80,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

django_balancer = aws.lb.LoadBalancer("django-balancer",
 load_balancer_type="network",
 internal=False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

django_listener = aws.lb.Listener("django-listener",
 load_balancer_arn=django_balancer.arn,
 port=80,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": django_targetgroup.arn
 }])

The application is built into a Docker image and pushed to our ECR repository we created earlier.

def get_registry_info(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])

app_registry = app_ecr_repo.registry_id.apply(get_registry_info)

django_image = docker.Image("django-dockerimage",
 image_name=app_ecr_repo.repository_url,
 build="./frontend",
 skip_push=False,
 registry=app_registry
)

To help with debugging our application, we will create a Cloudwatch instance to store all logs.

django_log_group = aws.cloudwatch.LogGroup("django-log-group",
 retention_in_days=1,
 name="django-log-group"
)

Our project is special because it uses two independent ECS services—one that sets up the MySQL database, and one that continuously runs and handles the website.

While it is possible to have a single ECS service set up both the database and the server, doing so presents the risk of exposing important information to the internet. To set up the database, the ECS service would need to know our MySQL admin credentials. However, since the service would also handle the website, it would mean that a successful attack on our Django website could allow the attacker to access the credentials for our backend.

First, let’s create the task definition that runs once and sets up the database.

django_database_task_definition = aws.ecs.TaskDefinition("django-database-task-definition",
 family="django_database_task_definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(
 # We must do Output.all() to access Pulumi secrets
 django_image.image_name,
 django_secret_key,
 mysql_database.name,
 sql_admin_name,
 sql_admin_password,
 django_admin_name,
 django_admin_password,
 mysql_rds_server.address,
 mysql_rds_server.port).apply(lambda args: json.dumps([{
 "name": "django-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "SECRET_KEY", "value": args[1] },
 { "name": "DATABASE_NAME", "value": args[2] },
 { "name": "USER_NAME", "value": args[3] },
 { "name": "USER_PASSWORD", "value": args[4] },
 { "name": "DJANGO_NAME", "value": args[5] },
 { "name": "DJANGO_PASSWORD", "value": args[6] },
 { "name": "DATABASE_ADDRESS", "value": args[7] },
 { "name": "DATABASE_PORT", "value": str(int(args[8])) },
 ],
 "logConfiguration": {
 "logDriver": "awslogs",
 "options": {
 "awslogs-group": "django-log-group",
 "awslogs-region": "us-west-2",
 "awslogs-stream-prefix": "djangoApp-database",
 },
 },
 # We override the command in the Dockerfile with a new one
 "command": ["/mysite/setupDatabase.sh"]
 }])))

We can now launch our first service on Fargate.

django_database_service = aws.ecs.Service("django-database-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=django_database_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": django_targetgroup.arn,
 "container_name": "django-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[django_listener]),
)

Finally, let’s create the second task definition to run the website.

django_site_task_definition = aws.ecs.TaskDefinition("django-site-task-definition",
 family="django-site-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(
 django_image.image_name,
 django_secret_key,
 mysql_database.name,
 sql_user_name,
 sql_user_password,
 mysql_rds_server.address,
 mysql_rds_server.port).apply(lambda args: json.dumps([{
 "name": "django-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "SECRET_KEY", "value": args[1] },
 { "name": "DATABASE_NAME", "value": args[2] },
 { "name": "USER_NAME", "value": args[3] },
 { "name": "USER_PASSWORD", "value": args[4] },
 { "name": "DATABASE_ADDRESS", "value": args[5] },
 { "name": "DATABASE_PORT", "value": str(int(args[6])) },
 ],
 "logConfiguration": {
 "logDriver": "awslogs",
 "options": {
 "awslogs-group": "django-log-group",
 "awslogs-region": "us-west-2",
 "awslogs-stream-prefix": "djangoApp-site",
 },
 },
 }])))

Similarly, we will launch our second task definition on Fargate.

django_site_service = aws.ecs.Service("django-site-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=django_site_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": django_targetgroup.arn,
 "container_name": "django-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[django_listener]),

To connect to our application, we export the DNS name of the Django balancer and open it in a browser window.

pulumi.export("app-url", django_balancer.dns_name)

To add questions to the database, add “/admin/” to the end of the URL, and log in using your Django admin credentials.

The Administration panel allows you to manage the state of your application. Groups, users, choices, and questions can all be added easily.

Comparison between Flask and Django:

In this example, I described how to set up a basic Django voting application and deploy it to AWS. Although our Django and Flask applications use very different AWS services, we seamlessly replaced the previous infrastructure with what was needed.

Next week, we’ll explore PostgreSQL, Express, React, and Node.js, and use them to create a simple application.

The blog post’s full code and an in-depth explanation for each component are on Github.

Creating a Python AWS Application Using Flask and Redis

Vova Ivanov — Thu, 13 Aug 2020 15:06:26 -0700

Meet Vova Ivanov—one of the Pulumi summer interns. He’ll be writing about his experiences learning Pulumi while modernizing a web app and its underlying infrastructure.

I’ve recently started developing with Pulumi, and have begun to explore its inner workings and mechanisms. I decided to construct a production-level application and to document each step that I take and my progress as I go along.

This blog post recreates the existing Typescript voting app example step by step in Python with Flask as the frontend and Redis as the backend. In future blog posts, we will explore how to change the front and backends, how to upgrade the app with additional AWS services, and migrating from one cloud provider to another.

The first step is to clone the Typescript voting app example. We’ll use a sparse clone to copy only the aws-ts-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-python.

$ mkdir aws-py-voting-app && cd aws-py-voting-app
$ pulumi new aws-python

Since we won’t be modifying the application’s frontend, we simply copy the frontend/ folder from the example into our directory.

$ cp -r aws-ts-voting-app/frontend/ aws-py-voting-app/frontend

The requirements.txt file lists the libraries that the project depends on. We will need to add the following:

pulumi-docker>=2.0.0,<3.0.0

The biggest difference between our project and the example is the code describing the cloud infrastructure we want to create. Instead of an index.ts file, the project uses a __main__.py file. The first few lines of the file will indicate which libraries to import and describe a pair of configuration options used by the application.

import json
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker

config = pulumi.Config()
redis_password = config.require("redis-password")
redis_port = 6379

After setting up the imports and configurations, we create an Elastic Container Service Cluster. A Cluster represents a group of tasks and services that work together for a certain purpose. In this instance, the purpose is to provide users with a voting application.

app_cluster = aws.ecs.Cluster("app-cluster")

To allow different tasks within our cluster to communicate, we create a Virtual Private Cloud and an associated subnet.

app_vpc = aws.ec2.Vpc("app-vpc",
 cidr_block="172.31.0.0/16",
 enable_dns_hostnames=True)

app_vpc_subnet = aws.ec2.Subnet("app-vpc-subnet",
 cidr_block="172.31.32.0/20",
 vpc_id=app_vpc)

A gateway and routing table are needed to allow the VPC to communicate with the Internet. Once created, we associate the routing table with our VPC.

app_gateway = aws.ec2.InternetGateway("app-gateway",
 vpc_id=app_vpc.id)

app_routetable = aws.ec2.RouteTable("app-routetable",
 routes=[
 {
 "cidr_block": "0.0.0.0/0",
 "gateway_id": app_gateway.id,
 }
 ],
 vpc_id=app_vpc.id)

app_routetable_association = aws.ec2.MainRouteTableAssociation("app_routetable_association",
 route_table_id=app_routetable.id,
 vpc_id=app_vpc)

To control traffic flow between applications running inside our VPC, we create a security group.

app_security_group = aws.ec2.SecurityGroup("security-group",
 vpc_id=app_vpc.id,
 description="Enables HTTP access",
 ingress=[{
 'protocol': 'tcp',
 'from_port': 0,
 'to_port': 65535,
 'cidr_blocks': ['0.0.0.0/0'],
 }],
 egress=[{
 'protocol': '-1',
 'from_port': 0,
 'to_port': 0,
 'cidr_blocks': ['0.0.0.0/0'],
 }])

To start our services, we need to create an Identity and Access Management (IAM) role, and attach execution permissions to it.

app_exec_role = aws.iam.Role("app-exec-role",
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "ecs-tasks.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 }""")

exec_policy_attachment = aws.iam.RolePolicyAttachment("app-exec-policy", role=app_exec_role.name,
 policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_TASK_EXECUTION_ROLE_POLICY)

Likewise, our ECS service will need to have a task role to manage it, along with its own set of permissions.

app_task_role = aws.iam.Role("app-task-role",
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "ecs-tasks.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 }""")

task_policy_attachment = aws.iam.RolePolicyAttachment("app-access-policy", role=app_task_role.name,
 policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_FULL_ACCESS)

An Elastic Container Registry Repository stores the application Docker images that we want to run. The life cycle policy automatically removes the oldest untagged image that we uploaded.

app_ecr_repo = aws.ecr.Repository("app-ecr-repo",
 image_tag_mutability="MUTABLE")

app_lifecycle_policy = aws.ecr.LifecyclePolicy("app-lifecycle-policy",
 repository=app_ecr_repo.name,
 policy="""{
 "rules": [
 {
 "rulePriority": 10,
 "description": "Remove untagged images",
 "selection": {
 "tagStatus": "untagged",
 "countType": "imageCountMoreThan",
 "countNumber": 1
 },
 "action": {
 "type": "expire"
 }
 }
 ]
 }""")

With the basic infrastructure in place, we can begin to set up the application itself. First, we will start with the Redis backend.

We need to create a target group, a balancer, and a listener to allow Redis to communicate with the other services in our VPC, w

redis_targetgroup = aws.lb.TargetGroup("redis-targetgroup",
 port=redis_port,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

redis_balancer = aws.lb.LoadBalancer("redis-balancer",
 load_balancer_type="network",
 internal= False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

redis_listener = aws.lb.Listener("redis-listener",
 load_balancer_arn=redis_balancer.arn,
 port=redis_port,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": redis_targetgroup.arn
 }])

Every service running within ECS requires a task definition, specifying what hardware, internet, and container settings it will use.

redis_task_definition = aws.ecs.TaskDefinition("redis-task-definition",
 family="redis-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=json.dumps([{
 "name": "redis-container",
 "image": "redis:alpine",
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": redis_port,
 "hostPort": redis_port,
 "protocol": "tcp"
 }],
 "command": ["redis-server", "--requirepass", redis_password],
 }]))

Now that we have our task definition blueprint ready, we can launch a service using it. An endpoint is created to hold information needed to connect the backend with the frontend.

redis_service = aws.ecs.Service("redis-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=redis_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": redis_targetgroup.arn,
 "container_name": "redis-container",
 "container_port": redis_port,
 }],
 opts=pulumi.ResourceOptions(depends_on=[redis_listener]),
)

redis_endpoint = {"host": redis_balancer.dns_name, "port": redis_port}

The Redis backend is complete, and now all that’s left is to create the Flask frontend.

A similar set of a target group, balancer, and listener is created for the frontend. Unlike Redis, Flask will serve as the webpage and, as such, needs to receive its data from port 80.

flask_targetgroup = aws.lb.TargetGroup("flask-targetgroup",
 port=80,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

flask_balancer = aws.lb.LoadBalancer("flask-balancer",
 load_balancer_type="network",
 internal=False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

flask_listener = aws.lb.Listener("flask-listener",
 load_balancer_arn=flask_balancer.arn,
 port=80,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": flask_targetgroup.arn
 }])

The application is built into a Docker image and pushed to our ECR repository we created earlier.

def get_registry_info(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])

app_registry = app_ecr_repo.registry_id.apply(get_registry_info)

flask_image = docker.Image("flask-dockerimage",
 image_name=app_ecr_repo.repository_url,
 build="./frontend",
 skip_push=False,
 registry=app_registry
)

A task definition is created for the frontend; this time, using the flask image that we built. The redis endpoint is passed in as a group of environment variables, informing our app of our database’s host and port values.

flask_task_definition = aws.ecs.TaskDefinition("flask-task-definition",
 family="frontend-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(flask_image.image_name, redis_endpoint).apply(lambda args: json.dumps([{
 "name": "flask-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "REDIS", "value": args[1]["host"] },
 { "name": "REDIS_PORT", "value": str(args[1]["port"]) },
 { "name": "REDIS_PWD", "value": redis_password },
 ],
 }])))

The service for our Flask frontend is launched. It can receive requests from the Internet, process them, and change the Redis database accordingly.

flask_service = aws.ecs.Service("flask-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=flask_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": flask_targetgroup.arn,
 "container_name": "flask-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[flask_listener]),
)

To connect to our application, we export the DNS name of the Flask balancer, and open it in a browser window.

pulumi.export("app-url", flask_balancer.dns_name)

In this example, I showed how straightforward it is to convert infrastructure code defined in TypeScript to one defined in Python. Teams often choose to standardize on one language and Pulumi expands that choice, allowing them to standardize on one language for the frontend, the backend, and the infrastructure.

Next week, I’ll show how to modernize the application by changing the frontend from Flask to Django, and swapping out the database from Redis to Amazon RDS—using Pulumi to stand up new resources as I go.

The full code for the blog post can be found on github.

Build a Pulumi VS Code Devcontainer Environment

Sophia Parafina — Tue, 14 Jul 2020 00:00:00 +0000

One of the major advantages of using containers for development is reducing the need to install software and associated dependencies. Developers can start writing code without configuring a development environment that emulates production. The Visual Studio Code Remote - Containers extension lets you develop inside a container. If you want to use Pulumi’s infrastructure as code engine without installing the Pulumi CLI, this blog post is for you!

Pre-requisites

To set up your development environment, you’ll need Visual Studio Code, Docker Desktop, and the Remote - Containers extension.

Check to see if Docker is running:

$ docker --version
Docker version 19.03.8, build afacb8b

Check to see if the Remote - Containers is installed:

Click on the green icon in the Remote Status Bar to display the commands:

Building the Pulumi container

Now we can start!

Create a directory for your project.
```
$ mkdir my_devcontainer
$ cd my_devcontainer
```
Open the directory in VS Code.
Next, we’ll create the devcontainer configuration files. Click on the Remote Status button and select Add Development Container Configuration Files ….
Select the latest version for Alpine. We’ll replace the Dockerfile with the Pulumi Dockerfile.

VS Code will display a popup asking to build the container. Dismiss the popup, we will replace the Dockerfile with the Pulumi Dockerfile.

In your project directory, you’ll see a .devcontainer directory with a Dockerfile that builds the container and a devcontainer.json for configuring the container environment.

Dockerfile

For this example, we’ll use the standard Pulumi Dockerfile, which includes the SDK for all the supported languages. Download the Dockerfile and save it to the .devcontainer directory.

$ cd .devcontainer
$ curl "https://raw.githubusercontent.com/pulumi/pulumi/master/.devcontainer/Dockerfile" -o Dockerfile

Container configuration

Next, we’ll clean up the .devcontainer/devcontainer.json file and set up the necessary credentials.

First edit devcontainer.json as follows:

Replace the “name” field with Pulumi
Remove the “args” line
Replace the “remoteUser” value with user

Regardless of your cloud provider, you will have to provide your credentials to deploy resources. If you want to save your project to Github or another repository, you don’t want your credentials stored in the repository. There are several ways to do this securely. Using AWS as an example, you can export them inside the container, e.g.,

Linux/macOS:

export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Windows:

SET AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
SET AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

The disadvantage of this method is that you will have to set these environment variables every time you start the devcontainer.

Use local environment variables

One alternative is to set the environment variables on your local machine and pass them to the container through the devcontainer.json file. This method requires that you have VS Code in your path and launch it from the shell.

Use CTRL+SHIFT+P or CMD+SHIFT+P to bring up the command palette and type Shell Command: Install ‘code’ command in PATH. You can set your credentials as environmental variables and launch the VS Code from the shell
```
$ export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ code .
```

Next, set your provider credentials on your machine and add the following to devcontainer.json.

"containerEnv": {
 "AWS_ACCESS_KEY_ID": "${localEnv:AWS_ACCESS_KEY_ID}",
 "AWS_SECRET_ACCESS_KEY": "${localEnv:AWS_SECRET_ACCESS_KEY}"
},

Use an .env file

Another alternative is to use a devcontainer.env file, which contains your credentials.

Create a file with your credentials.

AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Then add it to the devcontainer.json file.

"runArgs": ["--env-file","devcontainer.env"],

To prevent it from exposing your credentials on a public repository, add the file to .gitignore or similar for you repository, e.g.,
```
echo devcontainer.json > .gitignore
```

There are many more configuration options available, including mounts and remote configurations.

Try it out

Again, use the command palette to start the container. Use the Open Folder in Container… command, and select the project folder. If you’ve already got the folder open locally, you could also use the Reopen in Container command.

Check to see if your credentials are set. Open a terminal in VS Code if it isn’t already open.

$ user@973a612c1555:/workspaces/my_devcontainer# env | grep AWS
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE

Now we ready to use Pulumi in our VS Code development environment! Let’s try it out by creating an S3 bucket. First, we’ll make a directory for our project and use pulumi new to create a default project.

$ mkdir my_project
$ cd my_project
$ pulumi new aws-typescript

Follow the prompts, then run pulumi up to deploy.

$ pulumi up
Previewing update (dev):
 Type Name Plan
 + pulumi:pulumi:Stack my_project-dev create
 + └─ aws:s3:Bucket my-bucket create

Resources:
 + 2 to create

Do you want to perform this update? yes
Updating (dev):
 Type Name Status
 + pulumi:pulumi:Stack my_project-dev created
 + └─ aws:s3:Bucket my-bucket created

Outputs:
 bucketName: "my-bucket-cbdec06"

Resources:
 + 2 created

Duration: 31s

Permalink: https://app.pulumi.com/spara/my_project/dev/updates/1

You can CMD + click on the link to open a browser and display the status of your project on the Pulumi Service.

Build a Pulumi examples devcontainer

Pulumi has a repository of examples, and we can create a devcontainer from the repository so you can try them out.

Clone the repository, then follow the steps above to create and configure the Pulumi container image.
```
$ git clone https://github.com/pulumi/examples.git
```
Create the configuration files, add the Pulumi Docker file, and add your credentials.
Let’s deploy a Kubernetes cluster! With pulumi stack init, we can create a stack, but we will need to install the project dependencies and set configuration variables such as the region.
```
$ cd aws-ts-eks
$ pulumi stack init
$ npm install
$ pulumi config set aws:region us-west-2
$ pulumi up
```

Here’s the kubeconfig file’s output, which you can use with kubectl to manage the cluster.

Outputs:
 kubeconfig: {
 apiVersion : "v1"
 clusters : [
 [0]: {
 cluster: {
 certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EY3hNVEl5TXpZMU9Wb1hEVE13TURjd09USXlNelkxT1Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHFvCk9wTDBiQWo2TFFMNlh2anRsMFZ6TUN5WTJmYlo4cXEvNGd3bkJxcVI2RFV6Q1RwbkJmOHJVRjVrUXhBQXRUNysKTTFZblp5VEFBYndOZ3NIL3lObk5sMnFCQmRSQ0JZTVVlN1I2MVBxcGdabFZpN0JaNHRlek9DalRqeGV0SXlaMwoxRW9TdTNBYnhzTmh6dDVBTjVYZklnZlBuM3B2eEtwYTA1MEg3aWhFaUptSEl3M0Z5TVovUHpxY2JHczh3RHVCCk1jYjNzOHMwZDBIVExsb2dCL2svOVBLbGVpZDgyWFkxMEdBN0YxeFM2cFdZODkwQ1BwNXpWZ0pRWGhuRGR2S1gKVlFValdCTDZQYU14U0lBZEZQM1BZYk1FdXk5Vyt0c2lqc2VydThVNTVpUDlWSUdMdTZzdEdycTNuQUhtUk40UQpqeXRCRWd4RjN1R0dtZ0dMOStFQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEcUtyVGVnZmhWbElNUklQNjFVZEtvaGtnS3MKblUyZjUvTHBzU3ozbmliUCt3Y2VpNktYL3RSNjAzTHhFMlVXUTlmR0ZycTBCMTlJR1JaK0VVQjl2Vy9JdWowTApVK2UwUUtWc01qd2dMWm4rS3IyN0VQNUNGay8vZEVCRUZqa01PMmNlTFFRaFcvUXFzSFhJb2ZjZ0UvZnJ3YzErCmQwdGROWmxJa3JRdzJwYnVhcVI0VmRlWlpVSitqbW8rdEo1SlZ5a2Rwa29JYml1ZEZ6Q0N5YVB1LzhJc09CTUsKcWtvTFpNazdBMzh5SWxDK2RCTnJwdkQvRTF5U1lhcXlDaE5CdVBkd1VqcWZaWmJmaWR5N050emd2VjMvUWFtWgpuZUJJcWs0c1AzSWU0TUN0QmV4bU9iSWowdVBLTnVCQ01UTStxSVI2UWdnYXdEbWxXcXE4V0ZiUlNJMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
 server : "https://3FE47B904E57AF9E16E0B7F00C7EEABA.yl4.us-west-2.eks.amazonaws.com"
 }
 name : "kubernetes"
 }
 ]
 contexts : [
 [0]: {
 context: {
 cluster: "kubernetes"
 user : "aws"
 }
 name : "aws"
 }
 ]
 current-context: "aws"
 kind : "Config"
 users : [
 [0]: {
 name: "aws"
 user: {
 exec: {
 apiVersion: "client.authentication.k8s.io/v1alpha1"
 args : [
 [0]: "eks"
 [1]: "get-token"
 [2]: "--cluster-name"
 [3]: "cluster-eksCluster-367b66b"
 ]
 command : "aws"
 }
 }
 }
 ]
 }

You can now try out any of our infrastructure as code examples with this devcontainer.

Conclusion

As you can see, we’ve built our development with VS Code and Docker. Using a Pulumi devcontainer lets you build and deploy infrastructure without installing Pulumi or any associated dependencies on your computer. Moreover, you can share your development environment by creating a repository. Give it a try and let us know what you’re building on Slack or Twitter.

Deploy Kubernetes and Applications with Go

Levi Blackstone — Tue, 14 Apr 2020 00:00:00 +0000

We’re excited that Go is now a first-class language in Pulumi and that you can build your infrastructure with Go on AWS, Azure, GCP, and many other clouds. Users often ask, “Can I use Pulumi to manage Kubernetes infrastructure in Go today?” With the release of Pulumi 2.0., the answer is “Yes!”

Building your Kubernetes infrastructure with Infrastructure as Code offers several benefits:

Strongly-typed inputs and outputs for resources and invokes
First-class language support in editors/IDEs like vim, VS Code, GoLand, and atom
Reusable components and classes: Abstract standard functionality into classes and libraries for code reuse, and clean infrastructure design, instead of copy/pasting pages of YAML.
Embed Pulumi programs within other Go-based tools

These benefits provide a productive experience for Go developers using Kubernetes.

Tour of Kubernetes with Go

Let’s start with a basic example, e.g. deploying an Nginx pod into a Kubernetes cluster.

pod, err := corev1.NewPod(ctx, "pod", &corev1.PodArgs{
 Spec: corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 }
 },
 },
})
if err != nil {
 return err
}

We can deploy this with pulumi up.

$ pulumi up
Previewing update (nginx):
Type Name Plan
+ pulumi:pulumi:Stack kubernetes-go-nginx create
+ └─ kubernetes:core:Pod pod create
Resources:
+ 2 to create
Do you want to perform this update? yes
Updating (nginx):
Type Name Status
+ pulumi:pulumi:Stack kubernetes-go-nginx created
+ └─ kubernetes:core:Pod pod created
Resources:
+ 2 created
Duration: 16s

Although programming languages are imperative, Pulumi describes the desired state of the infrastructure. If we change our program, Pulumi will compute the minimum delta and apply it to our Kubernetes cluster, which will transition to the new desired state. For example, we can modify our Kubernetes resources in place inside the cluster by adding a label to our Pod:

pod, err := corev1.NewPod(ctx, "pod", &corev1.PodArgs{
+ Metadata: &metav1.ObjectMetaArgs{
+ Labels: pulumi.StringMap{"app": pulumi.String("nginx")},
+ },
 Spec: corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 }},
 },
})
if err != nil {
 return err
}

When we update the deployment with ‘pulumi up`, the preview shows that the pod is updated in place with the new label.

$ pulumi up
Previewing update (nginx):
Type Name Plan Info
pulumi:pulumi:Stack kubernetes-go-nginx
~ └─ kubernetes:core:Pod pod update [diff: ~metadata]
Resources:
~ 1 to update
1 unchanged
Do you want to perform this update? details
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:nginx::kubernetes-go::pulumi:pulumi:Stack::kubernetes-go-nginx]
~ kubernetes:core/v1:Pod: (update)
[id=default/pod-yxmgq7q2]
[urn=urn:pulumi:nginx::kubernetes-go::kubernetes:core/v1:Pod::pod]
~ metadata: {
~ labels: {
+ app: "nginx"
}
}
Do you want to perform this update? yes
Updating (nginx):
Type Name Status Info
pulumi:pulumi:Stack kubernetes-go-nginx
~ └─ kubernetes:core:Pod pod updated [diff: ~metadata]
Resources:
~ 1 updated
1 unchanged
Duration: 13s

The benefits of Go start to shine when we extract common code into a reusable component. For example, we can create a new ServiceDeployment component that combines both a Kubernetes Service and Deployment with opinionated defaults. Our ServiceDeployment component can describe entire Kubernetes applications (100s of lines of YAML), in a short snippet of Go:

// Initialize config
conf := config.New(ctx, "")

// Minikube does not implement services of type `LoadBalancer` so
// require the user to specify if we're running on minikube. If so,
// create only services of type ClusterIP.
isMinikube := conf.GetBool("isMinikube")

// Redis leader Deployment + Service
_, err := NewServiceDeployment(ctx, "redis-master", &ServiceDeploymentArgs{
 Image: pulumi.String("k8s.gcr.io/redis:e2e"),
 Ports: pulumi.IntArray{pulumi.Int(6379)},
})
if err != nil {
 return err
}

// Redis follower Deployment + Service
_, err = NewServiceDeployment(ctx, "redis-slave", &ServiceDeploymentArgs{
 Image: pulumi.String("gcr.io/google_samples/gb-redisslave:v3"),
 Ports: pulumi.IntArray{pulumi.Int(6379)},
})
if err != nil {
 return err
}

// Frontend Deployment + Service
frontend, err := NewServiceDeployment(ctx, "frontend", &ServiceDeploymentArgs{
 AllocateIPAddress: true,
 Image: pulumi.String("gcr.io/google-samples/gb-frontend:v4"),
 IsMinikube: pulumi.Bool(isMinikube),
 Ports: pulumi.IntArray{pulumi.Int(80)},
 Replicas: pulumi.Int(3),
})
if err != nil {
 return err
}

if isMinikube {
 ctx.Export("frontendIP", frontend.Service.Spec.ApplyT(
 func(spec *corev1.ServiceSpec) *string { return spec.ClusterIP }))
} else {
 ctx.Export("frontendIP", frontend.FrontendIP)
}

The complete implementation of the ServiceDeployment component is detailed in the Guestbook example in the Pulumi Examples repo on GitHub.

Building Docker Images for Kubernetes with Go

In the previous examples, we specified the Docker image to deploy as part of our Kubernetes Deployments by referring to an image in a registry. If we wanted to push our custom Docker image that uses our application’s source code, and deploy that in our Kubernetes Pod or Deployment, we can do it with the pulumi-docker package. For example, we can deploy our custom Nginx Docker image with the following:

image, err := docker.NewImage(ctx, "node-app", &docker.ImageArgs{
 ImageName: pulumi.String("my-username/my-nginx"),
 Build: &docker.DockerBuildArgs{
 Context: pulumi.String("./app"),
 },
})
if err != nil {
 return err
}

pod, err := corev1.NewPod(ctx, "pod", &corev1.PodArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{"app": pulumi.String("nginx")},
 },
 Spec: corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: image.ImageName,
 }},
 },
})
if err != nil {
 return err
}

We can use our Dockerfile from the app folder in our Pulumi project, instead of using the default Nginx image from DockerHub. For example, adding the following in our Dockerfile deploys customized files in the app/content folder to our Nginx server:

FROM nginx
COPY content /usr/share/nginx/html

When we deploy this with Pulumi, the Dockerfile is built locally, pushed to a registry, then made available by specifying the image name in the registry referenced from the Pod in Kubernetes. Our changes happen automatically, seamlessly deploying and versioning both the application code and infrastructure together with a simple pulumi up.

We can also push to another Docker container registry (like ACR, GCR, ECR, or others) by using additional parameters on the pulumi.docker.ImageArgs class.

Cloud + Kubernetes with Go

Pulumi works with both Kubernetes and cloud providers (AWS, Azure, GCP, and more.) Within the same Pulumi program, you can build a Kubernetes cluster and then deploy applications and services into that cluster.

For example, we can deploy a managed GKE cluster, and then deploy a Pod into it:

masterVersion := "1.14.10-gke.27"
cluster, err := container.NewCluster(ctx, "demo-cluster", &container.ClusterArgs{
 InitialNodeCount: pulumi.Int(2),
 MinMasterVersion: pulumi.String(masterVersion),
 NodeVersion: pulumi.String(masterVersion),
 NodeConfig: &container.ClusterNodeConfigArgs{
 MachineType: pulumi.String("n1-standard-1"),
 OauthScopes: pulumi.StringArray{
 pulumi.String("https://www.googleapis.com/auth/compute"),
 pulumi.String("https://www.googleapis.com/auth/devstorage.read_only"),
 pulumi.String("https://www.googleapis.com/auth/logging.write"),
 pulumi.String("https://www.googleapis.com/auth/monitoring"),
 },
 },
})
if err != nil {
 return err
}

ctx.Export("kubeconfig", generateKubeconfig(
 cluster.Endpoint, cluster.Name, cluster.MasterAuth))

k8sProvider, err := providers.NewProvider(
 ctx, "k8sprovider", &providers.ProviderArgs{
 Kubeconfig: generateKubeconfig(
 cluster.Endpoint, cluster.Name, cluster.MasterAuth),
}, pulumi.DependsOn([]pulumi.Resource{cluster}))
if err != nil {
 return err
}

namespace, err := corev1.NewNamespace(ctx, "app-ns", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("demo-ns"),
 },
}, pulumi.Provider(k8sProvider))
if err != nil {
 return err
}

appLabels := pulumi.StringMap{
 "app": pulumi.String("demo-app"),
}
_, err = appsv1.NewDeployment(ctx, "app-dep", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: namespace.Metadata.Elem().Name(),
 },
 Spec: appsv1.DeploymentSpecArgs{
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: appLabels,
 },
 Replicas: pulumi.Int(3),
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: appLabels,
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("demo-app"),
 Image: pulumi.String("jocatalin/kubernetes-bootcamp:v2"),
 }},
 },
 },
 },
}, pulumi.Provider(k8sProvider))
if err != nil {
 return err
}

This example first deploys a GKE Kubernetes Cluster into GCP, then deploys Kubernetes resources into that GKE Cluster. This example combines infrastructure and application deployment in a few dozen lines of declarative and strongly typed Go code.

Check out the full GKE in Go example on GitHub.

Conclusion

Kubernetes support is one of several significant new additions made available to Pulumi’s Go community and more improvements are on the way. Get started with Kubernetes and Go today, and let us know what you think!

Deploy Serverless Containers on Google Cloud Run with Pulumi

Mikhail Shilkov — Tue, 04 Feb 2020 00:00:00 +0000

Google Cloud Run is the latest addition to the serverless compute family. While it may look similar to existing services of public cloud, the feature set makes Cloud Run unique:

Docker as a deployment package enables using any language, runtime, framework, or library that can respond to an HTTP request.
Automatic scaling, including scale to zero, means you pay for what you consume with no fixed cost and no management overhead.
HTTP load-balancing out of the box simplifies the usage.

Cloud Run is targeted very specifically at stateless web applications. It uses ephemeral containers, and each execution is limited to 15 minutes.

Today, we will deploy our first Cloud Run services with Pulumi. Then, we’ll discuss pricing and compare Cloud Run to the competition.

Hello Cloud Run

We’ll start by deploying a pre-built container image provided by Google to a Cloud Run service.

To follow along, create a new Pulumi project with the Get Started with Google Cloud guide.

Enable Cloud Run for the project

As with any other service of Google Cloud, you need to enable it for the target project before the first deployment. You can do so with the gcp.projects.Service resource:

const enableCloudRun = new gcp.projects.Service("EnableCloudRun", {
 service: "run.googleapis.com",
});

Note: If you destroy the Pulumi stack and delete the projects.Service resource, the Cloud Run service will be disabled again. If you use Cloud Run in multiple Pulumi stacks, you should move the service management to a central location, i.e., a shared stack.

Choose a location

Cloud Run is a regional service: all container instances run in a single location of your choice.

Set the region setting to one of the currently supported locations: us-central1 (Iowa), us-east1 (South Carolina), europe-west1( Belgium), or asia-northeast1 (Tokyo).

pulumi config set gcp:region <region>

Then, the program can read the value and reuse it for all resources:

// Location to deploy Cloud Run services
const location = gcp.config.region || "us-central1";

Deploy a Cloud Run service

Google provides a pre-deployed “Hello Cloud Run” image at gcr.io/cloudrun/hello. The following resource deploys that image to your GCP project with the default settings.

const helloService = new gcp.cloudrun.Service("hello", {
 location,
 template: {
 spec: {
 containers: [
 { image: "gcr.io/cloudrun/hello" },
 ],
 },
 },
}, { dependsOn: enableCloudRun });

Expose unrestricted HTTP access

By default, Google does not expose HTTP endpoints of a Cloud Run service to the Internet. To make it publicly available, you should grant the roles/run.invoker role to allUsers.

const iamHello = new gcp.cloudrun.IamMember("hello-everyone", {
 service: helloService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

Note that the Contributor role is required for the user who executes Pulumi deployment to grant the access.

Try it out

Each Cloud Run service automatically gets a run.app subdomain. You can export the exact URL as a Pulumi output.

// Export the URL
export const helloUrl = helloService.status.url;

Run pulumi up to deploy all the resources.

$ pulumi up
...
Outputs:
~ helloUrl: "https://hello-585ad15-q4wszdxb2a-ew.a.run.app"

Navigate to the URL to see the welcome screen:

Congratulations, your first Cloud Run service is up and running. Now, it’s time to deploy some custom code.

Deploy a Custom Application

Cloud Run can run any Docker container so that you can write the application code in a language of your choice. I picked Ruby as an example of a language that is not supported by Cloud Functions.

Create a Ruby web app

Let’s create an app subfolder and place all application files there. The first file app.rb is a Hello World web application in Ruby.

require 'sinatra'

set :bind, '0.0.0.0'

get '/' do
 target = ENV['TARGET'] || 'Pulumi'
 "Hello #{target}!\n"
end

Gemfile and Gemfile.lock files configure the required gems (packages).

Define a Docker image

Dockerfile defines the container image to deploy. It is based on a generic Ruby image, copies the application files, and runs the web application.

# Use the official lightweight Ruby image.
FROM ruby:2.7-slim

# Install production dependencies.
WORKDIR /usr/src/app
COPY Gemfile Gemfile.lock ./
ENV BUNDLE_FROZEN=true
RUN bundle install

# Copy local code to the container image.
COPY . ./

# Run the web service on container startup.
CMD ["ruby", "./app.rb"]

Note that nothing in this web application is specific to Cloud Run.

Build and publish the Docker image

Cloud Run can only deploy images from Google Cloud Registry (GCR). Therefore, our Pulumi program needs to build the Docker image with the sample Ruby application and push it to GCR.

Run gcloud auth configure-docker in your command line to configure your local Docker installation to use GCR endpoints.

Run npm i @pulumi/docker to install the Pulumi Docker SDK. Then, add the docker.Image resource to the Pulumi program.

const myImage = new docker.Image("ruby-image", {
 imageName: pulumi.interpolate`gcr.io/${gcp.config.project}/my-ruby-app:v1.0.0`,
 build: {
 context: "./app",
 },
});

Pulumi takes care of building the image in the app folder and uploading it to GCR.

Deploy to Cloud Run

Now, you can deploy another Cloud Run service and point it to the custom image.

const rubyService = new gcp.cloudrun.Service("ruby", {
 location,
 template: {
 spec: {
 containers: [{
 image: myImage.imageName,
 resources: {
 limits: {
 memory: "1Gi",
 },
 },
 }],
 containerConcurrency: 50,
 },
 },
}, { dependsOn: enableCloudRun });

This snippet illustrates two additional configuration values. The memory limit defines the amount of RAM available to each container. Container concurrency defines how many requests each container instance may process in parallel at any given time. Both parameters influence the price of the deployment, as explained in the Pricing section below.

Currently, it’s not possible to use partial vCPU or multiple vCPUs per instance: your instance has always one vCPU assigned. You can adjust the memory from 128Mb to 2Gb.

Finally, don’t forget to enable unrestricted access and export the public URL.

const iamRuby = new gcp.cloudrun.IamMember("ruby-everyone", {
 service: rubyService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

export const rubyUrl = rubyService.status.url;

After deploying the program, you should be able to access the application with an HTTP call.

$ curl "$(pulumi stack output rubyUrl)"
Hello Pulumi!

You can find the complete example in the Pulumi examples Github repository.

Pricing

Cloud Run pricing is entirely consumption-based. After you exhaust the free tier, you start paying for four components, combined:

Per CPU time ($24.00 / million seconds)
Per memory consumption ($2.50 / million GB-seconds)
Per request ($0.40 / million requests)
Egress traffic, same as any other Google Cloud usage

Both time-based measurements are rounded up to the nearest 100 ms. All the provisioned memory is charged, the actual Mb/Gb consumption doesn’t matter.

It’s important to understand that the CPU and memory metrics are calculated per active instance (host, provisioned container), not per request. Each Cloud Run instance can handle multiple requests concurrently. Overlapping executions aren’t double charged: billable time begins with the start of the first request and ends at the end of the last request.

The following picture illustrates the pricing for three executions running on the same container host.

Multiple requests can share the allocated CPU and memory, so it makes sense to set the concurrency setting as high as possible for a given application (but not higher). This model is a big difference to Cloud Functions, which are charged for each request independently.

Comparing Cloud Run to Other Services

Despite the existence of multiple services that look somewhat similar, Cloud Run is unique in its capabilities.

Google Cloud Run vs Functions

Google Cloud Functions (GCF) service deploys snippets of code as functions, while Cloud Run deploys a web application packaged as a Docker image. Currently, GCF only supports three runtimes (Node.js, Python, and Go), while Cloud Run can run practically any language and any runtime.

GCF has a notion of events and triggers: it can natively integrate with Pub/Sub, Cloud Storage, Cloud Firestore. Cloud Run is all about handling HTTP requests: Any connection to another service has to go via HTTP.

Google Cloud Run vs AWS Fargate

AWS Fargate deploys container images. It requires an ECS cluster to run on and imposes more configuration burden on the user, including networking, load balancing, auto-scaling, and service discovery. Pulumi Crosswalk for AWS can help with these tasks.

Fargate is capable of hosting long-running workloads. Therefore, Fargate’s scaling model is not tied to individual requests, and there is no scale-to-zero out of the box.

With Cloud Run, there’s no notion of a cluster to manage, the scaling and billing models are based on individual requests. However, the workloads are HTTP-only, with a maximum duration of 15 minutes per call.

Google Cloud Run vs Azure Container Instances

Azure Container Instances (ACI) can also run arbitrary containers and has built-in HTTP endpoints. However, there’s no auto-scaling: you get a single host for each instance. Also, there’s no load balancing capability across multiple instances.

Cloud Run is elastically scalable, which positions it much better to host applications with variable workloads.

Conclusion

Cloud Run is a service with a unique combination of serverless ease of use and pricing with the full power of container deployments. It is easier to manage than Fargate, has automatic scaling missing from ACI, and greater flexibility compared to GCF. With concurrent requests model, Cloud Run is well-positioned to be significantly cheaper than Cloud Functions for some scenarios.

If you are excited, follow the example to Get Started with Cloud Run and Pulumi.

Managing AWS Containers with Pulumi

Sophia Parafina — Mon, 06 Jan 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

The Amazon Web Services (AWS) Cloud ecosystem is large and vibrant, so vast and vibrant that at times, it can be challenging to know where best to start! In the case of containers, Abby Fuller tweeted a descriptive summary about using AWS container services.

It looks straightforward, but it’s easy to get lost in the details when configuring multiple services to get your application running on AWS. Although AWS has extensive documentation and a point and click interface to these services, replicating then can be challenging, and it isn’t conducive to understanding the architecture. However, using infrastructure as code lets you see the details and results in a reproducible deployment that you can expand on. Let’s implement the diagram using Pulumi to deploy your container infrastructure with code in AWS.

Store your images

The AWS Elastic Container Registry (ECR) is a container registry that supports private container registries. ECR makes it easy to store, build, manage, and deploy container images and eliminates the need to operate your registry or use public registries, all in a highly available and scalable architecture.

In this example, we use Pulumi’s Crosswalk for AWS. Crosswalk for AWS is a collection of frequent tasks and best practices that simplify deploying infrastructure on AWS. We first declare a repository; this creates the AWS Container Repository resource. The last line of the code exports the URL so that we can access the repository after we have updated our Pulumi stack. After we’ve run pulumi up, the repository is ready to go, and you can use the Docker CLI to push, pull, and manage images.

import * as awsx from "@pulumi/awsx";

// Create a repository, as before.
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// And publish its URL, so we can push to it if we'd like.
export const url = repo.repository.repositoryUrl;

Choose your scheduler

There are two services for running containers in the AWS Cloud: Elastic Cloud Service (ECS), or Elastic Kubernetes Service (EKS). ECS is a proprietary service for running containers in the AWS Cloud. In contrast, EKS is a managed Kubernetes service for containers using a Kubernetes control plane. It has three master nodes distributed across three availability zones to provide high availability.

ECS is designed to work with other AWS services and provides more straightforward configuration and integration with them while providing high availability. In contrast, EKS is a managed Kubernetes service, and because Kubernetes is open-source, your infrastructure is portable to other cloud providers. Also, Kubernetes can provide fine-grain control over deployed services. The choice of which scheduler to use depends on your requirements.

You can create either an ECS or EKS cluster using Pulumi. If you wish to use ECS, @pulumi/aws provides all the primitives needed to build infrastructure on AWS.

import * as awsx from "@pulumi/aws";

// Create an ECS cluster explicitly, and give it a name tag.
const cluster = new aws.ecs.Cluster("custom", {
 tags: {
 "Name": "my-custom-ecs-cluster",
 },
});

If you wish to use EKS, use the Pulumi EKS library (@pulumi/eks) for Kubernetes primitives on AWS.

import * as eks from "@pulumi/eks";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

Choose where/how to run, Fargate or EC2

Fargate is an AWS service that runs containers. It is suited to running small workloads or batch workloads with occasional bursts that require scaling quickly. AWS EC2 is ideal for large workloads requiring many CPU cores and gigabytes of memory.

Fargate

In this example, we create a load balancer open on port 80, spin up two instances of our container, and publish the endpoint URL.

import * as pulumi from "@pulumi/pulumi";

const lb = new awsx.lb.ApplicationLoadBalancer("nginx");

const service = new awsx.ecs.FargateService("service", {
 cluster: cluster.arn,
 assignPublicIp: true,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: image.imageUri,
 essential: true,
 portMappings: [{
 containerPort: 80,
 targetGroup: lb.defaultTargetGroup,
 }],
 }
 }
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const appURL = pulumi.interpolate`http://${loadbalancer.loadBalancer.dnsName}`;

EC2 with Kubernetes

With Kubernetes, we create a deployment for the application and a service to make the application accessible via port 80. We publish both application endpoint to make it accessible and cluster config for use with kubectl CLI.

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Build!

Let’s put it all together for both ECS and EKS. To get started, follow the Pulumi AWS documentation to install Pulumi, install AWS Client set your AWS environment variables, and create a new project in Typescript.

$ mkdir get-started && cd get-started
$ pulumi new aws-typescript

In this example, we build the application in code using the awsx.ecr.Image resource. This resource uses Docker to build the image locally and push it to our repository. Make sure that you have Docker installed and running.

The container application for this example is an HTML page in NGINX. Make a ./app directory in your Pulumi project and add the Dockerfile below.

Dockerfile:

FROM nginx
COPY content /usr/share/nginx/html

Create a ./content director in the ./app directory and add the HTML file below.

<html>
 <head><meta charset="UTF-8">
 <title>Hello, Pulumi!</title>
 </head>
 <body>
 <p>Hello, containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

Now that setup is complete you project directory should look similar to this:

get-started/
├── Pulumi.dev.yaml
├── Pulumi.yaml
├── app
│   ├── Dockerfile
│   └── content
│   └── index.html
├── index.ts
├── node_modules
├── package-lock.json
├── package.json
└── tsconfig.json

ECS

Now we’re ready to start building our infrastructure. Replace the generated index.ts file with the example below. The example shows how to deploy containers on ECS using Fargate.

import * as awsx from "@pulumi/awsx";

// Create a repository
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// Build an image from the "./app" directory
// and publish it to our ECR repository.
export const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Create an ECS cluster explicitly, and give it a name tag.
const cluster = new aws.ecs.Cluster("custom", {
 tags: {
 "Name": "my-custom-ecs-cluster",
 },
});

// load balancer on port 80
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const service = new awsx.ecs.FargateService("nginx", {
 cluster,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: image.imageUri,
 memory: 512,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Publish the load balancer address so it's accessible.
export const appURL = lb.endpoint.hostname;

// Publish the repository URL, so we can push to it.
export const repoURL = repo.repository.repositoryUrl;

To deploy the application, run pulumi up. You’ll see a preview of the resources to be deployed. Accept the deployment by selecting yes, and when finished, your program lists the URLs for the application and the repository.

Outputs:
 appURL : "nginx-0296a2a-996897041.us-east-1.elb.amazonaws.com"
 image : "153052954103.dkr.ecr.us-east-1.amazonaws.com/my-repo-41f69f4:37c8add0fbd01474806c9c2319e62a857737de9e1460b09f935133860a069f4e"
 repoURL: "153052954103.dkr.ecr.us-east-1.amazonaws.com/my-repo-41f69f4"

Test to see if your application is working using either a browser or curl to see the web page.

$ curl nginx-0296a2a-996897041.us-east-1.elb.amazonaws.com
<html>
 <head><meta charset="UTF-8">
 <title>Hello, Pulumi!</title>
 </head>
 <body>
 <p>Hello, containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

EKS

The EKS example uses the @pulumi/eks and @pulumi/kubernetes node modules. Add the eks and kubernetes modules to the project.

$ node install “@plulumi/eks”
$ node install “@plulumi/kubernetes”

Like the ECS example, the EKS example builds the application image locally and pushes it into our custom repository. The application deployment has two replicas in the configuration. The service uses the EKS cluster, my-cluster, we declared to create a load balancer with port 80 open.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create a repository, as before.
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// Build an image from the "./app" directory. and publish it to our ECR repository.
export const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster")

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: image.imageUri,
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

// Publish the repository URL.
export const repoURL = repo.repository.repositoryUrl;

// Publish the URL for the load balanced service.
export const appURL = service.status.loadBalancer.ingress[0].hostname;

As with the ECS example, run pulumi up to deploy the application. To check the deployment, use either curl or a browser to see the page.

Want to know more?

Although these examples are simple, they demonstrate the basic building blocks for building, storing, and managing AWS Containers. They also show how to create ECS or EKS clusters for deploying apps. To get started with AWS and Pulumi check out the AWS Guide for core services and the Crosswalk for AWS guide for convenience APIs that simplify deploying infrastructure as code. For a deeper dive into managing containers on AWS, check out our blogs on ECS vs Fargate vs EKS: The Lowdown on Containers in AWS and How to Scale Your Amazon EKS Cluster: EC2, Managed Node Groups, and Fargate.

Run Your Own RSS Server on AWS with Pulumi

Christian Nunciato — Mon, 16 Sep 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

It’s been a few years since Google shut down Google Reader, and while a number of nice commercial alternatives have sprung in its wake, none of them has ever been quite the right fit for me personally.

So a while back, after far too much time spent wandering the blogsphere manually, typing URLs into address bars by hand, I decided to go looking to see whether the universe had produced an open-source solution to this problem — and to my surprise and delight, it had! Miniflux is an excellent little open-source RSS server and reader, written in Go and backed by PostgreSQL, that also happens to be packaged as a Docker container. So in this post, I’ll show how easy it is to deploy a Miniflux server of your own on AWS, using only Pulumi and a few lines of TypeScript.

If you’re already comfortable with Pulumi, and you just want to get up and running, I’ve set up a GitHub repo (complete with a Deploy with Pulumi button!) that should have all you need to get going. Just click the button, set a few configs (like your RSS server’s administrative password, which will be stored as an encrypted Pulumi secret, and follow the prompts. Your shiny new server should be up and running within minutes.

Sketching it Out

First, let’s have a look at the app we’ll be building. It’ll consist mainly of two parts:

The Miniflux service, which runs as a web application and API service, and
The Miniflux database, a PostgreSQL instance whose schema and data are fully managed by the Miniflux service.

As I mentioned, the service is packaged as a Docker container that runs on port 8080 by default, and exposes a number of environment variables we can use to configure it, including:

DATABASE_URL, a PostgreSQL connection string used by the service to communicate with the PostgreSQL database,
ADMIN_USERNAME and ADMIN_PASSWORD, which we’ll use to sign into the service for the first time, and
flags for CREATE_ADMIN and RUN_MIGRATIONS, which denote whether to create the administrative user and run database migrations on startup.

We’ll set these properties in the course of developing our program. The service itself, as a container, can easily be run on AWS Fargate, so we can use the @pulumi/awsx package to declare it, and for the database, we’ll use @pulumi/aws to provision a small RDS instance of PostgreSQL. And finally, to make the Miniflux service publicly accessible (initially over HTTP; we’ll switch to HTTPS later), we’ll use an AWS Network Load Balancer, since we’ll only need to be able to route traffic by ports and protocols.

Our architecture is therefore shaping up to look something like this:

Let’s get started!

Setting Up

We’ll start by creating a new Pulumi project. (If you’re completely new to Pulumi, it might be good to begin with our Getting Started guide, which walks you through installing Pulumi and configuring it for your cloud provider.) We’ll use the built-in aws-typescript project template:

$ mkdir miniflux && cd miniflux
$ pulumi new aws-typescript

Once you’ve stepped through the prompts, you should be left with a minimal index.ts file that defines and exports an S3 bucket — but since we won’t be needing an S3 bucket for this project, we can remove that code, and keep only the imports we’ll need:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

Now let’s define a few configuration values for the project.

Configuring the Stack

While we could certainly hard-code all of these values into our program, it’d be better to use Pulumi to set them, since doing so give us the option to vary them by stack (say, if we wanted to run this particular app in multiple environments), but more importantly, to set some passwords for the database user and service administrator. So let’s do that first, so we’ll have them all ready as we develop our program:

$ pulumi config set db_name miniflux
$ pulumi config set db_username miniflux
$ pulumi config set db_password somesupersecretpassword --secret
$ pulumi config set admin_username admin
$ pulumi config set admin_password anothersupersecretpassword --secret

You can change any of these values if you like (and you should definitely change the two passwords, of course), but note that the passwords are set with the --secret flag, which encrypts their values using Pulumi’s built-in support for secrets.

Okay. With our configuration set, we’re ready to start building.

Writing the Code

The Pulumi program we’re writing will:

import our newly created Pulumi configuration, so we can use its values in our program;
create a new DB Subnet Group using the default VPC for your region;
create a new instance of PostgreSQL, with minimal settings, placing it into the subnet group and giving it access to your default ECS Cluster;
create a new Network Listener to define a publicly accessible URL for the Miniflux service;
create a new Fargate service for the Miniflux app, in your default ECS Cluster, passing it the newly created DB connection and Pulumi config settings; and finally,
export the URL as a Pulumi stack Output so we can navigate to the service.

Let’s get coding!

Replace the contents of index.ts with the following code, which comprises the whole program:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Import our Pulumi configuration.
const config = new pulumi.Config();
const dbName = config.require("db_name");
const dbUsername = config.require("db_username");
const dbPassword = config.require("db_password");
const adminUsername = config.require("admin_username");
const adminPassword = config.require("admin_password");

// Get the default VPC and ECS Cluster for your account.
const vpc = awsx.ec2.Vpc.getDefault();
const cluster = awsx.ecs.Cluster.getDefault();

// Create a new subnet group for the database.
const subnetGroup = new aws.rds.SubnetGroup("dbsubnets", {
 subnetIds: vpc.publicSubnetIds,
});

// Create a new database, using the subnet and cluster groups.
const db = new aws.rds.Instance("db", {
 engine: "postgres",
 instanceClass: aws.rds.InstanceTypes.T3_Micro,
 allocatedStorage: 5,
 dbSubnetGroupName: subnetGroup.id,
 vpcSecurityGroupIds: cluster.securityGroups.map(g => g.id),
 name: dbName,
 username: dbUsername,
 password: dbPassword,
 skipFinalSnapshot: true,
});

// Assemble a connection string for the Miniflux service.
const connectionString = pulumi.interpolate `postgres://${dbUsername}:${dbPassword}@${db.endpoint}/miniflux?sslmode=disable`;

// Create an NetworkListener to forward HTTP traffic on port 8080.
const listener = new awsx.lb.NetworkListener("lb", { port: 8080 });

// Create a Fargate service consisting of just one container instance (since that's all we
// really need), passing it the cluster, DB connection and Pulumi config settings.
const service = new awsx.ecs.FargateService("service", {
 cluster,
 desiredCount: 1,
 taskDefinitionArgs: {
 containers: {
 service: {
 image: "miniflux/miniflux:latest",
 portMappings: [
 listener
 ],
 environment: [
 { name: "DATABASE_URL", value: connectionString },
 { name: "RUN_MIGRATIONS", value: "1" },
 { name: "CREATE_ADMIN", value: "1" },
 { name: "ADMIN_USERNAME", value: adminUsername },
 { name: "ADMIN_PASSWORD", value: adminPassword },
 ]
 }
 }
 }
});

// Export the publicly accessible URL.
export const url = pulumi.interpolate `http://${listener.endpoint.hostname}:${listener.endpoint.port}`;

With the program in place, we can bring it to life with pulumi up:

$ pulumi up

In a few minutes, you should see that Pulumi has created 32 new resources:

And if you examine the logs for this stack using pulumi logs -f, you’ll see that the database was created, migrations were run, administrative user was created, and the service is now listening:

Finally, looking back at the output of pulumi up, you’ll see the URL we exported, to which you can navigate, sign in with the administrative user created by our program, and start RSSing:

Excellent! You’ve now got your very own fully functioning RSS service, created with only a few dozen lines of code.

Technically, we’re done — but there are a couple of things we might like to do to make this project a bit more complete.

Running on Port 80

Right now, our service’s load balancer is configured to listen and forward on the same port, 8080. If we wanted to have the load balancer listen on port 80 (or some other port) instead, we’d need to change its declaration a bit, to create the load balancer and target group more explicitly. Let’s do that:

// Create a load balancer, target group and network listener explicitly.
const listener = new awsx.lb.NetworkLoadBalancer("lb")
 .createTargetGroup("group", { port: 8080 })
 .createListener("listener", { port: 80 });

Run pulumi up, and you’ll see that the service is now accessible at a new host and port (though the database was of course left unchanged):

Great! But what if we wanted to run over HTTPS, instead?

Running over HTTPS

In order to run the Miniflux service over HTTPS, you’ll need to have obtained an SSL certificate from AWS Certificate Manager that corresponds with the domain you plan to use for the service.

Provided you’ve obtained that certificate, it’s defined in the same AWS region as the one you’ve configured for your Pulumi stack, and you’re able to make changes to the DNS records for the domain associated with the certificate, then updating the Pulumi program is easy — just change the listener declaration to use TLS and port 443, and add a certificateArn property to apply the certificate:

// Run the service over HTTPS, terminating SSL at the load balancer and forwarding to port 8080 on the container.
const listener = new awsx.lb.NetworkLoadBalancer("lb")
 .createTargetGroup("group", { port: 8080, protocol: "TCP" })
 .createListener("listener", { port: 443, protocol: "TLS", certificateArn: "arn:aws:acm:..." });

Run pulumi up one last time, and you should see your service now running on port 443 — but you won’t be able to access it until you add CNAME record to the DNS settings for your domain (I happen to use Google for mine) mapping a new subdomain to your load balancer’s hostname:

And with that, you should now be able to browse your RSS server securely:

Finishing Up

In this post, we’ve seen how easy it is to run a container as a service connected to an RDS database with Pulumi, and to expose that container securely on the web. If we wanted, we could go even farther — we could refactor the program into Pulumi Components, perhaps (one for the service, one for the database), package it up for sharing on npm, and so on.

But we’ll leave those improvements for another day. For now, let’s enjoy what we’ve created! And start catching up on all that reading we’ve missed.

ECS vs Fargate vs EKS: The Lowdown on Containers in AWS

Joe Duffy — Thu, 20 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

An updated breakdown of when to prefer Fargate and when EC2 can be found in Fargate vs EC2

Amazon offers multiple solutions for running containers in AWS, through its managed Elastic Container Service (ECS). This includes three major approaches: ECS managed automatically with Fargate, ECS backed by EC2 instances, and Elastic Kubernetes Service (EKS), delivering the full power of Kubernetes. It's not always easy to choose between these, so in this article we provide some basic guidance on the tradeoffs you'll encounter when choosing.

The Options: ECS vs Fargate vs EKS

At Pulumi, we work with customers to deploy AWS applications all the time, from early development to scaling production environments worldwide, using infrastructure as code and often continuous delivery. During this, we’ve run across a breadth of container workloads and requirements, and developed best practices, which this article distills into some basic advice. As with any advice, the right answer depends on your scenario’s specific requirements.

Here are the three major options, roughly speaking from easiest to hardest:

Option 1 - ECS Fargate

Amazon ECS with Fargate launch type allows you to deploy and scale your containerized workloads, automatically, without needing to manage clusters of EC2 instances by hand. You focus on your Dockerized applications, with minimal extra declarative metadata about the requirements of those applications, and not the details of the underlying compute itself. One could say this is “serverless containers.” Of course, there are servers running your containers, however the ECS control plane makes informed decisions on your behalf about when to add capacity, where, and by how much.

To run a workload in ECS, you use a standard Dockerfile to author, build, and publish your container image, and then run that image inside your ECS cluster. Tasks and services are authored using ECS definitions, which specify the container image to run, instance count, and any CPU, memory, networking, and disk requirements your workload has.

Using Pulumi, these definitions can be easily expressed declaratively in code, like so:

import * as awsx from "@pulumi/awsx";

// Create a load balancer on port 80 and spin up two instances of Nginx.
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const nginx = new awsx.ecs.FargateService("nginx", {
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: "nginx",
 memory: 128,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const url = lb.endpoint.hostname;

This example runs 2 load balanced instances of an NGINX web server using ECS Fargate.

ECS Fargate is often a great choice when getting started, when you don’t have stringent requirements around cost, and/or if you’re a small to medium sized organization who prefers simpler solutions. Many teams are successful running it scale too, however Fargate doesn’t give you as much control over persistent storage, privileged containers, custom scheduling, or special VM types like GPUs or FPGAs. For those, EC2 backed ECS is a better choice.

Option 2 - ECS Backed by EC2 Instances

Amazon ECS with EC2 launch type gives you similar capabilities to ECS managed by Fargate – in that you can deploy containerized workloads, declaratively specify their requirements, and let ECS do the placement and overall scheduling of them – but with full control over the compute environment those containers run in. That includes control over the precise numbers of servers, AMIs they are running, and their auto-scaling policies.

To use an EC2 backed cluster, you will need to create a cluster and configure the EC2 launch and autoscaling configurations, and then simply use the awsx.ecs.EC2Service class to create your service instead. Other than these differences, the resulting code would look very similar to Option 1 - ECS Fargate above.

For sophisticated AWS organizations who already know how to fine tune and run compute at scale efficiently and cost effectively, managing your ECS cluster’s EC2 instances explicitly is often a good choice. The good news is that you can easily start with Fargate, and then over time, shift to managing the EC2 compute by hand if you prefer. Managing your cluster amounts to managing fleets of EC2 instances, CloudWatch logging, and standard AWS services.

Option 3 - Elastic Kubernetes Service (EKS)

Amazon Elastic Kubernetes Service (EKS) offers a very different approach to running containerized workloads than ECS Fargate or EC2. EKS is a fully managed offering that runs Kubernetes inside of your AWS account, making it easier to operate Kubernetes, in addition to integrating with many AWS services like ELB for load balancing, IAM for cluster authentication, and CloudWatch for logging and diagnostics. Using EKS, you don’t need to explicitly provision, manage, and upgrade the Kubernetes control plane’s configuration, compute, and storage.

Kubernetes is an open standard for running containers across a variety of public and private cloud environments. Most public cloud vendors now offer an EKS-like managed offering, including Azure (AKS), Google Cloud (GKE), and Digital Ocean, and the availability of on-premises private cloud configurations enables hybrid cloud for large organizations. Kubernetes in inherently more complicated to operate than the equivalent ECS solutions mentioned earlier, because it requires becoming an expert in Kubernetes, which requires more specialization in its tools and management practices. But the potential payoff is larger for organizations wanting to go “all in” on cloud native technologies to facilitate either multi-cloud or hybrid computing – challenges most large enterprises are facing today.

In the EKS case, as with ECS, we continue using standard Dockerfiles, however we use the Kubernetes object model to configure and run workloads. Instead of task definitions, you use Kubernetes objects like Namespaces, Pods, Deployments, and Services. This full API is available to specify in code using Pulumi, like so:

import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Like the ECS example earlier, this spins up 2 load balanced instances of the NGINX web server.

You will find that the entirety of Kubernetes is more feature rich and offers more control over scheduling and scaling your workloads, but with that richness also comes added complexity that will increase training, and time to ramp up, in addition to ongoing maintenance costs. To learn more about Pulumi’s support for Kubernetes, go here.

Private ECR Container Registries

In all of these cases, Amazon Elastic Container Registry (ECR) lets you publish Docker container images to a privately hosted repository inside your AWS account. The benefit of this is that images are stored within your AWS account, where access is governed by IAM and, where they are stored closer to your workloads (maximizing push/pull efficiency). This is an alternative approach to, say, publishing images to the Docker Hub which entails a different authentication and pushing and pulling images over the Internet, where performance can be an issue.

Next Steps

Pulumi Crosswalk for AWS supports three main options for running containers in AWS – ECS Fargate, ECS with EC2 instances, and EKS – each of which integrates deeply with other AWS services like IAM, ELB, and CloudWatch. It’s possible to use any of these services without Pulumi, but there are many benefits to Pulumi’s infrastructure as code.

In short, to get your containers up on AWS, in an easy yet production-ready way, get started for free with Pulumi for AWS today!

After getting started, here are some additional resources to go deeper on specific topics:

Introducing Pulumi Crosswalk for AWS: The Easiest Way to AWS

Luke Hoban — Mon, 10 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Amazon Web Services provides an incredible platform for developers to build cloud-native applications, and is used by millions of customers of all sizes. The building block services that AWS offers enable teams to offload undifferentiated heavy-lifting to AWS. To maximally benefit from these services though, cloud engineering teams must learn how to compose all of these building blocks together to build and deliver their own applications. Today, this is still too hard. Getting from your laptop to a production-ready AWS deployment frequently takes days or weeks instead of minutes or hours. And AWS building block services frequently leave you to re-implement (and re-discover) best-practices instead of providing these as smart defaults.

Pulumi Crosswalk for AWS is a new open source library of infrastructure-as-code components that make it easier to get from zero to production on AWS, easier to adopt AWS best practices by default, and easier to evolve your AWS infrastructure as your application needs mature.

Crosswalk for AWS offers higher-level components for a wide range of the AWS platform - from serverless (Lambda, API Gateway and more) to containers (ECS, Fargate) to Kubernetes (EKS) and across networking (VPCs, SecurityGroups), monitoring (Cloudwatch Dashboards and Alarms) and load balancing (ALB and NLB). Because Crosswalk builds on the primitive building blocks of the AWS platform, you can use 100% of what AWS offers, and can seamlessly move between high-level components and low-level platform primitives, mixing-and-matching as needed.

In some of these domains, there have been successful tools that have helped developers more easily work with one specific aspect of AWS - Serverless Framework for serverless, Docker Compose for ECS, eksctl for EKS. But these tools have traditionally been limited to one specific kind of compute, limiting what users can accomplish, fragmenting development of complete cloud solutions, and leading to sharp cliffs when application needs evolve. Crosswalk for AWS offers the same kind of simpler and more productive developer interface as these tools, but applied uniformly across the entire AWS platform. Simple things are simple, and arbitrarily complex things are possible.

Whether you are starting a new SaaS project at a startup, migrating an enterprise application to cloud-native infrastructure, or delivering a new service as part of an established cloud application, Crosswalk for AWS provides the easiest way to get from zero to production, and to then evolve with your projects’ needs. You can get started with Crosswalk for AWS today!

Note: Crosswalk for AWS projects are authored using the Pulumi infrastructure-as-code tools. Pulumi allows you to define your infrastructure using code. As your infrastructure code evolves, Pulumi makes the minimal required changes to your cloud resources, ensuring that your infrastructure is updated without downtime (if possible).

From Zero to Production And Beyond with Crosswalk for AWS

Serverless (Lambda, API Gateway)

When getting started with a new cloud project, the starting point is often just serving a simple API on the internet. With just a few lines of code and tens of seconds to deploy, Crosswalk for AWS gives us our API.

import * as awsx from "@pulumi/awsx";

const api = new awsx.apigateway.API("api", {
 routes: [{
 method: "GET",
 path: "/",
 eventHandler: async (ev, ctx) => {
 return {
 statusCode: 200,
 body: JSON.stringify(ev),
 };
 }
 }],
})

export const endpoint = api.url;

By building on top of great AWS building blocks like Lambda, API Gateway, IAM and more, we avoid needing to worry about infrastructure, pay nearly zero fixed costs, and gain the ability to iterate quickly. With just a few more lines of code, we can wire up to our own domain, add authorization, provision a database, or chain together a more complex event-driven application.

Containers (ECS, Fargate)

As our needs grow, we might find we need to deploy containers - either custom applications or existing Docker images. Again, just a few lines of code gets our container running in production in AWS.

import * as awsx from "@pulumi/awsx";

// Create the cluster
const cluster = new awsx.ecs.Cluster("cluster");

// Configure a Load Balancer
const alb = new awsx.lb.ApplicationLoadBalancer("lb", {
 securityGroups: cluster.securityGroups,
});
const web = alb.createListener("web", { port: 80, external: true });

// Define the scaled-out Fargate service
const appService = new awsx.ecs.FargateService("app-svc", {
 cluster,
 taskDefinitionArgs: {
 container: {
 image: awsx.ecs.Image.fromPath("app", "./app"),
 portMappings: [ web ],
 },
 },
 desiredCount: 5,
});

export const url = web.endpoint.hostname;

Using ECS, Fargate, ECR and ELB, we get a robust production-ready container deployment - horizontally scaled out, load-balanced, and integrated with a private image repository. With just a few more lines of code, we can add autoscaling, customize our ECS cluster, or wire through advanced container configuration (volumes, environment variables, and more).

Networking (VPC)

To adopt best practices security posture, we may want to move our cluster into a custom VPC, and run our containers in private subnets. We can define and configure our VPC, and then redeploy our containers into this new VPC with ease:

import * as awsx from "@pulumi/awsx";

const vpc = new awsx.ec2.Vpc("vpc", {
 numberOfAvailabilityZones: 3,
});

const cluster = new awsx.ecs.Cluster("cluster", {
 vpc: vpc,
});

// ... same as before ...

The Vpc component builds on best-practices Virtual Private Cloud design patterns based on AWS guidance and documentation. By building in simple defaults for routing, subnet structure, NATs and multi-AZ you can get started quickly without having to re-discover these best practices. And as your needs grow, you can deeply customize and evolve your VPC structure - defining custom CIDR blocks, customizing Internet and NAT Gateways or additional additional private subnets.

Kubernetes (EKS)

As our project continues to evolve, we may want to move our container workloads into Kubernetes using EKS. We can stand up an EKS cluster with just a few lines of code, then customize our cluster and deploy Kubernetes workloads into our cluster.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("vpc");

// Create the EKS cluster itself.
const cluster = new eks.Cluster("cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.privateSubnetIds,
 instanceType: "m5.large",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 deployDashboard: true,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Building on EKS we get all the benefits of managed Kubernetes paired with the platform capabilities of AWS. With Crosswalk for AWS, we can get started quickly, and then evolve to take advantage of all of these platform features, like customized node groups, private networking, and even deploying Kubernetes YAML and Helm charts to our cluster.

Monitoring (CloudWatch)

Getting our infrastructure up and running in AWS is great, but we also want to be able to monitor it. We can add a few lines to any of the solutions above to define metrics, dashboards, alerting and more to monitor our AWS infrastructure.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create a bucket and a function to log all new object created events
const bucket = new aws.s3.Bucket("b");
const subscription = bucket.onObjectCreated("newObject", async (ev) => {
 console.log(JSON.stringify(ev));
});

// Create a CloudWatch Dashobard for our functions invocations
const dashboardName = "funcDashboard";
const dashboard = new awsx.cloudwatch.Dashboard(dashboardName, {
 widgets: [
 new awsx.cloudwatch.LineGraphMetricWidget({
 title: "Lambda invocations",
 width: 14,
 metrics: awsx.lambda.metrics.invocations({
 function: subscription.func, statistic: "Sum", period: 60, }), }), ], });

// Export the URL of the dashboard in the AWS console
export const dashboardUrl =
 `https://${aws.config.region}.console.aws.amazon.com/cloudwatch/home?` +
 `region=${aws.config.region}#dashboards:name=${dashboardName}`;

With CloudWatch being deeply integrated into all AWS services, we can easily build up robust logging, alerting and dashboarding solutions across our AWS infrastructure. With just a few more lines of code we can get logs for our functions and containers, create alarms when we cross critical thresholds, and create rich dashboard layouts all of which will version along with our infrastructure!

Early Feedback

We’ve been working on pieces of Crosswalk for AWS since before we launched Pulumi, and we’re excited to be launching Crosswalk for AWS today to bring these together for all AWS users. Over the last few months, users have built on Crosswalk for AWS for all of the scenarios above and more.

Tableau has used Crosswalk for AWS to accelerate their Kubernetes migration on AWS, building on the networking and EKS capabilities of Crosswalk for AWS.

“We’ve been happily using Pulumi’s EKS support for more than three months now. Our team was looking for an end-to-end solution to tame the complexity of Kubernetes on AWS and ensure we adhere to AWS best practices. Pulumi’s Crosswalk effort has equipped our team to scale far better, our delivery is now automated, and we can now deliver new application and infrastructure features with much faster turn-around, sometimes in just a few hours. Pulumi is a key contributor to our team’s improvement in productivity.” - Pankaj Dhingra, Sr Director, Cloud Engineering, Tableau

Compute Software has used Crosswalk for AWS to go from zero to production, and then to evolve their infrastructure in AWS, all without needing a dedicated cloud infrastructure engineering team.

“We use Crosswalk for AWS with Fargate. It’s been instrumental in getting us into production. Deploying to the cloud is necessarily complex. Crosswalk streamlines the process by exposing the controls you care about, always allowing you to dive deeper when needed.” - Kenny Williams, Co-Founder, Compute Software

We’re excited by the success these teams and many more have already had with Crosswalk for AWS, but even more excited about the potential to truly democratize access to the AWS platform and unlock highly productive cloud development for all kinds of engineering organizations.

What’s Next?

We’re still just scratching the surface of what’s possible with Crosswalk. We’re continuing to evolve the open source projects to simplify additional features of the AWS platform. And we’re also working to bring the same approach to other cloud platforms, including Azure, GCP and Kubernetes.

We believe that making it easier for cloud engineers to develop and deploy natively in the cloud will unlock creativity and empower developers to take advantage of more of the rich capabilities of AWS. We can’t wait to see what you build with Crosswalk!

Crosswalk for AWS is free and open source, and you can get started today.

For more on Crosswalk for AWS see:

Pulumi and Docker: Development to Production

Sean Gilespie — Wed, 15 May 2019 00:00:00 +0000

Here at Pulumi, we’re big fans of Docker: at this point there is little doubt that Docker has completely revolutionized the way that we think about deploying software. However, even in 2019, it’s quite difficult to get Docker containers to production. docker run is great, and we all love it, but unfortunately it’s quite a big leap from docker run to running your container in a production-ready environment.

We recently wrote a blog post about using AWS Fargate to run your docker containers with our open source packages. In this blog post we’re going to focus on another interesting aspect of Pulumi: being able to re-use your infrastructure code for both development and production!

It’s All Code: Composing Docker Containers

In addition to the number of cloud and infrastructure providers that Pulumi supports, Pulumi also supports defining Docker resources in code. Let’s look at this code snippet of Pulumi TypeScript code:

// This program encodes a complete application: a container running Redis Commander,
// and a container running Redis. Redis Commander connects directly with Redis.
//
// In order for the two containers to communicate directly, they must be placed in the
// same network. Before creating the containers themselves, we first create the network
// that will contain both of them.
const network = new docker.Network("net");
// Also before creating a container, we must first obtain a "RemoteImage", which is a reference to an external image
// that is downloaded to the local machine. In this case, we're referring to an image on Docker Hub.
const redisImage = new docker.RemoteImage("redis-image", {
name: "redis:latest",
keepLocally: true, // don't delete the image from the local machine when deleting this resource.
});
// We can create a container using a reference to the name of the image we just downloaded and a reference to the name
// of the network that this container should use.
const redisContainer = new docker.Container("redis", {
image: redisImage.name,
networksAdvanced: [{ name: network.name }],
restart: "on-failure",
});
// We do the same thing for the Redis Commander container.
const redisCommanderImage = new docker.RemoteImage("redis-commander-image", {
name: "rediscommander/redis-commander:latest",
keepLocally: true,
});
const redisCommanderContainer = new docker.Container("redis-commander", {
image: redisCommanderImage.name,
restart: "on-failure",
networksAdvanced: [{ name: network.name }],
envs: [
// Pulumi resources have "output" properties, which are properties that are set upon successful completion of
// the resource operation. These "outputs" are like promises - they can't be used directly but instead must be
// manipulated through use of the `apply` function.
//
// Here, we are using the name of the Redis commander to build an environment variable `REDIS_HOST` that points
// to the network identity of the Redis container we just created.
redisContainer.name.apply(name => `REDIS_HOST=${name}`),
],
// Finally, we expose some ports. Redis Commander listens on port 8081, so we'll map that to external
// port 3000 for easy consumption on the user machine.
ports: [{
internal: 8081,
external: 3000,
}]
});

Here we are looking at a complete Pulumi program that does a bunch of things with Docker:

We create a Docker network named net, which we’ll use to connect two containers together
We pull the redis:latest image from Docker Hub and create a container using it, named redis, which we attach to the network we just created
We pull the rediscommander/redis-commander:latest image from Docker Hub and also create a container using it, named redis-commander, which likewise is attached to the network and exposes some ports on the host machine.

This is fairly nontrivial, but we’ve done it in code right here. Using code we’ve replicated many of the features offered by tools like docker-compose, but without writing a single line of YAML. Running pulumi up with this code results in a working instance of Redis Commander listening on port 3000 on your machine. Nice and simple!

Moving to Production

Orchestrating Docker containers on your local machine while developing is great, but when it comes time to pushing containers to production, you can’t rely on Docker alone anymore. There are a number of excellent container orchestrators out there but there few are as successful as Kubernetes. The way that you launch production workloads in Kubernetes is significantly different than how you would do it using docker run on your own, so it’s often the case that you’d need to write another series of YAML documents to get your application into production.

With Pulumi, we’re just writing code. Since we’re just writing code, we can do exactly what we would do if we were writing application code: abstract!

// Since we're using a general-purpose programming language, we can now use that language to provide multiple
// implementations of the same abstraction. In this case, we define an abstract "Redis" component whose only property is
// that it has some network identity (host) that consumers can connect to and talk to a Redis connection.
export abstract class Redis extends pulumi.ComponentResource {
 public abstract readonly host: pulumi.Output<string>;

 public static create(name: string, args: RedisArgs, opts?: pulumi.ComponentResourceOptions): Redis {
 // Since we are using TypeScript and RedisArgs is the union of argument types for each resource. Since args.type
 // has type "amazon" | "docker" | "kubernetes", the compiler assists us in making sure that we don't
 // accidentally mix up argument types.
 switch (args.type) {
 case "amazon":
 return new AmazonRedis(name, args, opts);
 case "docker":
 return new DockerRedis(name, args, opts);
 case "kubernetes":
 return new KubernetesRedis(name, args, opts);
 }
 }

 constructor(ty: string, name: string, args: RedisArgs, opts?: pulumi.ComponentResourceOptions) {
 super(ty, name, args, opts);
 }
}

export type RedisArgs = DockerRedisArgs | KubernetesRedisArgs | AmazonRedisArgs;

We’ve departed a bit from our previous example, but we’re still writing code. In particular, what we’re doing here is defining a new, abstract, Redis component that takes the Redis container from the previous example and abstracts it away a little bit. This class we’ve created still represents Redis, but completely abstracts away where and how the Redis instance is running. In this case, it can run in any one of three different configurations:

Redis backed by AWS ElastiCache (“amazon”)
Redis running as a Docker container on your machine (“docker”)
Redis running as a Deployment on a Kubernetes cluster (“kubernetes”)

Now, you can use this Redis component completely in the abstract, like this:

const redis = Redis.create("redis", {
 type: "docker",
 network: network,
});

We’re requesting a “docker” container, but we could just as easily switch this out for “amazon” and “kubernetes”, and suddenly our Redis workload is transparently deployed in different ways, all without disrupting our development environment!

Pulumi: Infrastructure as Software

We often say on this blog that Pulumi is “infrastructure as code”, and that sentence is definitely true, but I prefer calling it “infrastructure as software” instead. The distinction is subtle, but in my mind Pulumi brings to the realm of infrastructure the variety of tools we already use for software engineering:

Abstraction, encapsulation, and code reuse for infrastructure and applications
Testing, both unit and integration
IDEs and tools for detecting errors extremely early in a developer’s inner loop, instead of at deployment time

Pulumi is open source, free to use, and works today with a variety of clouds and bring a little more software and less code into your infrastructure! If you’d like to see more about this particular code demo, check out my DockerCon EU 2018 talk where I dive into this in detail. The code for this post and the talk are in this repository.

Get Started with Docker on AWS Fargate using Pulumi

Joe Duffy — Tue, 30 Apr 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Update: Check out the Learning Machine Case Study where provisioning went from 3 weeks to 1 hour with Pulumi and AWS.

“The impact of serverless capabilities was also transformative for the Learning Machine business. Pulumi enabled a rapid shift to Amazon ECS, AWS Fargate and AWS Lambda — the net effect of which was a 67% reduction in AWS charges. This enabled the team to spend less time focused on maintaining existing infrastructure and more time deploying new applications on AWS and adding new customers.

Pulumi is the foundational technology that allowed us to transform our organization,” said Hughes. The entire DevOps process was streamlined and in addition to realizing better productivity and higher quality, the team has new insight into their SaaS offering that they never thought possible.”

The Docker Getting Started tutorial shows how to develop, build, and run a modern containerized application, from a single custom Docker container published to the Docker Hub, to a scaled out service with load balancing. But there are challenges: it requires you to program in YAML, run (or script) many CLI commands, and manage your own Swarm or Kubernetes cluster. There is an easier way. By using Pulumi’s infrastructure as code, we can build a custom Docker image, publish it to a private AWS container registry, and spin up an AWS Fargate load balanced service running that container, all in 28 lines of TypeScript code and a single pulumi up command. The result leverages the best of what AWS has to offer, with the entire platform at your fingertips, with a single approach. In this article, we’ll see how.

Defining Everything in Code

The Docker Getting Started tutorial goes from a simple developer desktop scenario to a fully scaled out service in production. There is a lot of ceremony, however: YAML, many CLI commands, running your own Swarm cluster by hand, etc. This is complicated and doesn’t lend itself to repeatable deployments across many environments.

What if we could just write 28 lines of TypeScript instead, and accomplish the same end result, while leveraging the latest innovations in the AWS platform, natively?

In fact, we can. This article will show you how Pulumi’s infrastructure as code solution enables exactly this.

*Note: Everything we’re about to define will be in a single TypeScript file, index.ts. Other languages are available, including using plain Node.js JavaScript. Pulumi also supports other clouds, including Azure and GCP, however to fully exploit the power of the underlying AWS platform, these examples will leverage native capabilities like VPCs, ALBs, and Fargate.

To cut to the chase, here is the entire program. We’ll spend the rest of the post walking through it line by line:

import * as awsx from "@pulumi/awsx";

// Step 1: Create an ECS Fargate cluster.
const cluster = new awsx.ecs.Cluster("cluster");

// Step 2: Define the Networking for our service.
const alb = new awsx.lb.ApplicationLoadBalancer(
    "net-lb", { external: true, securityGroup: cluster.securityGroup });
const web = alb.createListener("web", { port: 80, external: true });

// Step 3: Build and publish a Docker image to a private ECR registry.
const img = awsx.ecs.Image.fromPath("app-img", "./app");

// Step 4: Create a Fargate service task that can scale out.
const appService = new awsx.ecs.FargateService("app-svc", {
    cluster,
    taskDefinitionArgs: {
        container: {
            image: img,
            cpu: 102 /*10% of 1024*/,
            memory: 50 /*MB*/,
            portMappings: [ web ],
        },
    },
    desiredCount: 5,
});

// Step 5: Export the Internet address for the service.
export const url = web.endpoint.hostname;

Step 1. Create a Cluster

The opening stanza imports Pulumi’s open source AWSX NPM package, @pulumi/awsx. It contains high level AWS best practices and patterns, and leverages real languages to eliminate boilerplate YAML templating:

import * as awsx from "@pulumi/awsx";

To provision an entire Elastic Container Service (ECS) cluster, AWS’s native container orchestration technology, we just new one up in a single line of code:

const cluster = new awsx.ecs.Cluster("cluster");

This will use the default Virtual Private Cloud (VPC) in our account. If we want to customize this aspect along with the auto-scaling policies, we can easily override the defaults by passing the appropriate arguments.

Step 2. Define the Networking

Next we need to think about the networking for our desired application. First, we will need a load balancer, as we’re going to create a scaled-out application and want a fixed address. Second, we will want to expose port 80 to the Internet, for incoming traffic. This leads to these two lines:

const alb = new awsx.lb.ApplicationLoadBalancer(
    "net-lb", { external: true, securityGroups: cluster.securityGroups });
const web = lb.createListener("web", { port: 80, external: true });

An Elastic Application Load Balancer (ALB) in AWS is a fully featured Layer 7 load balancer, with advanced features around SSL termination, content based routing, and HTTP/2. The AWSX package enables you to whip together simple routing when appropriate, while still having the ability to dig deeper into the advanced capabilities as you scale your workload to handle more sophisticated scenarios. Simple things simple, hard things possible!

Step 3. Build and Publish the Container Image

The next line declares a Docker image:

const img = awsx.ecs.Image.fromPath("app-img", "./app");

This single line is quite the work horse, and does a lot. Let’s unpack it a bit.

First and foremost, this allocates a private Elastic Container Registry (ECR) repository for your Docker image. This is AWS’s hosted Docker registry solution, privately encapsulated inside your AWS account.

But there is more: this also builds the Dockerfile underneath ./app and publishes the result to said registry, each time you run pulumi up! Any Dockerfile will work. This means you get a single command line to both provision and update your infrastructure, in addition to publishing and rolling out your images.

Step 4. Create a Scale-Out Service

Next up, let’s deploy our published image as a containerized service, behind our load balanced network on port 80, using our ECS Fargate cluster. Fargate is a managed container orchestrator that lets us skip the messy details of installing and managing Swarm on our own. This effectively replaces the docker-compose.yml from the Docker Getting Started tutorial, with a similarly simple sequence of code, and which gives us full access to the AWS platform:

const appService = new awsx.ecs.FargateService("app-svc", {
    cluster,
    taskDefinitionArgs: {
        container: {
            image: img,
            cpu: 128 /*~10%*/,
            memory: 50 /*MB*/,
            portMappings: [ web ],
        },
    },
    desiredCount: 5,
});

Notice that img just refers to the Docker image built in Step 3, and the portMappings structure references our load balancer’s listener defined in Step 2. Pulumi tracks all the dependencies so that it knows how to perform deployment activities in the right order. The task definition structure here exposes all of the underlying AWS ECS capabilities.

Step 5. Deploy the App and Kick the Tires!

On to the final step! Let’s deploy our infrastructure and application, and kick the tires.

The last line of code exposes the load balancer address so it’s easy to access:

export const url = web.endpoint.hostname;

The net result is that, with this project defined with the 28 lines of code above, we can simply run pulumi up; this command will show us a preview of the changes it will make and then, once confirmed, deploy everything to our AWS account, and hand back our service’s hostname:

After deploying everything, you’ve got a fully functioning application. To give it a try, let’s grab the URL exported by our program from the stack output command, and curl it:

$ curl http://$(pulumi stack output url)
<h3>Hello World!</h3>
<b>Hostname:</b> ip-172-31-39-18.ec2.internal<br/>
<b>Visits:</b> <i>cannot connect to Redis, counter disabled</i>

Or open it in our web browser:

$ open http://$(pulumi stack output url)

As an added bonus, the entire time we are doing these deployments, the Pulumi Service is keeping track of deployment state, to ensure everything works reliably and in a team setting, similar to how Git works with GitHub. For example, I can see a history of my deployments:

Congratulations!

In this article, you’ve seen how to take 28 lines of code, and turn them, plus a Dockerfile, into a fully functioning AWS microservice, including the infrastructure and application resources. No YAML required, no complex sequence of CLI commands, and no manual cluster installation.

Pulumi’s “everything is code” approach means deploying everything can be done with a CLI command, unlocking the power of the entire Docker platform with a great inner development loop, that works from the desktop all the way to production. This entire flow can be easily integrated into your favorite CI/CD pipeline, including GitOps workflows.

If we want to augment our service with other AWS resources – like S3 buckets, Aurora or RDS databases, Lambda functions, CloudWatch dashboards, etc – they can be provisioned and managed the same was created our clusters and load balancers: just new them up!

Remember that all of this not only works for native AWS ECS containers, but also Azure and GCP cloud providers, in addition to Kubernetes!

Want to go deeper?

Program the Cloud with 12 Pulumi Pearls

Joe Duffy — Wed, 25 Jul 2018 00:00:00 +0000

In this post, we’ll look at 12 “pearls” – bite-sized code snippets – that demonstrate some fun ways you can program the cloud using Pulumi. In my introductory post, I mentioned a few of my “favorite things”. Now let’s dive into a few specifics, from multi-cloud to cloud-specific, spanning containers, serverless, and infrastructure, and generally highlighting why using familiar languages is so empowering for cloud scenarios. Since Pulumi lets you do infrastructure-as-code from the lowest-level to the highest, we will cover a lot of interesting ground in short order.

If you want to follow along and try some of this out, Pulumi is open source on GitHub, free to download and use, and the quickstart will acquaint you with the CLI. Most of the examples are directly runnable and available in our examples repo, and are just a pulumi up away, unlike other approaches that require you to point-and-click around in your cloud’s console, and/or author reams of yucky YAML. And you get to use familiar languages!

Here is an index of the pearls in case you want to dive straight into one in particular:

Infrastructure:

Declare cloud infra using a familiar language (with loops!)
Make a reusable component out of your cloud infra

Serverless:

Go serverless without the YAML
Capture state in your serverless funcs, like real lambdas
Simple serverless cron jobs
Run Express-like serverless SPAs and REST APIs at near zero cost

Containers:

Deploy production containers without the fuss
Use containers without Dockerfiles
Invoke a long-running container as a task

General Tips and Tricks:

Use code to avoid hard-coding config
Use config to enable multi-instantiation and code reuse
Give your components runtime APIs

Even if you’re uninterested in low-level infrastructure, it can be fun to work through these examples; it’s “turtles all the way down” with Pulumi and doing so can help understand how the system works. And similarly, it can be fun to see the high-level scenarios these building blocks facilitate, even if you just want to stand up containers and functions.

And with that, let’s dive in.

Infrastructure

Familiar Languages

[ Runnable Example ]

Pulumi gives you a way to express infrastructure configuration using your favorite programming language. In this article, we’ll use TypeScript on Node.js, as it delivers great productivity by blending dynamic and static typing with the NPM ecosystem of reusable packages.

At the lowest layer, there are packages for all major cloud providers – AWS, Azure, Google Cloud – in addition to Kubernetes and OpenStack. So, if you want an EC2 instance, Azure CosmosDB, or Kubernetes Pod, you just new up an object. From there, Pulumi uses an infrastructure-as-code approach similar to using AWS CloudFormation, Azure Resource Manager, Google Resource Manager, Kubernetes or Helm YAML, or Terraform – just without the YAML DSLs.

The strength of using a language truly begins to shine when you go beyond new, however. Using a language gives you control structures, like for loops and if branches. This is easy to take for granted, but compared to existing YAML DSLs, it is very powerful.

For example, let’s create three EC2 VMs and export their IP addresses. Normally you’d need to copy-and-paste, however for loops to the rescue!

index.ts:

import * as aws from "@pulumi/aws";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

let webServers = [];
for (let i = 0; i < 3; i++) {
 webServers.push(new aws.ec2.Instance(`web-server-${i}`, {
 instanceType: "t2.micro",
 ami: "ami-6869aa05", // us-east-1-only
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello, World #${i}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 }));
}
export const publicHostnames = webServers.map(s => s.publicDns);

This provisions three web servers, each of which runs a simple Python webserver.

Of course, this can unlock far more sophisticated scenarios. For example, our own AWS Infrastructure package offers core AWS infrastructure components using AWS’s own best practices, and a community member built an AWS Virtual Private Cloud (VPC) component that automatically distributes subnets across all availability zones in a region, including smartly calculating the CIDR addresses, hiding lots of the messy details of AWS networking.

Reusable Components

[ Runnable Example ]

Speaking of components, this is another benefit of using familiar languages: abstraction. This includes the ability to hide uninteresting details through encapsulation, and create bigger things out of smaller things, a key to how we’ve become so productive with programming languages generally over the years. Applied to infrastructure, this enables some powerful scenarios.

For instance, perhaps you want a standard base class for all servers in your organization, enforcing security policy or even just naming conventions. Or maybe you want to ensure there is an Envoy proxy sidecar for all of your Kubernetes pods. No matter the scenario, we call these things “components” and Pulumi has a rich model that supports parents and children.

To stick with the running example, let’s make a WebServer class that internally creates the EC2 VM so that callers don’t need to know messy details like how to create and configure a SecurityGroup and choosing the right image:

web.ts:

import * as aws from "@pulumi/aws";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

export class WebServer {
 public readonly vm: aws.ec2.Instance;

 constructor(name: string) {
 this.vm = new aws.ec2.Instance(`${name}`, {
 instanceType: "t2.micro",
 ami: "ami-6869aa05", // us-east-1-only
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello from ${name}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 });
 }
}

Now we can go back to our original program and tidy it up a bit:

index.ts:

import { WebServer } from "./web";

let webServers = [];
for (let i = 0; i < 3; i++) {
 webServers.push(new WebServer(`web-server-${i}`));
}
export const publicHostnames = webServers.map(s => s.vm.publicDns);

Not only is the result simpler, but we now have a way to divvy up responsibilities within a team, helping Dev and DevOps to work better together.

This example used purely local components, but you can of course publish and consume any old packages from your favorite package manager – including private registries for your own team.

We will return to these examples later on and improve them to be more reusable through configuration, however let’s first tour some fun serverless and container pearls.

Serverless

No YAML

[ Runnable Example ]

Let’s shift gears and jump into serverless. Pulumi is unique in that its goal is to be a platform for all aspects of modern cloud programming, so that includes serverless, in addition to the abovementioned infrastructure examples and the soon-to-be-mentioned container ones. Most real-world examples we see in fact mix all of these things together in the same program.

Because of the way it uses familiar languages, there are great benefits to serverless programming in Pulumi. No YAML required – just write lambdas in your favorite language, and our compiler/runtime magic will deal with them. But first, let’s start simple.

The AWS Serverless package provides event sources for all of the possible AWS Lambda triggers. This leverages the aws.serverless.Function abstraction that takes a lambda in your language and translates it into a package suitable for uploading to AWS. No need to embed code in YAML files, manually upload to S3, etc. Pulumi handles all of this including versioning.

Let’s look at a simple example that simply posts a Slack message for every new message that arrives in an SQS queue (a new capability recently added to AWS). The way you express serverless functions in Pulumi is by writing ordinary lambdas and the way you hook them up to event sources is by using an event-driven style on your existing cloud resources:

index.ts:

import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

let queue = new aws.sqs.Queue("mySlackQueue");
serverless.queue.subscribe("mySlackPoster", queue, async (e) => {

let slack = require("@slack/client");
let client = new slack.WebClient(config.slackToken);
 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*:n${rec.body}n`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to ${config.slackChannel}`);
 }
});
export const queueURL = queue.id;

The code and instructions for running this example are available in our examples repo.

Serverless programs written in this style feel just like your favorite event-driven frameworks, and – because these are just ordinary lambdas in your language of choice – they compose in all the usual ways, unlocking some fascinating future possibilities (like serverless Spark and/or RxJS). Behind the scenes, Pulumi is doing the hard work to extract out the body of the function, rewrite it minimally but as needed to wire it up to AWS Lambda, and package it up. Subsequent deployments are as easy as the first one, as Pulumi does diffing to compute the minimal deltas.

Not everybody will want to mix infrastructure definitions and runtime code like this. Because these are just ordinary programs, you are free to split code into submodules and structure your workspace as you see fit. In fact, you can create distinct stacks for the different pieces.

This example is AWS-specific, however the Pulumi Cloud Framework offers several multi-cloud serverless (and container) capabilities. We will see some examples of that package in action below, however being able to target your cloud of choice directly means you can use all of its underlying features.

Real Lambdas

[ Runnable Example ]

Pulumi also lets you capture state in your serverless functions. This is one of the major advantages to this approach; normally you’d need to awkwardly arrange for such state to be passed into your function through some sideband mechanism, such as environment variables.

In Pulumi, we just capture state. Just like in your favorite language.

Let’s say we want the Slack information above to be configurable – a good idea, since the Slack token is a secret that should be configured using the –secret flag. To do so, first let’s create a config module:

config.ts:

import * as pulumi from "@pulumi/pulumi";
let config = new pulumi.Config(pulumi.getProject());

export const slackChannel = config.get("slackChannel") || "#general",
export const slackToken: config.require("slackToken");

And now let’s go back to our above program, and use the config module for these properties. Notice that we simply capture a reference to the config module – that’s it!

index.ts:

import * as config from "./config";
import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

let queue = new aws.sqs.Queue("mySlackQueue");
serverless.queue.subscribe("mySlackPoster", queue, async (e) => {
 let slack = require("@slack/client");
 let client = new slack.WebClient(config.slackToken);

 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*:n${rec.body}n`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to ${config.slackChannel}`);
 }
});
export const queueURL = queue.id;

We often call this “capture by serialization” because, of course, the captured state is marshaled into the serverless function’s package, and then unmarshaled at runtime at the time of use. The semantics of capture are specific to each language, but for JavaScript, we do a deep serialization of the object, including its prototype chain, so that the full structure is preserved.

Because of this, functions are even available. We use this capability to let you capture resources by-value that internally have references to a live cloud resource. We will see that next in our serverless cron job example.

Cron Jobs

[ Runnable Example ]

Pulumi’s Cloud Framework has an ultra-straightforward timer module that lets you schedule cron jobs that run serverless functions. This is the easiest way to get up and running with serverless functions, because you don’t even need any other resources to trigger events from.

There are several ways to schedule a timer, depending on your stylistic preferences. These examples simply print the current time to the console on a given interval:

index.ts:

import * as cloud from "@pulumi/cloud";

// Run a timer every minute:
cloud.timer.interval("interval-timer", { minutes: 0 }, () => {
 console.log(`interval-timer: ${Date.now()}`);
});

// Run a timer every minute (cron-style expression):
cloud.timer.cron("cron-timer", "0 * * * * *", () => {
 console.log(`cron-timer: ${Date.now()}`);
});

// Run a timer every day at 7:30 UTC:
cloud.timer.daily("daily-timer", { hourUTC: 7, minuteUTC: 30 }, () => {
 console.log(`daily-timer: ${Date.now()}`);
});

// Run a timer at the 45th minute UTC of every hour:
cloud.timer.hourly("hourly-timer", { minuteUTC: 45 }, () => {
 console.log(`hourly-timer: ${Date.now()}`);
});

To make things slightly more interesting, let’s write a serverless timer that fetches the Hacker News homepage every hour and stashes it into a document database:

index.ts:

import * as cloud from "@pulumi/cloud";

let snapshots = new cloud.Table("snapshots");
cloud.timer.daily("daily-yc-snapshot", { hourUTC: 0, minuteUTC: 0 }, () => {
 let req = require("https").get("https://news.ycombinator.com", (res) => {
 let content = "";

 res.setEncoding("utf8");
 res.on("data", (chunk) => { content += chunk });
 res.on("end", () => {
 snapshots.insert({ date: Date.now(), content: content });
 });
 });
 req.end();
});

We are using the ability to capture a reference to our cloud.Table and call functions on it, like insert. Of course, the body of that function can do absolutely anything that you’d like.

REST APIs

[ Runnable Example ]

Us Node.js programmers love web frameworks, whether that be serving static content using a single page application (SPA) built using React.js, Vue.js, or Angular.js, a program built using dynamic server-side logic using Express.js, or some combination thereof. Serverless offers a way to run these programs at incredible density and low cost, often reducing a VM- or container-based solution to 1/10th its cost (or better). The problem is, this usually means abandoning productive frameworks, and resorting to YAML, hand-authoring Swagger files, and configuring complex API Gateway machinery – all foreign concepts when you just want to run a website.

Pulumi lets you just write code to host both static and dynamic content simply, using its cross-cloud framework’s cloud.API abstraction. It has been designed to have a familiar Node.js feel. The following example serves all static content underneath my www directory at the root, and provides a simple GET endpoint at /hello that returns a { hello: "World!" } JSON object:

index.ts:

import * as cloud from "@pulumi/cloud";
// Create a new API endpoint:
let app = new cloud.API("my-app");
// Serve static files from the `www` folder (using AWS S3):
app.static("/", "www");
// Serve a simple REST API on `GET /hello` (using AWS Lambda):
app.get("/hello", (req, res) => res.json({ hello: "World!" }));
// Export the public URL for the HTTP service:
export const url = app.publish().url;

All of the usual verbs are available – post, patch, etc. – as is support for middleware.

The structure of my local filesystem will demonstrate how some of this ties together:

.
├── Pulumi.yaml
├── index.ts
├── package.json
└── www
├── favicon.png
└── index.html
1 directory, 5 files

Again, all we need to do is pulumi up, and Pulumi will figure out what has changed. So, if we are building an SPA using React.js, etc., then we would simply have a build step that updates the content, and the next time we run pulumi up, it will figure out what has changed and make the minimal set of updates. Similarly, if we edit a dynamic function, it will update just the function.

As you can imagine, things get fun when you start mixing scenarios. Say we wanted a web frontend to our Hacker News timer fetcher above. Thanks to the way we can simply capture a reference to the snapshots database and use it in the body of our API, this is extremely trivial.

Per the earlier conversation around mixing infrastructure and application code, some folks prefer to split the routes from the application code. This is a fine convention, and easy to do:

index.ts:

import * as cloud from "@pulumi/cloud";
import * as routes from "./app/routes";

// Create an API like before, but use a routes module instead of defining inline:
let app = new cloud.API("my-app");
app.get("/hello", routes.hello);

export const url = app.publish().url;

For now, Pulumi only offers an AWS implementation of these APIs, but more clouds are on their way!

Containers

Easy Deployment

[ Runnable Example ]

Let’s shift gears again and take a look at containers. Pulumi supports programming against raw container orchestrators – Amazon’s ECS, Azure’s ACS, Google Cloud’s GKE, Kubernetes directly, and so on – using the cloud resources directly. In this mode, Pulumi is entirely unopinionated about how containers are built, published, and deployed to your orchestrator.

It is important that Pulumi supports that low-level approach, and is what distinguishes it compared to PaaS-like offerings. This entails significant spelunking at a level that is only loosely connected to the task of standing up a container, however, which can be time-consuming and frustrating for many developers. To make this common case easier, Pulumi also offers higher-level abstractions to make these common cases easier, similar in spirit to Docker Compose, except that you can mix these abstractions with all your other cloud resources.

For example, to create a load balanced Nginx container using an image from the Docker Hub:

index.ts:

import * as cloud from "@pulumi/cloud";

let nginx = new cloud.Service("nginx", {
 image: "nginx",
 ports: [{ port: 80 }],
 replicas: 2,
});

export const url = nginx.defaultEndpoint;

This is all you need to run Nginx with 2 load balanced replicas listening on port 80. By default, this will run in Fargate when targeting AWS, which means you can skip the complications of configuring an orchestrator. After running pulumi up, the auto-assigned URL will be printed.

The image directive supports any normal Docker image URL, so private registries work just as well as the default public Docker Hub. And there are a number of other supported properties for memory and CPU constraints, advanced port mapping, and volumes.

Now things get a bit more interesting. Let’s say we wanted to build our own custom container instead. Normally that would mean defining a Dockerfile, provisioning a private registry, and manually building and publishing your image to the registry, and then consuming that container image from the registry. That’s a lot of boilerplate scripting!

Pulumi unifies all container building, publishing, and consumption underneath a simple pulumi up:

index.ts:

import * as cloud from "@pulumi/cloud";

let nginx = new cloud.Service("nginx", {
 build: ".",
 ports: [{ port: 80 }],
 replicas: 2,
});

export const url = nginx.defaultEndpoint;

Dockerfile:

FROM nginx
COPY ./www /usr/share/nginx/html

This is trivial Dockerfile that derives from the nginx base image and copies the ./www directory into the Nginx HTML target so that it will be served up. This is our directory structure:

.
├── Dockerfile
├── Pulumi.yaml
├── index.ts
├── package.json
└── www
├── favicon.png
└── index.html
1 directory, 5 files

Of course, this Dockerfile could be arbitrarily complex, using any Docker image.

No Dockerfiles

[ Runnable Example ]

The above example showed you how to create services that use Docker containers, either via a pre-built image stored in a registry like the Docker Hub, or by building one from a Dockerfile. The cloud.Service class supports a third possibility, which is really fun, as it takes advantage of Pulumi’s serverless programming model earlier to define a container and its service.

Imagine we want to define a simple Node.js webserver rather than using Nginx like this. Perhaps it uses Express.js. For this blog post, however, let’s stick with the most basic http-module-based webserver:

index.ts:

import * as cloud from "@pulumi/cloud";

let www = new cloud.Service("www", {
 function: () => {
 let rand = Math.random();
 require("http").createServer((req, res) => {
 res.end(`Hello, world! (from ${rand})`);
 }).listen(80);
 },
 ports: [{ port: 80 }]
 memory: 128,
 replicas: 2,
});

export const url = www.defaultEndpoint;

This code defines a load balanced service, backed by a container that Pulumi builds on your behalf out of the body of the function provided in the program, all done with a simple pulumi up gesture. This function, of course, can close over other state just like the earlier serverless examples, including cloud resources, making it as easy as can be to connect your microservices.

Long Running Containers as Tasks

[ Runnable Example ]

The above examples are great if what you want is a load balanced, long-running service. Sometimes a container is useful simply for executing a long-running task, or one that has very specific requirements about its execution environment (such as a specific base image).

For those cases, a cloud.Task is just what the doctor ordered. Task supports all the same construction techniques as Service, so you can give it a pre-built image, a build directive, or an inline function. For purposes of illustration, let’s build a custom Dockerfile that performs an FFMPEG transformation, to extract a thumbnail anytime someone adds a video to a bucket:

index.ts:

let cloud = require("@pulumi/cloud-aws");
// A bucket to store videos and thumbnails.
let videos = new cloud.Bucket("bucket");
// A task which runs an FFMPEG transform to extract a thumbnail image.
let ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: ".",
 memoryReservation: 512,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
videos.onPut(
 "onNewVideo",
 args => {
 let file = args.key;
 ffmpegThumbnailTask.run(
 {
 environment: {
 "S3_BUCKET": videos.bucket.name.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':'),
 "OUTPUT_FILE": file.substring(0, file.indexOf('_')) + '.jpg',
 },
 });
 },
 { keySuffix: ".mp4" });

exports.bucketName = videos.bucket.name;

Dockerfile:

FROM jrottenberg/ffmpeg
RUN apt-get update && apt-get install python-dev python-pip -y && apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task... " &&
echo "Copying video from s3://${S3_BUCKET}/${INPUT_VIDEO} to ${INPUT_VIDEO}..." &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3://${S3_BUCKET}/${OUTPUT_FILE} ..." &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Notice that, just like before, this is backed by an ordinary Dockerfile. This example actually showcases serverless and containers living alongside one another in harmony. Notice how we kick off the task – simply by capturing a reference to the Task from the serverless event handler, and invoking its run method. This leverages Pulumi’s ability to give resources like Tasks custom APIs that can be invoked at runtime, something we will return to again later on.

General Tips and Tricks

Avoid Hard-Coding Config

Let’s return to our webserver example from earlier. Unfortunately, this code will only work in one region, because we hard-coded the "ami-6869aa05" image name (AMI). (Image names in AWS are region-specific.) In general, we prefer to build Pulumi programs that can be multi-instantiated across different regions. Pulumi’s stacks model streamlines this workflow, making it easy to spin up fresh copies of your entire application and/or infrastructure when you need to scale, while also reducing friction during development and testing. Such hard-coding is a definite code smell and impediment, however.

Because we are using a familiar language, we can do anything that general-purpose languages can do. That includes dynamically querying sources of information which, in this case, can be used to avoid hard-coding. We can even use the AWS SDK, which gives us full access to all AWS functionality:

ami.ts:

import * as aws from "@pulumi/aws";

export function getLinuxAmi(): Promise<string> {
 return new Promise<string>((resolve, reject) => {
 let ec2 = new (require("aws-sdk")).EC2({ region: aws.config.requireRegion() });
 ec2.describeImages(
 {
 Owners: [ "amazon" ],
 Filters: [{
 Name: "name",
 Values: [ "amzn-ami-hvm-????.??.?.x86_64-gp2" ],
 }],
 },
 (err: any, data: any) => {
 if (err) {
 reject(err);
 } else {
 let newestImage: string | undefined;
 let newestImageDate: number | undefined;
 for (let image of data.Images) {
 let imageDate = Date.parse(image.CreationDate);
 if (!newestImage || !newestImageDate ||
 imageDate > newestImageDate) {
 newestImage = image.ImageId;
 newestImageDate = imageDate;
 }
 }
 if (newestImage) {
 console.log(`AMI: ${newestImage} (${newestImageDate})`);
 resolve(newestImage);
 } else {
 reject("No Linux AMI found for the current region");
 }
 }
 },
 );
 });
}

I’ll be the first to admit that this is code that only a mother could love. Even better would be to wrap it up in a reusable NPM package that others can use without the boilerplate! Either way, it’s very powerful to have the full capabilities of a language at our fingertips, as this sort of thing simply isn’t possible in other infra-as-code solutions.

Now that we have this function, we can revisit the way we set the AMI property on our VMs:

web.ts:

import * as aws from "@pulumi/aws";
import { getLinuxAmi } from "./ami";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

export class WebServer {
 public readonly vm: aws.ec2.Instance;

 constructor(name: string) {
 this.vm = new aws.ec2.Instance(`${name}`, {
 instanceType: "t2.micro",
 ami: getLinuxAmi(),
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello from ${name}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 });
 }
}

Given this change, we can safely deploy our WebServer to any AWS region.

Multi-Instantiation and Code Reuse

In the webserver example, we used the aws.config.requireRegion() function to obtain the configured AWS region. Using configuration in this manner lets us multi-instantiate stacks; in other words, to easily stand up multiple copies of a service, application, or infrastructure.

We can even make the quantity configurable using Pulumi’s config system:

index.ts:

import { WebServer } from "./web";
import { Config, getProject } from "@pulumi/pulumi";

let config = new Config(getProject());
let webServers = [];
let webServersCount = config.getNumber("count") || 3;
for (let i = 0; i < webServersCount; i++) {
 webServers.push(new WebServer(`web-server-${i}`));
}

export const publicHostnames = webServers.map(s => s.vm.publicDns);

Given this specific program, we can easily change the number of servers created from the default of three, to something larger:

$ pulumi config set count 7

There are plenty of opportunities for configuration like this, like the machine size. By making such settings configurable, different environments can easily have different settings, while still sharing the same code beneath them, eliminating copy-and-paste and its associated pitfalls.

Give Components Runtime APIs

[ Runnable Example ]

As our final example, let’s see how to give your own components APIs that can be invoked at runtime. We’ve already seen this a few times now, such as managing data in a cloud.Table that has been captured in a lambda, and cloud.Task’s run method. Pulumi is built to be extensible, so you can define your own abstractions just like these with APIs too.

For this example, let’s take a simple Redis cache service and turn it into a component. After doing so, we can then add simple get/set APIs on it that let you interact with the cache at runtime in simple ways. By providing this sort of abstraction, we open up interesting possibilities. For instance, our default implementation can use a custom container-backed service, via cloud.Service, but opt to use a simpler hosted solution when running in a public cloud – AWS ElastiCache, Azure Redis Cache, or Memorystore in Google Cloud.

To make our component, we will simply allocate a cloud.Service in the constructor, taking an optional memory parameter to control its memory reservation. We are using configuration to read the Redis password using Pulumi’s built-in secrets mechanism. To give that component APIs, we just define methods on the class that access Redis through its own NPM package, and that’s it! The way lambda capture works will ensure that these functions are accessible:

cache.ts:

import * as cloud from "@pulumi/cloud";
import * as config from "./config";

// A simple cache abstraction that wraps Redis.
export class Cache {
 public readonly get: (key: string) => Promise<string>;
 public readonly set: (key: string, value: string) => Promise<void>;
 private readonly redis: cloud.Service;

 constructor(name: string, memory: number = 128) {
 this.redis = new cloud.Service(name, {
 image: "redis:alpine",
 memory: memory,
 ports: [{ port: 6379 }],
 command: ["redis-server", "--requirepass", config.redisPassword],
 });
 }

 public async get(key: string): Promise<string> {
 let client = await this.createRedisClient();
 return new Promise<string>((resolve, reject) => {
 client.get(key, (err: any, v: any) => {
 if (err) reject(err);
 else resolve(v);
 });
 });
 }

 public async set(key: string, value: string): Promise<void> {
 let client = await this.createRedisClient();
 return new Promise<void>((resolve, reject) => {
 client.set(key, value, (err: any, v: any) => {
 if (err) reject(err);
 else resolve();
 });
 });
 }

 private async createRedisClient(): Promise<any> {
 let ep = (await this.redis.defaultEndpoint).get();
 return require("redis").createClient(
 ep.port, ep.hostname, { password: config.redisPassword });
 }
 }
}

Because this is just a class, we have all the standard programming facilities available to us. If we wanted to specialize it to use our cloud’s hosted Redis services, we could do that using the standard techniques, including conditional code or, slightly more elegantly, using concrete subclasses for the different options.

Now that we’ve defined our cache, we can create an instance, and then use it from within a callback, such as a serverless API implementation. This is a fairly complex application that acts as a URL shortener, with both GET and POST endpoints, but really shows off the power of combining many of the techniques we’ve seen throughout this post:

index.ts:

import * as cloud from "@pulumi/cloud";
import * as cache from "./cache";

// Create a URL shortener app and a table to hold shortened URLs.
let app = new cloud.API("shortener");
let urlTable = new cloud.Table("urls", "name");

// Use our cache component to cache frequently accessed URLs.
let urlCache = new cache.Cache("urlcache");

// GET /url lists all URLs currently registered.
endpoint.get("/url", async (req, res) => {
 try {
 res.status(200).json(await urlTable.scan());
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

// GET /url/{name} redirects to the target URL based on a short-name.
endpoint.get("/url/{name}", async (req, res) => {
 let name = req.params["name"];
 try {
 // First try the Redis cache. If we miss, consult the table.
 let url = await urlCache.get(name);
 if (url) {
 res.setHeader("X-Powered-By", "redis");
 } else {
 let value = await urlTable.get({ name });
 if (value && value.url) {
 url = value.url;
 urlCache.set(name, url); // cache for next time.
 }
 }

 // If we found an entry, 301 redirect to it; else, 404.
 if (url) {
 res.setHeader("Location", url);
 res.status(302);
 res.end("");
 } else {
 res.status(404);
 res.end("");
 }
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

// POST /url registers a new URL with a given short-name.
endpoint.post("/url", async (req, res) => {
 let url = req.query["url"];
 let name = req.query["name"];

 try {
 // Insert into the table and our URL cache.
 await urlTable.insert({ name, url });
 await urlCache.set(name, url);
 res.json({ shortenedURLName: name });
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

export const url = endpoint.publish().url;

Notice that we can capture and use our Cache component just like we can the cloud.Table!

We’ve only scratched the surface of what’s possible with components and custom APIs, however hopefully you’re starting to get a sense of why this is such a powerful capability.

Winding Down

In this post, we’ve seen how Pulumi can define infrastructure, containers, and serverless cloud software, often combining many of the techniques together to build powerful distributed applications. This includes programming to specific resources that your cloud of choice provides, in addition to writing higher level cloud programs that are able to target many clouds (expect more multi-cloud progress here in the weeks to come – including Kubernetes). Each pearl is a topic unto itself; we’d love your feedback on what topics to dig into next.

Happy hacking – and have fun programming the cloud!

How we use Pulumi to build Pulumi

Luke Hoban — Tue, 26 Jun 2018 00:00:00 +0000

Here at Pulumi we are (perhaps unsurprisingly!) huge fans of using Pulumi to manage our cloud infrastructure and services. We author our infrastructure in strongly-typed programming languages, which allows us to to benefit from rich tooling - documenting and factoring our infrastructure using the same software engineering practices we apply to our application code. This also allows us to create reusable abstractions which accelerate our ability to deliver new features and services, and our ability to standardize and refactor infrastructure patterns across our services with relative ease.

Like other users, we use Pulumi at a variety of levels of abstraction. We use Pulumi for raw infrastructure provisioning, defining the core networking layer for our AWS-based backend infrastructure. And we use Pulumi to define how our application services are deployed into ECS using just a few lines of code. Pulumi hosts and manages static content for www.pulumi.com and get.pulumi.com. We use Pulumi to define the CloudWatch dashboards connected to our infrastructure. And for monitoring, Pulumi defines metrics and notifications/alarms in PagerDuty and Slack.

Best of all, we’ve been able to take things we’ve learned from these use cases, and others we’ve worked with beta users on over the last few months (thank you!), and factor common patterns out into reusable libraries like @pulumi/aws-infra and @pulumi/cloud for ourselves and others to build upon.

In this post, we’ll do a deeper dive into each of these use cases, highlighting unique aspects of how we use Pulumi itself, and some of our engineering processes around how we integrate Pulumi into the rest of our toolchain.

Infrastructure

Our backend infrastructure that supports app.pulumi.com and the Pulumi CLI is fairly typical of a cloud service/application building on AWS. Those who have done this before will be familiar with the combination of AWS VPC, EC2, ASG, ALB, ECS, S3, Aurora, CloudFront and Route53 that powers the core of many AWS-based web applications.

We started off building this infrastructure last year - before we even had completed support for managing AWS resources in Pulumi! As a result, we brought this up initially using CloudFormation. Moving this to Pulumi was an amazing experience. We immediately brought more engineering rigour to the codebase, factoring and decomposing functionality in clearer ways, extracting out useful patterns into named functions, using loops and conditionals instead of copy/paste, and getting completion lists and on-the-fly error checking from within our development tools.

Equally important, authoring our infrastructure in Pulumi gave us the ability to factor out reusable abstractions. We found that our own VPC definition was something that many early customers we worked with also needed - so we factored it out into new awsinfra.Network. We found that standing up an ECS cluster was another big pile of AWS resources that are commonly needed, but can be configured with a much smaller space of configuration values in the common case, so we factored out new awsinfra.Cluster.

Defining infrastructure in Pulumi made it easy to stand up many instances of our service, including per-developer stacks, with just pulumi stack init my-developer-stack and pulumi update. The ability to build reusable components which can be multi-instantiated into the same target cloud is critical here, and something that Pulumi takes care of by default. Within minutes, a developer can have an isolated copy of our full infrastructure stood up, which can be used to test out and quickly iterate on changes. This includes not just the basic infrastructure, but also provisioning of DNS entries in Route53 and creation of custom CloudWatch dashboards (see more on this below).

We have three shared environments - development, staging and production - each an isolated environment with separate configuration. For these shared environments we drive deployments of the Pulumi programs that define the infrastructure above via our CI/CD pipeline. Pulumi can be integrated into any existing CI/CD pipeline or toolchain, via direct use of pulumi preview and pulumi update as part of build and deployment stages. This allows us to reliably deliver a known desired state for each target environment, and to promote changes between environments with just a git merge.

API and App Services

Most of our backend application services are developed in Go, and we deploy them onto our production infrastructure as container images. However, instead of building separate CI pipelines for our container images and the infrastructure they need to function correctly (ECS Service, Application Load Balancer, ECR Repository, etc.), we drive the deployment of both infrastructure and application code via a single Pulumi program. This allows our application code to version along with its supporting infrastructure - when we need to expose a new port for example, we can make the changes to both application code and load balancer as part of a single commit, and promote it through CI as an atomic action. We have found this to significantly simplify the delivery of containerized services on AWS, as well as making it easy to stand up new services into our existing shared networking and cluster infrastructure.

Again, we found this to be a common pattern as we worked with other early customers, and factored out a library to simplify deployment of containerized services into a production deployment on AWS as part of the cloud.Service API in @pulumi/cloud. Here’s an example of using an application service defined in the ./airflow-container folder (with a Dockerfile) into an ECS cluster. This one declaration will provision a scaled-out ECS Service for running the container images, a ECR repository to store the image, optionally a load balancer if ports were exposed, and each pulumi update will build the container image, push it to the private repository, and turn-over each service task to use the new image. All of that workflow complexity is abstracted away behind a simple API and a single CLI command.

Metrics, Alarms and Dashboards

We also use Pulumi to define a few less traditional parts of our infrastructure - including our metrics, dashboards and alarming infrastructure on top of AWS CloudWatch.

Defining and versioning this along with our infrastructure has significant benefits: new features are built with metrics, dashboards and alarms lit up from the get-go, developers get a full set of dashboards as part of their personal isolated development environments, and promotions from development to staging to production can verify not just physical infrastructure, but also that alarms and dashboards are behaving as expected prior to pushing code to production. Here’s a sample of our CloudWatch dashboards:

We build up AWS resources representing CloudWatch dashboards using a compositional framework shown below. As this example shows, bringing software engineering approaches to infrastructure as code can lead to new approaches to how code is authored and factored - and ultimately to the creation of higher level frameworks for building domain-specific cloud infrastructure - in this case, CloudWatch dashboards.

The same metrics that power these dashboards are also used to power notifications and alarms. In these cases, we want to define both the infrastructure for the alarm - the SNS topics and registration of those topics with CloudWatch - as well as the small piece of code that fires off Pagerduty and Slack notifications on notifications being pushed to that SNS topic - all in one codebase. This ensures that we can version these together, and that we can deploy both consistently as part of updates. To do this, we include the event handlers for these infrastructure events directly inline in our Pulumi programs. Note in the code snippet below how we define the infrastructure for a Topic, and then conditionally wire up event handlers on that topic, including the code to execute at runtime on these events directly in our Pulumi program. This code can even reach out and reference configuration used to provision our Pulumi deployment - key to enabling this to be deployed seamlessly into a variety of different development environments.

Static Content

We serve three properties as static content from S3, each described using Pulumi. One of these is https://get.pulumi.com which hosts downloads of our pulumi CLI installer. It’s a very simple program, and even better - it’s open source on GitHub. It combines AWS S3, CloudFront and Route53.

This infrastructure is deployed and managed using the Pulumi Service at app.pulumi.com, allowing us to track the history of deployments and changes to the infrastructure and quickly jump in to the resources defined by this stack in the AWS console.

Conclusion

Treating cloud infrastructure as software has changed how we think about infrastructure engineering. Using Pulumi for both traditional cloud infrastructure as well as to get application code and supporting service infrastructure into production provides a simple workflow and the ability to create simple high-level APIs to make new services easy to define and deploy.

Try out Pulumi yourself today!

Build a Video Thumbnailer on AWS

Donna Malayeri — Thu, 21 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build cloud applications that use a combination of containers, lambdas, and connected data services and infrastructure: Colada apps.

An example of a Colada app is extracting a thumbnail from a video. A serverless function can only run for 5 minutes, so we’ll run a container in AWS Fargate to do the video processing.

In this app, a Lambda function is triggered whenever a new video is uploaded to S3. This function launches a task in Fargate that uses FFmpeg to extract a video thumbnail. A second Lambda function is triggered when a new thumbnail has been created.

This post was inspired by a blog post from Serverless, Inc that shows how to combine AWS Fargate and Lambda for a long-running process. Let’s see how a similar app would be implemented in Pulumi entirely in code.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the App

We’ll create a Pulumi project, define the infrastructure and app code in JavaScript, and create a Dockerfile for the Fargate task.

First, to create a new Pulumi project, run the following commands:

mkdir hello-colada && cd hello-colada
pulumi new aws-javascript

This creates a new project in the hello-colada directory.

Second, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");

// A bucket to store videos and thumbnails.
const bucket = new cloud.Bucket("bucket");
const bucketName = bucket.bucket.id;

// A task which runs a containerized FFMPEG job to extract a thumbnail image.
const ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: "./", // folder containing the Dockerfile
 memoryReservation: 128,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
// Use the time index specified in the filename (e.g. cat_00-01.mp4 uses timestamp 00:01)
bucket.onPut("onNewVideo", async (bucketArgs) => {
 console.log(`*** New video: file ${bucketArgs.key} was uploaded at ${bucketArgs.eventTime}.`);
 const file = bucketArgs.key;

 const thumbnailFile = file.substring(0, file.indexOf('_')) + '.jpg';
 const framePos = file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':');

 await ffmpegThumbnailTask.run({
 environment: {
 "S3_BUCKET": bucketName.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": framePos,
 "OUTPUT_FILE": thumbnailFile,
 },
 });
 console.log(`Running thumbnailer task.`);
}, { keySuffix: ".mp4" });

// When a new thumbnail is created, log a message.
bucket.onPut("onNewThumbnail", async (bucketArgs) => {
 console.log(`*** New thumbnail: file ${bucketArgs.key} was saved at ${bucketArgs.eventTime}.`);
}, { keySuffix: ".jpg" });

// Export the bucket name.
exports.bucketName = bucketName;

This code uses cloud.Task, a high-level, convenient component for working with containers. The component automatically provisions a container registry instance in ECR, runs a Docker build, and saves the Docker image to the provisioned ECR instance. It also defines an ECS task and configures it to use the built image. All this in just 4 lines of code!

Next, in the same directory, create a Dockerfile with the following contents. We’ll use an existing FFmpeg container and install the AWS CLI. When the container is started, it copies the video file from S3, runs ffmpeg, and copies the output back to S3.

 FROM jrottenberg/ffmpeg
RUN apt-get update &&
apt-get install python-dev python-pip -y &&
apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task..." &&
echo "Copying video from S3" &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3" &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Install the @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud-aws @pulumi/cloud

Finally, configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

$ pulumi config set cloud-aws:useFargate true

That’s it! Now, let’s deploy the app.

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation. During the preview phase, Pulumi invokes docker build.

Choose the “yes” option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to the Resources tab and filter to S3 resources. Then, on the S3 bucket, click the link to go to the AWS Console.

Upload a Video

Upload an .mp4 video to your S3 bucket, making sure to encode the desired time index in the filename. For instance, use myvideo_00-02.mp4 to extract the frame at 0 minutes and 2 seconds.

You can use this video of my cat, which you should save as cat_00-01.mp4.

Upload this video in the S3 console. Or, use the AWS CLI along with the pulumi stack output command:

$ aws s3 cp cat_00-01.mp4 s3://$(pulumi stack output bucketName)
upload: cat_00-01.mp4 to s3://bucket-3a4f226/cat_00-01.mp4

To see what the app is doing, use the pulumi logs command, with the --follow or -f parameter. This command aggregates all the logs for your compute, so the logs for the Lambda function and the Fargate task are all in one place!

Once the thumbnail has been generated, either view it in the S3 console, or download it with the AWS CLI:

$ aws s3 cp s3://$(pulumi stack output bucketName)/cat.jpg .
download: s3://bucket-0c91106/cat.jpg to ./cat.jpg

Clean Up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next Steps

In this post, we saw how easy it is to use containers and serverless functions in one application. With Pulumi, you get the best of both worlds, and don’t have to choose one or the other.

The sample code for this post is available in the Pulumi examples repo on GitHub.

For a version of this sample that includes AWS Rekognition, see the Video Thumbnailer with Machine Learning JavaScript example.

Deploying production-ready containers with Pulumi

Donna Malayeri — Wed, 20 Jun 2018 00:00:00 +0000

Containers are a great way to deploy applications to the cloud, especially with new execution models like AWS Fargate. Pulumi makes it easy to deploy production Docker containers, handling details such as creating a container registry instance in ECR, creating task definitions in ECS, and configuring a load balancer. With Pulumi, deploying a container to production is almost as easy as running it locally!

In this blog post, we’ll deploy a simple Docker container running NGINX.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the app

We’ll create a Pulumi project, define the infrastructure in JavaScript, and create a Dockerfile.

To create a new Pulumi project, run the following commands:

mkdir hello-containers && cd hello-containers
pulumi new aws-javascript

This creates a new project in the directory hello-containers. Next, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");
let service = new cloud.Service("pulumi-nginx", {
 containers: {
 nginx: {
 build: "./app",
 memory: 128,
 ports: [{ port: 80 }],
 },
 },
 replicas: 2,
});

/* export just the hostname property of the container frontend */
exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

These 15 lines of code are everything you need to deploy a custom container! We’re using cloud.Service, which is a high-level, convenient interface for building containers and provisioning an AWS container service. Using the build property, we point to a folder containing a Dockerfile, which is app in this case.

Now, let’s create a Dockerfile and a static page. Create a subfolder app with the following files.

Add the following as Dockerfile:

FROM nginx
COPY index.html /usr/share/nginx/html

Add the following as index.html:

<html>
 <head>
 <title>Hello World</title>
 <meta charset="UTF-8">
 </head>
 <body>
 <p>Hello containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

You should have the following directory structure:

Install the @pulumi/cloud-aws NPM package:

npm install –save @pulumi/cloud-aws @pulumi/cloud
Configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

pulumi config set cloud-aws:useFargate true

Deploy the app

Choose the yes option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to this link and click the Resources tab. You’ll see all the resources you’ve created. Notice that Pulumi has created an ECR repository, a load balancer, an ECS service and task definition, and IAM roles.

Now, go back to the stack details page, which shows the stack configuration, as well as the stack output property. The following line creates the stack output url:

exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

If you navigate to the link for url you’ll see the following page:

View logs

You can view container frontend and compute logs via the pulumi logs command. You can see here that most of the traffic is hitting routes that don’t exist, aside from the last log line where I navigated to the site using Chrome. Since this is an HTTP server that’s open to the internet, it’s not uncommon for attackers to try to find a security vulnerability on a site. Fortunately, our container is serving only static content. The logs command makes it easy to get details on your site traffic.

$ pulumi logs
Collecting logs for stack hello-containers-dev since 2018-06-15T15:01:18.000-07:00.
2018-06-15T15:27:47.468-07:00[ pulumi-nginx] 2018/06/15 22:27:47 [error] 5#5: *4 open() "/usr/share/nginx/html/proxychecker.axd" failed (2: No such file or directory), client: 62.210.157.152, server: localhost, request: "GET http://proxy.dazhou.net/proxychecker.axd?e=457758511%40qq.com&p=http%3A%2F%2F34.201.27.186%3A80&s=AS HTTP/1.1", host: "proxy.dazhou.net"
2018-06-15T15:27:47.468-07:00[ pulumi-nginx] 62.210.157.152 - - [15/Jun/2018:22:27:47 +0000] "GET http://proxy.dazhou.net/proxychecker.axd?e=457758511%40qq.com&p=http%3A%2F%2F34.201.27.186%3A80&s=AS HTTP/1.1" 404 170 "-" "Go-http-client/1.1" "-"
2018-06-15T15:32:02.128-07:00[ pulumi-nginx] 2018/06/15 22:32:02 [error] 5#5: *382 "/usr/share/nginx/html/phpmyadmin/index.html" is not found (2: No such file or directory), client: 178.239.177.212, server: localhost, request: "HEAD http://34.201.27.186:80/phpmyadmin/ HTTP/1.1", host: "34.201.27.186"
2018-06-15T16:07:05.874-07:00[ pulumi-nginx] 172.31.44.144 - - [15/Jun/2018:23:07:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36" "-"

Clean up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next steps

The sample code for this post is available in the Pulumi examples repo on GitHub. For an example application that connects two containers, see the Voting App TypeScript sample.

Pulumi Blog: Containers

105 Ways to Run Containers: The Cloud Container Iceberg

Constraints

Above the iceberg – The usual suspects

On the iceberg – The slightly less obvious

The kubernetes multiverse – Because one Universe of k8s wasn’t enough

Below the water – Kubernetes everywhere – Let a thousand K8s bloom

Middle of the iceberg – Serverless and function containers – Now you see me, now you don’t

Bottom of the iceberg – Specialized and hybrid platforms – The swiss army knife of container hosting

Below the iceberg – Containers on the edge of reason

The abyss – Where containers fear to tread

Beyond the abyss

Managing your container zoo with pulumi

The final word

Unified and Programmatic Approach to Infrastructure Management at BMW Using Pulumi

On this article:

The Challenges of a Sprawling Software Landscape

The Evolution of BMW’s Software Development Toolchain

Embracing Pulumi: Streamlining Infrastructure Management

Shared Modules: Promoting Reusability and Best Practices

Transformation Features: Protecting Legacy Services

Policy Enforcement: Ensuring Compliance and Security

The Benefits of Pulumi: Accelerating Development and Improving Maintainability

Accelerated Development with Shared Modules

Improved Maintainability and Consistency

Leveraging Python’s Ecosystem

Streamlined Cloud Migration

The Future of BMW’s Software Factory: Embracing the Cloud

Conclusion: Unlocking the Future of Automotive Software

Introducing The New Docker Build Provider

A brief history

Getting started

Migrating from docker.Image to docker_build.Image

Migrating from Docker provider v3

Migrating from Docker provider v4

Conclusion

Fast Docker Image Builds with Pulumi

TL;DR

What actually moves the needle

How does Pulumi build Docker images?

Why are my Docker builds slow?

How do I enable BuildKit and registry caching in Pulumi?

With the Docker Build provider (recommended)

With the Docker provider (docker.Image)

How do I write a Dockerfile that builds fast?

How do I build multi-platform images with Pulumi?

How do I see Docker build output in Pulumi?

How do I migrate an older docker.Image program?

Getting started

AWS Enterprise Container Management with Pulumi

Provisioning Container Infrastructure

Managing AWS Container Services

Support for the Entire Kubernetes API

Cluster Observability

Authentication and Authorization

Centralized Governance and Compliance Controls

Support for Hybrid Deployments

Automated Deployment of Infrastructure and Applications

Conclusion

Azure Container Services Compared: ACI vs App Service vs AKS

Azure Container Instances

What’s good about it?

Simplicity

Start-Up Times

Pricing

The Other Stuff

What’s not great about it?

Scaling

Configuration

What we’d use it for

Azure App Service

What’s it good for?

Integrations

Scaling & Availability

What’s not great about it?

Opinionated

What we’d use it for

Azure Kubernetes Service

What’s it good for?

Extensibility

With the Docker provider (`docker.Image`)

How do I migrate an older `docker.Image` program?