Pulumi Blog: Lee Zen

Demos from the Multi-Language Component Hackathon

Lee Zen — Thu, 24 Jun 2021 00:00:00 +0000

At Pulumi, we have a tradition of hosting hackathons every so often to play with concepts and ideas that we may not typically encounter in our day-to-day product building activities. This past week, we’ve had two separate back-to-back hackathons. Our first hackathon, which was open to the community, focused on using our new multi-language component capabilities. Multi-language components allow developers to author reusable infrastructure abstractions in one language and make them available to others in all the languages that Pulumi supports. We were really excited to see what everyone would build!

We had a kickoff meeting on Tuesday morning where Lee Briggs walked participants through the basic steps for bringing up a multi-language component. If you’re interested in seeing the kickoff meeting, we’ve recorded it and have made it available for viewing on Pulumi TV. After the walk-through, teams got down to business!

We had some great ideas, including but not limited to components for a simplified Google Cloud project setup, an Azure web app service container, and an Amazon SageMaker model with a frontend for serving it.

As part of the hackathon, we received some great feedback from all the participants on improving the multi-language authoring experience, including reducing boilerplate code, ways to improve on the schema authoring experience, and documentation suggestions. We’re keen to continue to refine the experience.

Thank you to all the participants! If you’d like to see some of the components in action, you can watch our demo session. We saw the following component demos:

“Run my d#!n container” which demonstrated a component to run a container on any of the major cloud providers.
A component that brings up a new Google Cloud Project.
A component for creating an AWS Lambda function fronted by API Gateway
A component for creating an Amazon CloudFront distribution backed by an S3 bucket
A component that runs an AWS Lambda function as a cron.
API Gateway component that simplifies creating Amazon API Gateway resources
A serverless component that consumes Serverless configuration
Replicated S3 Bucket component to simplify cross-region object replication
Miniflux component that deploys the Miniflux service
.NET Boilerplate for multi-language components

Stay tuned for our next post on our general hackathon results. In our general hackathon, the Pulumi team tried out new ideas and investigated ways to improve upon the Pulumi experience!

You can watch the hackathon including the intro multi-lang components, the hackathon kickoff, and the demo session.

Introduction to Building Multi-lang Components

Multi-lang Hackathon Kick-Off

Hackathon Demos!

How to Deploy Apps with AWS App Runner and Pulumi

Lee Zen — Tue, 18 May 2021 00:00:00 +0000

There are loads of benefits to packaging up an application as a container. You can ensure that your application has all the required dependencies and runs in the isolated, predictable environment you expect. When it comes to running that containerized application, there are many options, including Kubernetes, Amazon Elastic Container Service (ECS), and Docker. Often, running a container application at scale requires setting up a container orchestrator and providing network infrastructure to the containers. Configuring this can be complex, especially if you’re not familiar with virtual networking concepts such as virtual private clouds, load balancers, and the like.

AWS recently introduced App Runner, which allows you to easily deploy an application via either source code that lives in GitHub or container image repository. It handles all the configurations of networking, scaling, and security. In this blog post, we’ll show you how easy it is to use Pulumi to set up the necessary infrastructure to use App Runner.

In our case, we’ll build a container-based application. To create an App Runner service, we will be using Amazon’s Elastic Container Registry (ECR) to create a repository to host our container images. In addition, we’ll need to push our image to the repository and ensure that App Runner has permissions to pull images from our repository.

We’ll use Go for our examples today. First, let’s setup the ECR repository:

repo, err := ecr.NewRepository(ctx, "sampleapp", &ecr.RepositoryArgs{})

Note that we’re using Pulumi’s auto-naming support to generate the repository name.

Next, let’s make sure that the App Runner service can pull images from the repository. To do so, we’ll create a role:

role, err := iam.NewRole(ctx, "ecrAccessRole", &iam.RoleArgs{
 AssumeRolePolicy: pulumi.String(string(assumeRolePolicy)),
})

The assumeRolePolicy object is fairly straightforward, it’s just the JSON representation:

assumeRolePolicy, _ := json.Marshal(map[string]interface{}{
 "Version": "2012-10-17",
 "Statement": []map[string]interface{}{
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Sid": "",
 "Principal": map[string]interface{}{
 "Service": []string{
 "build.apprunner.amazonaws.com",
 },
 },
 },
 },
})

Finally, we assign the image pull policy to this role:

pullImagePolicy, _ := json.Marshal(map[string]interface{}{
 "Version": "2012-10-17",
 "Statement": []map[string]interface{}{
 {
 "Effect": "Allow",
 "Action": []string{
 "ecr:GetAuthorizationToken",
 "ecr:BatchCheckLayerAvailability",
 "ecr:GetDownloadUrlForLayer",
 "ecr:BatchGetImage",
 "ecr:DescribeImages",
 },
 "Resource": "*",
 },
 },
})

_, err = iam.NewRolePolicy(ctx, "ecrAccessRolePolicy", &iam.RolePolicyArgs{
 Role: role,
 Policy: pulumi.String(string(pullImagePolicy)),
})

With all of that setup, we can now build and push our container. For our example, we’ll create an app directory within our project containing a simple Dockerfile and index.html file:

FROM nginx:latest
COPY ./index.html /usr/share/nginx/html/index.html

Where index.html is simply:

<html>
 <head>
 <meta charset="UTF-8">
 <title>Hello, Pulumi</title>
 </head>
 <body>
 <p>Hello, world!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

To push the image, we simply use Pulumi’s Image component resource to build and push. We’ll need our ECR credentials, which we can obtain via a simple function call and pass that information as an argument to our Image resource:

ecrCredentials, err := ecr.GetAuthorizationToken(ctx, &ecr.GetAuthorizationTokenArgs{
 RegistryId: nil,
}, nil)
if err != nil {
 return err
}

// Build and push our image into the ECR repo.
dockerImage, err := docker.NewImage(ctx, "sampleapp", &docker.ImageArgs{
 ImageName: repo.RepositoryUrl,
 Build: docker.DockerBuildArgs{
 Context: pulumi.String("./app"),
 },
 Registry: &docker.ImageRegistryArgs{
 Server: repo.RepositoryUrl,
 Username: pulumi.String(ecrCredentials.UserName),
 Password: pulumi.String(ecrCredentials.Password),
 },
})
if err != nil {
 return err
}

At this point, we have everything we need to deploy our App Runner Service. Let’s deploy it!

appRunnerService, err := apprunner.NewService(ctx, "service", &apprunner.ServiceArgs{
 ServiceName: pulumi.String("myservice"),
 SourceConfiguration: &apprunner.ServiceSourceConfigurationArgs{
 AuthenticationConfiguration: &apprunner.ServiceSourceConfigurationAuthenticationConfigurationArgs{
 AccessRoleArn: role.Arn,
 },
 ImageRepository: &apprunner.ServiceSourceConfigurationImageRepositoryArgs{
 ImageIdentifier: dockerImage.ImageName,
 ImageRepositoryType: pulumi.String("ECR"),
 ImageConfiguration: &apprunner.ServiceSourceConfigurationImageRepositoryImageConfigurationArgs{
 Port: pulumi.String("80"),
 },
 },
 },
})
if err != nil {
 return err
}

And just like that, our service is up and running, and it only took a few dozen lines of code! With this kind of setup, we don’t have to worry about load balancers or scaling or anything like that – instead, we focused on building our application and stood up the bare minimum to have a repository image. We can export the URL of our service to visit it:

ctx.Export("serviceEndpoint", appRunnerService.ServiceUrl)

We’re excited at how easy it is to use Pulumi to deploy applications via AWS App Runner, and we can’t wait to see what exciting things you build with it! See AWS App Runner in action in this episode of Modern Infrastructure Wednesday.

Video Transcript

Hello, and welcome to another episode of Modern Infrastructure Wednesday. I’m your host, Lee Zen, and today we’re going to be covering how easy it is to use Pulumi to deploy an application using AWS App Runner. App Runner is a new service from Amazon that lets you deploy a container image or a source code repository as an application, without having to do too much around configuring anything. You don’t have to configure load balancers—you don’t have to configure any of that stuff you typically have to configure.

And so, here you can see I’m running the main.go file, and it’s going to create all the necessary resources along with, finally, the App Runner service. We’re going to go into all the source code to explain how all this works, but right now you can see what’s happening is—we’re actually doing the Docker build that’s going to publish. We’re doing the Docker build, and then after the Docker build completes, it’s pushing the image into our ECR repository. And then finally, you can see here the update succeeds, and then we actually get the service URL. So now let’s go take a look at our service. And while the service is updating—you can see it takes a little bit of time, not too much, but just a little bit of time to get going—we’re going to go in to see how our code actually works.

All right, so what’s actually happening here? Let’s look at our deployment.go file, which main.go actually calls. So we actually built an Automation API-based program, and you can see, you know, in the outputs, we create this image, we create these various things. And so let’s look at the code. The first thing we do is we create an ECR repository, and here we use auto-naming, so it just creates the name. Then next, we create a role so that App Runner can actually go ahead and pull the image from our repository. And so we create the role, we give it the correct policy, and then after that, we basically publish our image. You can see here we have the image resource, and there we actually just feed it the repository credentials.

So let’s take a quick look at the Dockerfile of the thing we’re building. It’s just copying that index.html file over—it’s a very simple file. So that’s really all there is to the infrastructure. It’s super simple. If we look at the Automation API side of things, all we’re doing is doing the usual Automation API stuff. We’re setting up a stack, we’re deploying our program. Actually, we point out the part where we’re deploying the program right up here, and as soon as we are complete, as soon as we finish deploying the program, we’re going to use the outputs of that program—namely, the image URL and the access role—and we’ll feed that to App Runner. That’ll basically invoke the Create Service API call here, and that’s actually what’s going to do the work.

So let’s go back to the console, and you can see here we have the app running—and voila! Exactly what we expect to see. So hopefully you enjoyed this demo. And really, yeah, like I said, just so easy to get everything running with App Runner in just a few simple lines of code and Pulumi. Thanks for watching, and have a great day.

Unit Testing Assets

Lee Zen — Thu, 11 Jun 2020 00:00:00 +0000

When deploying infrastructure, we want to ensure that what we’re deploying matches our expectations. One way to do so is via unit testing. We’ve talked about this concept in previous posts, such as in this overview and this post on deployments with .NET.

Often, when we’re creating cloud resources, we want to ensure that a resource’s underlying assets match certain properties. For example, the entrypoint or handler for a cloud function should be an executable function. Similarly, objects we’ll serve as static assets on a website should not exceed a certain size. We can use Pulumi’s unit testing framework along with language-specific constructs such as introspection or filesystem calls to ensure this type of correctness. We’ll walk through some examples to show just how easy it is to do so.

The examples below are in Python, but you can easily achieve the same types of tests using introspection/reflection in Node, .NET, and Go.

Check that Functions are Executable

When creating a Lambda function, we’re expected to pass in a function handler that gets called upon invocation. Typically, the value is something like mod.handler, where mod is the module name and handler is a function. A basic check would be to make sure that the module exists and that the function is callable.

We can easily construct this unit test like so:

test_infra.py:

import unittest
import pulumi

class MyMocks(pulumi.runtime.Mocks):
 def new_resource(self, type_, name, inputs, provider, id_):
 return [name + '_id', inputs]
 def call(self, token, args, provider):
 return {}

pulumi.runtime.set_mocks(MyMocks())

import infra

class TestInfrastructure(unittest.TestCase):

 # Test that the function entrypoint is actually an executable function.
 @pulumi.runtime.test
 def test_function_handler_callable(self):
 def check_entrypoint(handler):
 import importlib
 from types import ModuleType
 from inspect import signature, Parameter

 # The handler should be of the form <module_name>.<function>
 module_name, function_handler = handler.split('.', 1)

 # Check that we can load the module.
 module = importlib.import_module(module_name)
 self.assertIsInstance(module, ModuleType)

 # Check that the function handler is actually callable.
 fn = getattr(module, function_handler)
 self.assertTrue(callable(fn))

 # Check that it has two parameters: event and context
 sig = signature(fn)
 self.assertListEqual(['event', 'context'], list(sig.parameters.keys()))

 return infra.lambda_function.handler.apply(check_entrypoint)

Here, we use Python’s introspection capabilities to import a module based on a string that we obtain from the Lambda function definition. Similarly, after we import the module, we can verify that the function is callable and has the appropriate arguments for a Lambda function. At this point, we’ve confirmed that we’ve correctly “wired in” a function handler. Separately, we would expect another set of unit tests that verifies the correctness of the handler itself.

Here’s the corresponding infrastructure code to make the test pass, assuming there’s also a file called lambda_function.py, which contains a function handler(event, context).

infra.py:

import pulumi
import pulumi_aws as aws

lambda_role = aws.iam.Role('example-role', assume_role_policy=f"""{{
 "Version":"2012-10-17",
 "Statement":[{{
 "Effect": "Allow",
 "Principal":{{
 "Service": "lambda.amazonaws.com"
 }},
 "Action": "sts:AssumeRole"
 }}]
}}
""")

attach_lambda_role_execution = aws.iam.RolePolicyAttachment('example-attach-execute',
 policy_arn='arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole',
 role=lambda_role.name)

module_name = 'lambda_function'
module_file = f'{module_name}.py'
lambda_function = aws.lambda_.Function('example-function',
 handler=f'{module_name}.handler',
 role=lambda_role.arn,
 runtime='python3.7',
 code=pulumi.AssetArchive({
 module_file: pulumi.FileAsset(module_file)
 })
)

If lambda_function.py doesn’t exist or there’s no function handler within that module, the test will fail.

File Assets

Another example of checking for underlying asset correctness is when providing files to a resource. For example, we might want to check that the content type we provide to an object matches that of the resource:

test_infra.py:

# Test that the bucket object matches the correct mimetype
@pulumi.runtime.test
def test_bucket_object_content_type(self):
 def check_filetype(args):
 source, mimetype = args
 import filetype
 kind = filetype.guess(source.path)
 self.assertEqual(kind.mime, mimetype)

 return pulumi.Output.all(infra.bucket_obj.source, infra.bucket_obj.content_type).apply(check_filetype)

Here, we use the filetype module to guess at the type based on reading the file itself. We then check that the mime type specified on the bucket object matches.

For example, if the file represented by image.png isn’t a PNG, this test would fail:

infra.py:

bucket = aws.s3.Bucket('my-bucket')

bucket_obj = aws.s3.BucketObject('my-bucket-obj',
 bucket=bucket.id,
 content_type='image/png',
 source=pulumi.FileAsset('image.png'))

Secrets

When working with secrets, we want to ensure that provided values are secret and that corresponding outputs are secret. This helps ensure that our state never contains secrets as plaintext values. We can write a simple test to check this:

test_infra.py:

# Test that the secret input is marked as secret
@pulumi.runtime.test
def test_secret_input_and_output(self):
 self.assertTrue(infra.secret_password_v1.secret_string.is_secret().result())

If we try to write something like the following in our code, it’ll fail this test:

infra.py:

secret_password = aws.secretsmanager.Secret('example-secret')
secret_password_v1 = aws.secretsmanager.SecretVersion('example-secret-version',
 secret_id=secret_password,
 secret_string='example-string')

If we change secret_string='example-string' to secret_string=pulumi.Output.secret('example-string') our test will pass.

Conclusion

There are many other examples of resources where it’s useful to check the underlying properties of the input assets. For example, checking the validity of certificates, keys, files, and container images are all assets that benefit from making sure we’re wiring in resources correctly to our cloud infrastructure. We hope you take this opportunity to use Pulumi’s unit testing framework to check your infrastructure’s underlying assets for correctness. Please visit the unit testing guide to learn more.