Using Pulumi Inside Node.js Monorepos
Posted on
One of Pulumi’s core goals is to provide cloud engineers with access to the very best software engineering tooling available. Using traditional programming languages like Node.js, Python, Go, .NET and Java means the latest and greatest software engineering tools from each of these ecosystems is available to bring to bear on managing cloud infrastructure, natively integrated with your existing development environments.
In the Node.js ecosystem, we’ve seen an explosion of great tooling over the last couple of years around support for monorepos - larger repositories built out of many smaller projects, and sharing code and dependencies smartly across all the various projects. We’ve seen many of our Pulumi Node.js users adopting these tools and repo structures, including tools like Yarn Workspaces, pnpm, Turborepo, and especially Nx.
While it has always been possible to apply these tools to Pulumi Node.js projects in TypeScript or JavaScript just like any other Node.js project, we’ve recently made a number of enhancements and fixes to make sure that Pulumi works truly seamlessly with these tools.
In this post, we’ll show how you can build a seamless development workflow by integrating Pulumi code level abstractions, such as Component Resources, with a monorepo-based build system like Nx.
Component resources let us create reusable components that manage logical groupings of resources. With modern monorepo tooling, and our recent improvements to make Pulumi aware of monorepo setups, we can colocate these reusable components, Pulumi infrastructure programs, and application code all in one monorepo, and have Nx manage the build and deploy-time dependencies for us.
We will walk through an example project that deploys a website built with Astro to AWS S3. The complete code can be found at pulumi/examples/nx-monorepo.
Our example monorepo has the following structure:
- website: A website built with Astro.
- components/s3folder: A component resource that manages a S3 bucket and its access policies.
- components/website-deploy: A component resource resource that manages files in a S3 bucket
- infra: A Pulumi program that uses the
s3folderandwebsite-deploycomponent resources to deploy the generatedwebsite.
By using npm workspaces we can have multiple npm packages managed from a singular top-level package. Npm will take care of installing the dependencies for all our packages and enables packages within the monorepo to reference each other. In the package.json at the root of the monorepo we setup the workspaces matching our folder structure. Yarn workspaces work similarly and are fully supported by Pulumi.
// package.json
{
...
"workspaces": [
"components/*",
"infra",
"website"
]
}
TypeScript
Pulumi has builtin TypeScript support and compiles your code on the fly without manual build-step, however this is currently limited to TypeScript 3.8. We are working on providing more choice here, but in the mean time Nx makes it easy to add a build-step to compile code using any version of TypeScript. For this example we are using the latest and greatest, TypeScript 5.4.
Declaring Dependencies
If we were to manually build and deploy the code in our monorepo, we would have to run the following steps:
- Run
astro buildto generate the HTML output that we want to deploy. - Run
npm run buildfors3folderandwebsite-deploy. - Run
npm run buildforinfraafterwards, since this importss3folderandwebsite-deploy. - Run
pulumi upto deploy our website.
Instead of running these steps manually, we will leverage Nx’s task running mechanism, which can be made aware of the dependencies between the packages in the monorepo. This ensures we don’t accidentally forget a step. On top of that, Nx provides caching, so we also avoid re-running unnecessary tasks on subsequent runs.
Because we are using npm workspaces, Nx can understand the dependencies declared in the package.json for each of the packages in our monorepo. For example in infra/package.json we declare that we depend on s3folder, website and website-deploy.
We also let Nx know that the deploy targetΒ in the infra package depends on the build step, and the HTML generation step of the website package:
// infra/package.json
{
"name": "infra",
"dependencies": {
"@pulumi/pulumi": "latest",
"s3folder": "*",
"website-deploy": "*",
"website": "*"
},
"scripts": {
"build": "tsc",
"deploy": "pulumi up --stack dev"
},
"nx": {
"targets": {
"deploy": {
"cache": true,
"dependsOn": [
"build",
"website:generate"
]
}
}
}
}
In nx.json we let Nx know that each package’s build-step depends on its dependencies’ build-steps. With that Nx now knows that in order build infra, it first needs to build its dependencies.
// nx.json
{
"extends": "nx/presets/npm.json",
"$schema": "./node_modules/nx/schemas/nx-schema.json",
"targetDefaults": {
"build": {
"cache": true,
"dependsOn": [
"^build"
]
}
}
}
The full dependency graph, as understood by Nx, can be visualized by running npx nx deploy infra --graph.
Walkthrough
Now that we have taught Nx all about the dependencies in our monorepo, we can run npx nx deploy infra and Nx will run all the required steps in the correct order, using parallelism where possible.
β 4/4 dependent project tasks succeeded [0 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
> nx run infra:build
> infra@1.0.0 build
> tsc
> nx run infra:deploy
> infra@1.0.0 deploy
> pulumi up --stack dev
The stack 'dev' does not exist.
If you would like to create this stack now, please press <ENTER>, otherwise press ^C:
Created stack 'dev'
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/julienp/nx-monorepo/dev/previews/fc7630fd-7dc4-4c7e-baa0-3d6e014fc90a
Type Name Plan
+ pulumi:pulumi:Stack nx-monorepo-dev create
+ ββ pulumi:examples:WebsiteDeploy my-website create
+ β ββ aws:s3:BucketObject index.html create
+ ββ pulumi:examples:S3Folder my-folder create
+ ββ aws:s3:Bucket my-folder create
+ ββ aws:s3:BucketPublicAccessBlock public-access-block create
+ ββ aws:s3:BucketPolicy bucketPolicy create
Outputs:
websiteUrl: output<string>
Resources:
+ 7 to create
Do you want to perform this update? yes
Updating (dev)
...
NX Successfully ran target deploy for project infra and 4 tasks it depends on (27s)
The output tells us that Nx found the 4 dependant tasks for the deploy task of the infra package and ran them successfully.
If we now run npx nx deploy infra again, we can see Nx’s caching mechanism in action. Nx notices that there are no code changes to the infra package, and that none of its dependencies have changed either, so it just replays the output from the previous run.
β 3/3 dependent project tasks succeeded [3 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
>nx run infra:build [existing outputs match the cache, left as is]
> infra@1.0.0 build
> tsc
> nx run infra:deploy [existing outputs match the cache, left as is]
> infra@1.0.0 deploy
> pulumi up --stack dev
Previewing update (dev)
...
NX Successfully ran target deploy for project infra and 4 tasks it depends on (29ms)
Nx read the output from the cache instead of running the command for 5 out of 5 tasks.
Now let’s create a new page in our website by adding a markdown file in the src/pages folder.
echo "Hello, World!" > website/src/pages/hello.md
When we run npx nx deploy infra again, Nx notices that there is a change to the website and reruns the HTML generation step. Since one of the dependencies of infra has now changed, the stack is re-deployed.
β 2/2 dependent project tasks succeeded [2 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
> nx run infra:build
> infra@1.0.0 build
> tsc
> nx run infra:deploy
> infra@1.0.0 deploy
> pulumi up --stack dev
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/v-julien-pulumi-corp/nx-monorepo/dev/previews/8d9db825-77e1-4d1a-8f9b-8279481e7e40
Type Name Plan
pulumi:pulumi:Stack nx-monorepo-dev
ββ pulumi:examples:WebsiteDeploy my-website
+ ββ aws:s3:BucketObject hello/index.html create
Resources:
+ 1 to create
8 unchanged
Do you want to perform this update? yes
Updating (dev)
...
NX Successfully ran target deploy for project infra and 4 tasks it depends on (19s)
Nx read the output from the cache instead of running the command for 2 out of 5 tasks.
We can see that Nx used the cached results for 2 of our tasks, s3folder and website-deploy didn’t change, so Nx skips rebuilding them. Nx also ran the other 3 tasks to generate the website’s HTML, building the infra package and finally deploying the infra package.
Conclusion
In this post we’ve shown how we can combine Pulumi’s code level abstractions with a build system to create a seamless developer workflow. By using a monorepo we can colocate our reusable components with the Pulumi programs that use the components, as well as our application code. Tools like Nx allow us to intelligently manage the dependencies between the packages in the monorepo, ensuring we are always deploying the correct version of the code.
Pulumi provides you with access to the latest and greatest software engineering tools (like Nx), and allows you to bring them to bear on managing cloud infrastructure, natively integrated with your existing development environments.
