How Pulumi works
Pulumi uses a desired state (declarative) model for orchestrating and managing infrastructure with the flexibility to author your infrastructure code using programming languages you know such as TypeScript, Javascript, Python, Go, C#, and Java. This model provides the advantages of programming structures like loops, conditionals and functions as well as your IDE with autocomplete, type checking, and documentation. When you author a Pulumi program the end result will be the state you declare, regardless of the current state of your infrastructure.
A Pulumi program is executed by a language host to compute a desired state for a stack’s infrastructure. The deployment engine compares this desired state with the stack’s current state and determines what resources need to be created, updated or deleted. The engine uses a set of resource providers (such as AWS, Azure, Kubernetes, and +150 more) in order to manage the individual resources. As it operates, the engine updates the state of your infrastructure with information about all resources that have been provisioned as well as any pending operations.
The following diagram illustrates the interaction between these parts of the system:
In the next section, we will describe each of these components and see how they all fit together during an invocation of pulumi up
.
Language hosts
The language host is responsible for running a Pulumi program and setting up an environment where it can register resources with the deployment engine. The language host consists of two different pieces:
- A language executor, which is a binary named
pulumi-language-<language-name>
, that Pulumi uses to launch the runtime for the language your program is written in (e.g. Node or Python). This binary is distributed with the Pulumi CLI. - A language SDK is responsible for preparing your program to be executed and observing its execution in order to detect resource registrations. When a resource is registered (via
new Resource()
in JavaScript orResource(...)
in Python), the language SDK communicates the registration request back to the deployment engine. The language SDK is distributed as a regular package, just like any other code that might depend on your program. For example, the Node SDK is contained in the@pulumi/pulumi
package available on npm, and the Python SDK is contained in thepulumi
package available on PyPI.
Deployment engine
The deployment engine is responsible for computing the set of operations needed to drive the current state of your infrastructure into the desired state expressed by your program. When a resource registration is received from the language host, the engine consults the existing state to determine if that resource has been created before. If it has not, the engine uses a resource provider to create it. If it already exists, the engine works with the resource provider to determine what, if anything, has changed by comparing the old state of the resource with the new desired state of the resource as expressed by the program. If there are changes, the engine determines if it can update the resource in place or if it must replace it by creating a new version and deleting the old version. The decision depends on what properties of the resource are changing and the type of the resource itself. When the language host communicates to the engine that it has completed the execution of the Pulumi program, the engine looks for any existing resources that it did not see a new resource registration and schedules these resources for deletion.
The deployment engine is embedded in the pulumi
CLI itself.
Resource providers
A resource provider is made up of two different pieces:
- A resource plugin is the binary used by the deployment engine to manage a resource. These plugins are stored in the plugin cache (located in
~/.pulumi/plugins
) and can be managed using thepulumi plugin
set of commands. - An SDK which provides bindings for each type of resource the provider can manage.
Like the language runtime itself, the SDKs are available as regular packages. For example, there is a @pulumi/aws
package for Node available on npm and a pulumi_aws
package for Python available on PyPI. When these packages are added to your project, they run pulumi plugin install
behind the scenes to download the resource plugin from Pulumi.com.
Putting it all together
Let’s walk through a simple example. Suppose we have the following Pulumi program, which creates two S3 buckets:
const mediaBucket = new aws.s3.Bucket("media-bucket");
const contentBucket = new aws.s3.Bucket("content-bucket");
const mediaBucket = new aws.s3.Bucket("media-bucket");
const contentBucket = new aws.s3.Bucket("content-bucket");
media_bucket = s3.Bucket('media-bucket')
content_bucket = s3.Bucket('content-bucket')
mediaBucket, _ := s3.NewBucket(ctx, "media-bucket", nil)
contentBucket, _ := s3.NewBucket(ctx, "content-bucket", nil)
using System.Threading.Tasks;
using Pulumi;
using Aws = Pulumi.Aws;
class Program
{
static Task<int> Main() => Deployment.RunAsync<MyStack>();
}
public MyStack : Stack
{
public MyStack()
{
var mediaBucket = new Aws.S3.Bucket("media-bucket");
var contentBucket = new Aws.S3.Bucket("content-bucket");
}
}
package myproject;
import com.pulumi.Context;
import com.pulumi.Exports;
import com.pulumi.Pulumi;
import com.pulumi.aws.s3.Bucket;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var mediaBucket = new Bucket("media-bucket");
var contentBucket = new Bucket("content-bucket");
}
}
resources:
mediaBucket:
type: aws:s3:Bucket
contentBucket:
type: aws:s3:Bucket
Now, we run pulumi stack init mystack
. Since mystack
is a new stack, the “last deployed state” has no resources.
Next, we run pulumi up
. Since this program is written in aws.s3.bucket
aws.s3.bucket
s3.Bucket
s3.NewBucket
Aws.S3.Bucket
new aws.s3.bucket
new aws.s3.bucket
s3.Bucket
s3.NewBucket
new Aws.S3.Bucket
In this case, since the last deployed state has no resources, the engine determines that it needs to create the media-bucket
resource. It uses the AWS resource plugin in order to create the resource and the AWS resource plugin uses the AWS SDK in order to go create it. Note that the engine does not talk directly to AWS, instead it just asks the AWS Resource Plugin to create a Bucket. As new resource types are added, you can update the version of a resource provider to gain access to these new resources without having to update the CLI itself. When the operation to create this bucket is complete, the engine records information about the newly created resource in its state file.
As the engine was creating the media-bucket
bucket, the language host continued to execute the Pulumi program. This caused another resource registration to be generated (for content-bucket
). Since there is no dependency between these two buckets, the engine is able to process that request in parallel with the creation of media-bucket
.
After both operations have completed, the language host exits as the program has finished running. Then the engine and resource providers shutdown. The state for mystack now looks like the following:
stack mystack
- aws.s3.Bucket "media-bucket653a4"
- aws.s3.Bucket "content-bucket125ce"
Note the extra suffixes on the end of these bucket names. This is due to a process called auto-naming, which Pulumi uses by default in order to allow you to deploy multiple copies of your infrastructure without creating name collisions for resources. This behavior can be disabled if desired.
Now, let’s make a change to one of resources and run pulumi up
again. Since Pulumi operates on a desired state model, it will use the last deployed state to compute the minimal set of changes needed to update your deployed infrastructure. For example, imagine that we wanted to make the S3 media-bucket
publicly readable. We change our program to express this new desired state:
const mediaBucket = new aws.s3.Bucket("media-bucket", {
acl: "public-read", // add acl
});
const contentBucket = new aws.s3.Bucket("content-bucket");
const mediaBucket = new aws.s3.Bucket("media-bucket", {
acl: "public-read", // add acl
});
const contentBucket = new aws.s3.Bucket("content-bucket");
media_bucket = s3.Bucket('media-bucket', acl="public-read") # add acl
content_bucket = s3.Bucket('content-bucket')
mediaBucket, _ := s3.NewBucket(ctx, "media-bucket", &s3.BucketArgs{Acl: "public-read"}) // add acl
contentBucket, _ := s3.NewBucket(ctx, "content-bucket", nil)
using System.Threading.Tasks;
using Pulumi;
using Aws = Pulumi.Aws;
class Program
{
static Task<int> Main() => Deployment.RunAsync<MyStack>();
}
public MyStack : Stack
{
public MyStack()
{
var mediaBucket = new Aws.S3.Bucket("media-bucket", new Aws.S3.BucketArgs
{
Acl = "public-read", // add acl
});
var contentBucket = new Aws.S3.Bucket("content-bucket");
}
}
package myproject;
import com.pulumi.Context;
import com.pulumi.Exports;
import com.pulumi.Pulumi;
import com.pulumi.aws.s3.Bucket;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var mediaBucket = new Bucket("media-bucket",
BucketArgs.builder()
.acl("public-read") // add acl
.build());
var contentBucket = new Bucket("content-bucket");
}
}
resources:
mediaBucket:
type: aws:s3:Bucket
properties:
acl: public-read # add acl
contentBucket:
type: aws:s3:Bucket
When you run pulumi preview
or pulumi up
, the entire process starts over. The language host starts running your program and the call to aws.s3.Bucket causes a new resource registration request to be sent to the engine. This time, however, our state already contains a resource named media-bucket
, so engine asks the resource provider to compare the existing state from our previous run of pulumi up
with the desired state expressed by the program. The process detects that the acl
property has changed from private
(the default value) to public-read
. By again consulting the resource provider the engine determines that it is able to update this property without creating a new bucket, and so it tells the provider to update the acl property to public-read
. When this operation completes, the current state is updated to reflect the change that had been made.
The engine also receives a resource registration request for “content-bucket”. However, since there are no changes between the current state and the desired state, the engine does not need to make any changes to the resource.
Now, suppose we rename content-bucket
to app-bucket
.
const mediaBucket = new aws.s3.Bucket("media-bucket", {
acl: "public-read", // add acl
});
const appBucket = new aws.s3.Bucket("app-bucket");
const mediaBucket = new aws.s3.Bucket("media-bucket", {
acl: "public-read", // add acl
});
const appBucket = new aws.s3.Bucket("app-bucket");
media_bucket = s3.Bucket('media-bucket', acl="public-read") # add acl
app_bucket = s3.Bucket('app-bucket')
mediaBucket, _ := s3.NewBucket(ctx, "media-bucket", &s3.BucketArgs{Acl: "public-read"}) // add acl
appBucket, _ := s3.NewBucket(ctx, "app-bucket", nil)
using System.Threading.Tasks;
using Pulumi;
using Aws = Pulumi.Aws;
class Program
{
static Task<int> Main() => Deployment.RunAsync<MyStack>();
}
public MyStack : Stack
{
public MyStack()
{
var mediaBucket = new Aws.S3.Bucket("media-bucket", new Aws.S3.BucketArgs
{
Acl = "public-read", // add acl
});
var appBucket = new Aws.S3.Bucket("app-bucket");
}
}
package myproject;
import com.pulumi.Context;
import com.pulumi.Exports;
import com.pulumi.Pulumi;
import com.pulumi.aws.s3.Bucket;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var mediaBucket = new Bucket("media-bucket", BucketArgs.builder()
.acl("public-read") // add acl
.build());
var contentBucket = new Bucket("app-bucket");
}
}
resources:
mediaBucket:
type: aws:s3:Bucket
properties:
acl: public-read # add acl
appBucket:
type: aws:s3:Bucket
This time, the engine will not need to make any changes to media-bucket
since its desired state matches its actual state. However, when the resource request for app-bucket
is processed, the engine sees there’s no existing resource named app-bucket
in the current state so it must create a new S3 bucket. Once that process is complete and the language host has shut down, the engine looks for any resources in the current state which it did not see a resource registration for. In this case, since we removed the registration of content-bucket
from our program, the engine calls the resource provider to delete the existing content-bucket
bucket.
Creation and deletion order
Pulumi executes resource operations in parallel whenever possible, but understands that some resources may have dependencies on other resources. If an output of one resource is provided as an input to another, the engine records the dependency between these two resources as part of the state and uses these when scheduling operations. This list can also be augmented by using the dependsOn resource option.
By default, if a resource must be replaced, Pulumi will attempt to create a new copy of the resource before destroying the old one. This is helpful because it allows updates to infrastructure to happen without downtime. This behavior can be controlled by the deleteBeforeReplace option. If you have disabled auto-naming by providing a specific name for a resource, it will be treated as if it was marked as deleteBeforeReplace
automatically (otherwise the create operation for the new version would fail since the name is in use).
Declarative and imperative approach
With Pulumi, you author your infrastructure in your preferred programming language. When you run pulumi up
, the Pulumi CLI starts both language host for your selected programming language as well as the required providers (via the Pulumi engine). The providers, coordinated by the Pulumi engine, are the ones that perform the actual creation, modification, or deletion of your infrastructure. The separation of language support from the engine is what makes Pulumi so powerful, providing the best of both imperative and declarative approaches for your infrastructure as code solutions.
Here is a breakdown of each component of Pulumi’s architecture:
- Language host:
imperative
(JavaScript/TypeScript, Go, Python, C#/F#/.NET, Java) and declarative (YAML) - Pulumi engine:
declarative
- Providers:
imperative
Thank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.