Pulumi Blog: Hackathon

Exploring how to solve circular dependencies

Daniel Bradley — Wed, 12 Jan 2022 00:00:00 +0000

This blog post originated from a hackathon and exploratory exercise, and should be considered in that context. None of the solutions exist in the Pulumi SDKs. For more information on this, please view our blog post on Circular Dependencies.

As part of our hackathon near the end of last year, we decided to explore solutions to a common problem when people are using Pulumi for their systems. A question that’s been asked in a few different forms is how to resolve circular dependencies between resources in a Pulumi program.

A simple example of this idea is a modern web application with a static front-end and an API, where the front-end needs to know the URL of the API to be able to call it and the API needs to know the source domain of the front-end to allow it access via CORS. As these two resources rely on one another to be created, they are circular dependencies.

For a deployable example, let’s build a serverless game of ping-pong—two functions that invoke one another until one misses!

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name, // <-- Can't access `pong` yet!
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

Programming languages don’t let us create one object that takes a value from another object that doesn’t yet exist. Therefore, we can’t express a circular dependency in this way.

The simplest workaround for this error is to get a reference to our own stack, which contains the outputs from the previous deployment. Now, we set the reference to pong within ping from the output of the last deployment.

const currentStack = new pulumi.StackReference(
 `acme-org/${pulumi.getProject()}/${pulumi.getStack()}`
);

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: currentStack
 .getOutput("pongName")
 .apply((pongName) => pongName ?? ""),
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

export const pingName = ping.name;
export const pongName = pong.name;

The downside of this approach is that we have to deploy twice to reach the desired state because we need the results of the previous deployment, though this approach might be fine for simple one-off circumstances.

How then could we make the process easier and clearer for users who need to resolve this kind of circular dependency regularly? We explored four options as part of the hackathon, so let’s go through each one, getting progressively closer to a possible solution.

Option 1: State

Instead of referencing outputs directly from our stack’s previous deploy and manually deploying twice, we could introduce a special type called “State”.

The State object does 3 tasks:

Read the value from the previous deployment, or provide a default if it’s the first deployment.
Update the value in the stack state using a “set” method.
Automatically triggers another deployment if its value has changed.

const pongName = new pulumi.State("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value,
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Engine re-runs "up" if this changes the value
pongName.set(pong.name);

The first deployment will match the preview, but when triggering the re-deployment, we’re making a change that wasn’t in the first preview. This action could also result in a never-ending loop where every deployment makes another change to the state and triggers another deployment. So let’s explore a different option that might not have this issue.

Option 2: Refs

This option looks almost identical to the “state” concept above, except we’re calling them references (refs):

const pongName = new pulumi.Ref("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value, // Special type the Pulumi engine will track
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Determine affected resources and re-update
pongName.set(pong.name);

The difference between refs and state is that, instead of triggering a whole re-deploy when the value changes, refs would be integrated into the dependency graph similarly to resources. When the value of the ref changes, we could internally work out which resources depend on that ref and change the target state for only those resources. This avoids the possibility of a never-ending update loop.

Both the “state” and “refs” approaches have the downside of not making the final outputs of the updated resource available to the rest of the program. The resource object would still have the values from after it first deployed. This means that other parts of the program might still deploy differently on subsequent deployments. Let’s explore another option that might address that problem.

Option 3: Create patch resources

We could explicitly model the third step by using a new resource type suffixed with “patch”:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const pingPatch = new CallbackFunctionPatch("ping-patch", {
 id: ping.id,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

The provider presents pseudo-resources for update or patch operations. The patch resource is using the same underlying resource as the original resource. This option is a very flexible approach that could be implemented by providers right now. These patch resources could also be used for resources that exist outside the Pulumi stack. The drawback is that the patch operation has to be implemented in each provider individually, and it significantly increases the overall size of each provider and its associated SDK. We’re still evaluating whether this option would be a good one considering that size increase, which increases download times as an example impact, and the engineering resources to implement the change. We have one more option to consider, as well.

Option 4: Allow a resource to update its goal state

When using the SDK, we could write something like the following code to create then update a little later on:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const updatedPing = ping.update({
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

When declaring a resource in a Pulumi program, you’re essentially saying, “This is the state I want my resource to be in.” The update method returns a whole new copy of the resource with new properties containing the outputs of the update.

What’s next?

The final option is quite close to a fully working solution, but there still remains a few tasks to complete before we could release it as a feature:

Finalise what the internal messages between the Pulumi’s internal components should look like.
Keep a record of circular dependencies in the stack state without causing knock-on issues elsewhere in Pulumi’s internal workings.
Decide what should happen if an update triggers a replacement—if this is potentially valid behaviour or not. We should be able to generate the set of properties which are able to be updated without causing replacement and provide these properties to the user as the arguments for the update method to avoid these issues before execution of the program.

Some other ideas that would be interesting to explore in the future are as follows:

How could we utilise patching of resources where a provider can support it?
Could we delete a resource that was created earlier in the program to allow the creation of temporary resources?

This hackathon project has opened up a number of new pathways for us to explore. If you want to participate, we always welcome contributions in our open-source repos at https://github.com/pulumi. We hope you have enjoyed this dive into how our team explored this concept as part of the hackathon. We’ll be exploring more of the hackathon projects soon on the blog!

2021 December Hackathon: Introduction

Laura Santamaria — Fri, 07 Jan 2022 09:39:28 -0600

Pulumi’s hackathon tradition continued in the last weeks of 2021 with our 2021 December hackathon. For one solid week, we had teams from across the company focus on improvements across the Pulumi ecosystem, and we brought in people from outside the engineering org to get perspectives on different needs. While there were some projects that were focused on internal work, there were still quite a few open-source projects that we can talk about publicly. We’ll get more details from some of those teams over a few more posts. In this post, however, we’re going to explore a bit about how we worked.

Setting up for success

We set up the teams for success before the hackathon began by starting discussions around project ideas in a Slack channel asynchronously. In in-person hackathons, teams often decide on projects the day of the start of the hackathon. By starting discussions early, we let folks decide on projects they’d be interested in contributing to and drove early architectural discussions so teams could hit the ground running. People also started to get excited about working on the various projects, and they were able to start connecting with folks outside of the engineering team to find time on their calendars early.

Using a survey, we gathered what sorts of projects or areas of the ecosystem people were interested in working on, their timezone, and how they would like to collaborate. By using this data to sort people into teams, we ensured that people who were interested in learning about new areas of the ecosystem could find the right projects, that people who needed to work in a more synchronous environment could be paired with others in their timezone, and that people who worked similarly could find people who worked like them. It was an interesting experiment.

Working remotely

One cool thing that we tracked this round is all of the different ways we collaborated across the different projects. In general, Pulumi is a remote workplace, and none of the teams were co-located for this hackathon. As a result, we had to consider how teams might work across timezones and outside of the classic physical co-located team dynamic. Each team got to pick the best ways for them to collaborate. Our teams’ preferences ranged from daily standups over Zoom to collaboration solely on Slack to extensive pair programming and Zoom hangouts. In all, each team used multiple ways to communicate and engage depending on which parts of their projects they were working on. Teams working on explorations of possible solutions often found that pair programming remotely was a great way to brainstorm, for example. Once teams had a solid sense of what needed to be done to get to the next part of a project, many still attended a hackathon-wide Zoom hangout session daily to have someone to hang out with and discuss any issues that arose. The whole org also had the opportunity to get to know one another a little better, and it did not have the feeling of lost time or forced socialization that weekly virtual watercoolers can have after a couple of meetings as many have found during this period of forced virtual interactions.

Showing off what we did

During the demo at the end of the hackathon, we asked teams to self-report which strategies they found the most helpful, what they learned, and obviously what the results of their hackathon project were. We found that everyone level-set to how their team chose to work, and the pairing of people based on location and work preferences helped a lot to have teams succeed.

We’ll have more from each team soon! Keep watch on this space for new posts about hackathon projects.

If you like our way of working and are interested in a new position, we have a lot of positions open right now and will have more in the future! Please head over to our careers page to find out how to apply.

Demos from the Multi-Language Component Hackathon

Lee Zen — Thu, 24 Jun 2021 00:00:00 +0000

At Pulumi, we have a tradition of hosting hackathons every so often to play with concepts and ideas that we may not typically encounter in our day-to-day product building activities. This past week, we’ve had two separate back-to-back hackathons. Our first hackathon, which was open to the community, focused on using our new multi-language component capabilities. Multi-language components allow developers to author reusable infrastructure abstractions in one language and make them available to others in all the languages that Pulumi supports. We were really excited to see what everyone would build!

We had a kickoff meeting on Tuesday morning where Lee Briggs walked participants through the basic steps for bringing up a multi-language component. If you’re interested in seeing the kickoff meeting, we’ve recorded it and have made it available for viewing on Pulumi TV. After the walk-through, teams got down to business!

We had some great ideas, including but not limited to components for a simplified Google Cloud project setup, an Azure web app service container, and an Amazon SageMaker model with a frontend for serving it.

As part of the hackathon, we received some great feedback from all the participants on improving the multi-language authoring experience, including reducing boilerplate code, ways to improve on the schema authoring experience, and documentation suggestions. We’re keen to continue to refine the experience.

Thank you to all the participants! If you’d like to see some of the components in action, you can watch our demo session. We saw the following component demos:

“Run my d#!n container” which demonstrated a component to run a container on any of the major cloud providers.
A component that brings up a new Google Cloud Project.
A component for creating an AWS Lambda function fronted by API Gateway
A component for creating an Amazon CloudFront distribution backed by an S3 bucket
A component that runs an AWS Lambda function as a cron.
API Gateway component that simplifies creating Amazon API Gateway resources
A serverless component that consumes Serverless configuration
Replicated S3 Bucket component to simplify cross-region object replication
Miniflux component that deploys the Miniflux service
.NET Boilerplate for multi-language components

Stay tuned for our next post on our general hackathon results. In our general hackathon, the Pulumi team tried out new ideas and investigated ways to improve upon the Pulumi experience!

You can watch the hackathon including the intro multi-lang components, the hackathon kickoff, and the demo session.