Pulumi Blog: Customers

Seven Rules for Building an AI-Native Software Factory

Ewan Dawson — Thu, 21 May 2026 06:00:00 -0700

Ewan Dawson is CTO of Compostable AI, where five engineers run an AI-native software factory: nineteen clients, custom AWS deployments, most of them shipped within a day of contract signing. This article is adapted from his recent Pulumi webinar, and covers rules in more depth than we had time for on stage.

For the past twenty years, I’ve viewed software development as a craft. The best engineers drew on decades of experience to get every function right.

But two years into the agentic AI revolution, I realised software is going to look more like a factory than a craft. The economics have changed. We can’t treat code as bespoke anymore. To scale, we have to think industrial — use the tools to ship more value with fewer engineers.

I joined Compostable AI soon after it was founded 2.5 years ago, and I built the engineering org AI-native from day one. The technology has come a long way since then, and so has my understanding of what AI-native actually means. Here are seven rules I keep coming back to.

An AI software factory is a software operation where autonomous agents write and ship most of the code. The engineers stop writing it by hand and spend their time deciding what gets built and talking to customers. The rules below are our rules for building and running an AI-native software factory.

1. Transform, don’t enhance

Going AI-native isn’t an upgrade to your existing process. If you treat AI as a way to hand your developers smarter tools, you leave most of the value on the table. You get the leverage by rebuilding how you write software — and the culture and processes around it.

I know that’s a tall order for a large, mature engineering org. My advice: start small. Pick one team or one business area and run it as a fully AI-native function. Take what you learn and roll it out from there. And do the political work early, especially with your Governance, Risk, and Compliance function. Get GRC on your side early. Otherwise AI becomes a compliance fight instead of a structural advantage.

Don’t bolt AI onto your existing workflow. Redesign the workflow around what agents can do.

Most of the leverage in this technology comes from rebuilding around it. The tool change is the small part.

2. Remove the problem, don’t solve it

Going AI-native flips which problems are hard and which are easy. The right move often isn’t to engineer a solution. It’s to reframe the problem so it goes away.

Here’s an example. Serving multiple clients with agents writing the code, blast radius wasn’t a hypothetical. One bad agent run could trash a customer’s database, or leak one client’s data into another’s. Our instinct was to build a secure multi-tenant sandbox with guardrails, approvals, rollback. But every version we tried still had agents loose in a shared environment, one bug away from making one customer’s data visible to another’s. So we removed the problem: every client gets two dedicated AWS accounts, one for production and one “digital twin” staging account. Agents iterate on staging until the work checks out. Only then does it ship to production. We have nineteen accounts now, one per client.

Managing nineteen AWS accounts with five engineers used to be an administrative nightmare. When code is cheap, infrastructure-as-code tools like AWS Control Tower and Pulumi make it the easier path.

Remove the problem before you try to solve it.

It’s cheaper to reframe the problem than to engineer your way through it.

3. Pick tools your agents can drive

Removing problems is the process side. The other side is tooling. If you want an automated factory, your tech stack has to be something agents can drive. This overlaps a lot with tools that have great developer experience. If a tool has a robust API plus a clean CLI, agents can drive it. If it’s heavy click-ops around a web UI, agents stop there.

We didn’t get there first try. Our first IaC tool worked fine when we had a couple of clients. As we added more, accounts drifted, deployments slowed, retries got complicated. We needed something built for where we were heading.

I went looking, and Pulumi fit. We express infrastructure as type-safe code — TypeScript, in our case, rather than HCL — and agents are good at writing it. Pair that with Pulumi Neo — pre-loaded with domain-specific Pulumi skills — and we ship infrastructure that follows best practices. One of my colleagues put it: “The scary thing about Neo is it just seems to know everything about what we do.” Pulumi IaC plus Pulumi ESC for configuration beats stitching tools together. And TypeScript lets us build higher-level abstractions that keep the AWS account fleet tractable.

“I don’t actually care if it’s HCL or TypeScript, as long as my software development agents can write it. And they do a better job with TypeScript than HCL.”

Tools have to share your AI-native mindset. If they don’t integrate deeply, the human becomes the glue.

If part of your stack still requires a human to click through a web UI to provision an account, your agents stop there.

4. Don’t let one agent do everything

When I first started with agents, I reached for a god prompt: one massive system prompt meant to guide a single agent through the whole software lifecycle. It didn’t work. Agents struggle when you give them multiple goals. The writer is lenient on its own work — it won’t catch what it just shipped. You don’t want it reviewing the code, checking for security flaws, or hunting bugs.

We get better results from a constellation of specialized agents, each handling one part of the line. Pulumi Neo handles infrastructure. Alongside it sit agents specialized in:

Code implementation
Code review and testing
Security auditing
Internal standards compliance
Documentation updates

Tasks pass down the line. Clean code comes out the other end, with almost no human involved.

Don’t let any agent mark its own homework. Specialize by job.

Treat agents the way you’d treat a team. The one who writes the code shouldn’t be the one signing it off.

5. Measure human hours per unit of value

Once we had agents writing and agents reviewing, throughput went up — but the bottleneck moved past the PR. Engineering hours were still the most expensive thing in the building, so my core metric is human hours per unit of value produced. Minimize that.

That means hunting for every step that still goes through a person — especially the mid-pipeline steps between ideation and production. Automate the human touchpoints along that line, and the factory runs 24/7.

Pushing automation this hard also forces good engineering. A chaotic, undocumented process is impossible to automate. Good engineering is still good engineering, AI or not. Agents won’t fix a weak process.

Measure human hours per unit of value. Treat every one as a bottleneck to remove.

You can’t automate what you can’t describe. Every human in the pipeline marks a piece that hasn’t been described yet.

6. Design for convergence, not one-shot correctness

Even with the human touchpoints removed, the agents don’t ship right the first try. Once you embrace the factory pipeline, you stop needing them to. We design for convergence instead — a system that lands on the right answer through automated iteration.

The loop we run looks like this:

Refinement: agents iterate on the Product Requirements Document until the problem is clear.
Planning: agents draft multiple technical approaches, and evaluation agents pick the best one.
Implementation: coding agents write the software.
Review: specialized checking agents look for bugs, API misuse, and security flaws.

If the checkers find a problem, they hand it back to the implementation agent. The loop repeats until the tests pass and the agents agree on a clean PR. Once it converges, we merge and deploy to staging.

Two things have to be true. You need a way to evaluate the output. Without that, you don’t know when to stop. And the loop has to converge — each pass has to get closer. A checker that fails every PR for a different reason isn’t helping — it just keeps the work going in circles. The feedback has to narrow the search, not widen it.

Once it converges, the question moves on. How cheap can we make it? Lower the time to PR, reduce token count, drop the overall cost. The optimization never really ends.

Don’t aim for one-shot correctness. Design for convergence.

It doesn’t matter how many tries it takes, as long as the loop closes without a human in it. Get convergence first. The optimization comes after.

7. Run the factory in the cloud, not on a laptop

Even a converged factory has to live somewhere. Try running a fully automated factory on individual developers’ laptops, and it falls apart. Laptops are highly trusted machines. Put autonomous agents on them and your security posture drops, fast. And the factory has to run 24/7. Events come from elsewhere — PR comments, Slack threads, errors in test environments.

Cloud also kills configuration drift across a dozen developer machines. The same prompts run against different model versions, and env vars sit half-set on half the laptops. The thing you’re trying to optimize lives in different states across the team. Cloud isn’t just where the factory runs; it’s the only place a team can iterate on it together. Keep everything in one place — AWS, Pulumi Cloud, GitHub. The specific stack matters less than the principle of one place.

And the part that matters most: the factory keeps running, testing, and deploying long after we’ve closed our laptops and gone to sleep — the dark factory pattern, where the line keeps producing with the lights off.

Build the factory somewhere you can work on it — not just somewhere it can run.

A factory scattered across laptops can’t be improved as a system. Cloud keeps it in one shape, 24/7, and lets the team iterate together.

How to build an AI-native software factory

There’s no single blueprint for how to build a software factory, but these seven rules are the software factory model we run at Compostable AI: transform the workflow rather than bolt AI onto it, remove problems instead of engineering around them, pick tools your agents can drive, split work across specialized agents, measure human hours per unit of value, design for convergence over one-shot correctness, and run the factory in the cloud so it operates 24/7.

Closing thought

I’ve shipped more code in the last two years than I did in the fifteen before that. Most of it in languages I couldn’t write by hand. And that’s after a stretch in leadership where I wrote almost none.

If you’re where I was two years ago: don’t ask how AI fits into what you already do. The factory is built one rule at a time, and it’s not a template — it’s the practice of finding where you’re taking advantage of the new economics and where you’re not, where your practices still need an update. The leverage is in finding these places and improving them.

Frequently asked questions

What is a software factory?

A software factory runs software delivery as a repeatable industrial process instead of hand-crafting each feature, the way a physical factory turns out products on a line. The idea predates AI. An AI-native software factory is the version where agents do most of the building and the engineers steer.

How is a software factory different from traditional software development?

Traditional development treats code as a craft, each function written and reviewed by hand. A software factory treats software delivery as an industrial process: agents handle the writing and checking through an automated, converging loop, and the expensive human hours move to defining the work and deciding what to build. The goal is to minimize human hours per unit of value shipped.

How do you keep AI agents from breaking things across customers?

Rather than build a shared multi-tenant sandbox, Compostable AI removes the problem: every client gets two dedicated AWS accounts, one for production and a “digital twin” staging account. Agents iterate on staging until the work checks out, and only then does it ship to production. Infrastructure-as-code tools like AWS Control Tower and Pulumi make running that account fleet tractable for a small team.

What tools does an AI-native software factory need?

Tools your agents can actually drive, anything with a solid API and a clean CLI, rather than click-ops around a web UI. Compostable AI expresses infrastructure as type-safe TypeScript with Pulumi, pairs it with Pulumi Neo for domain-specific infrastructure skills, and uses Pulumi ESC for configuration. If part of your stack still requires a human to click through a UI, your agents stop there.

Watch the original Pulumi webinar. Read the Compostable AI case study, and learn more about Compostable AI and Pulumi Neo.

Platform Engineering Buffet at SEITENBAU

Nico Thomas — Tue, 27 May 2025 00:00:00 +0000

This post is based on my video interview with Nico Thomas, a platform engineer at SEITENBAU. In this article, Nico shares how he and his team built a flexible, reusable platform to support 20+ diverse projects using Pulumi. Watch the video below or read on for Nico’s take on building an infrastructure buffet. - Adam Gordon Bell

Imagine your company isn’t Spotify with one product pipeline, but 20 independent government projects running on Ansible-provisioned VMs—some on Kubernetes, some delivered into other customer data centers, some operated by clients themselves. Each team is building its own CI/CD, secrets management, and integrations. You’re left with a million reinvented wheels.

That was the situation my team and I faced at SEITENBAU, a German software company specializing in custom development for both public and private sectors. Unlike the typical platform engineering stories—where teams build infrastructure for a single product with multiple microservices—we had a fundamentally different challenge: How do you build a platform that serves dozens of completely independent projects, each with their own technology choices, deployment targets, and operational models?

The answer? Stop trying to force everyone onto the same plate. Instead, build a buffet.

The Buffet Philosophy

Think of a traditional platform as a set menu—also known as prix fixe—where everyone gets the same courses, prepared the same way, in the same order. But what if your diners include vegans, carnivores, and people with various dietary restrictions? What if some want a quick snack while others need a seven-course meal?

That was exactly the situation we faced. We had:

Government projects that needed to run on-premises due to data sovereignty requirements.
Modern cloud-native applications leveraging Kubernetes and GitOps.
Legacy systems still operating on virtual machines managed with Ansible.
Projects we ran as managed services and others we simply delivered to customers.
Teams using various CI/CD tools, artifact repositories, and deployment strategies.

A one-size-fits-all platform would have been a disaster. Instead, my team built a platform offering a rich selection of pre-configured, production-ready components. Teams could mix and match according to their specific needs, creating their own unique infrastructure combinations.

The Challenge: Beyond Microservices

Most platform engineering resources assume you’re building infrastructure for a single product – Spotify’s Backstage, for example. But what happens when you’re dealing with over 20 completely independent projects, each with distinct requirements?

Each project is distinct, featuring its own:

Technology stack (Java predominates, but we support various frameworks and languages).
Infrastructure requirements (cloud, on-premises, or hybrid setups).
Security and compliance constraints.
Operational models (managed by us, the client, or jointly).
Release cadences and deployment strategies.

Initially, this diversity seemed chaotic, but we quickly noticed that underneath these variations, common patterns emerged. Every project consistently required source control, CI/CD pipelines, artifact storage, secrets management, and deployment targets. The critical realization was that we didn’t need to standardize what teams chose—we needed to standardize how they connected these building blocks.

That’s where Pulumi became essential. Its flexibility allowed us to construct an infrastructure buffet that genuinely accommodated the diverse appetites and needs of every team.

Building the Kitchen: Architecture with Pulumi

Behind every great buffet is a well-organized kitchen. For our infrastructure, that kitchen is powered by Pulumi and Python, orchestrating a complex web of infrastructure services, configurations, and deployments. Using Pulumi together with Python has proven extremely valuable—it makes our day-to-day tasks easier to maintain and understand.

The Multi-Stack Approach

Rather than creating a single monolithic infrastructure program, we adopted a modular approach with multiple Pulumi stacks. This decision was essential for managing complexity and empowering team autonomy.

Our architecture comprises several specialized Pulumi programs:

GitLab System: Manages installations and configurations of GitLab.
Project Configuration: Sets up team-specific GitLab groups, CI/CD runners, and agents.
Cluster Management: Provisions Kubernetes clusters complete with standard add-ons.
VM Infrastructure: Manages traditional VM-based deployments utilizing Ansible.
Platform Services: Configures shared services such as artifact repositories and identity providers.

Each program can maintain multiple stacks (dev, staging, prod), allowing us to manage various environments independently while leveraging shared infrastructure code.

From Ingredients to Ready-Made Dishes: Reusable Components

The real magic happens in our component library. Just as a buffet offers complete dishes rather than raw ingredients, we’ve built high-level components that encapsulate complex infrastructure patterns.

Example: The Kubernetes Storage Component

Consider our Longhorn storage component. Instead of requiring teams to understand Helm charts, storage classes, and snapshot configurations, they simply declare they need persistent storage.

The component handles all the complexity: installing Longhorn, configuring appropriate replication for test vs. production environments, setting up snapshot classes, and integrating with our internal certificate authority.

Two Years Later: The Buffet is Open

After two years of running this platform in production, we’ve learned what works and what we’d do differently.

Building a platform for diverse, independent projects requires a different mindset than traditional platform engineering. Instead of forcing standardization, we’ve embraced flexibility while maintaining consistency where it matters.

The buffet approach—powered by Pulumi’s flexibility and Python’s expressiveness—has allowed us to serve 20+ different projects efficiently while giving teams the autonomy they need. It’s not about having one perfect dish; it’s about having a kitchen flexible enough to satisfy everyone.

If you’re facing similar challenges with diverse project requirements, consider building a buffet instead of a prix fixe menu. Your teams will thank you for it.

Want to build your own infrastructure buffet? Get started with Pulumi and join our community Slack to connect with engineers solving similar challenges.

How We Used Pulumi to Safely Migrate Oso's Global Infrastructure

Shaun Verch — Mon, 05 May 2025 00:00:00 +0000

Infrastructure as Code (IaC) tools such as Pulumi can provide enormous amounts of leverage, but they must be used correctly to also provide safety. One of our main jobs as infrastructure engineers is to not break things, so leverage without safety is useless. If something is safe, we can change things easily without even thinking about it. If it isn’t, we’ll be up until 2 a.m. fixing what we broke.

At Oso, we recently had to do a large infrastructure migration and learned three key principles that make a huge difference in using Pulumi for safe migrations. First, understand how it’s even possible to safely change your infrastructure independently of Pulumi. Second, write reusable and predictable Pulumi code. Finally, refactor your Pulumi code to your new version using zero-diff refactors.

Osoʼs Migration

Oso provides Authorization as a Service, so every application decision must first query Oso. This means that if Oso goes down, our customers will also go down. We can’t afford a moment of downtime.

However, we’re also growing quickly, and one of the side effects of growth is that architectural decisions that worked for us in the past quickly reach their expiration date. One example of this is when we hit a scaling limit and needed to quickly migrate from Amazon ECS Fargate to ECS on our own self-managed EC2 instances.

We can’t overstate the Pandora’s box of complexity this opened up for us. From the lifecycle of our data to our deployment orchestration, this touched nearly every part of our stack. On top of this, because performance and data locality are critical for us, we are deployed globally in 12 different regions. We need to be extremely careful to do this correctly.

How the Migration Actually Worked

The most important thing to realize when using an infrastructure tool like Pulumi is that ultimately you’re dealing with real resources. It doesn’t matter how cleverly you refactor your Pulumi code; if something gets deleted that isn’t supposed to be deleted, or if at any point a critical component is missing, there will be an outage.

At Oso, we were already using a Route 53 Traffic Policy to do latency-based per-region routing, so Route 53 was a convenient tool for us in this migration. The plan was to create an entirely new Oso infrastructure and include it in our traffic policy, but send 0% of our traffic to it. Then, we could run tests ourselves directly against the new infrastructure and gradually ramp up traffic until we were fully migrated.

That is the first big lesson we took away from this. If we didn’t have a point in our infrastructure where we could easily reroute customer traffic from one place to the other, this transition would have been impossible without downtime.

Writing Reusable Pulumi Code

The holy grail of infrastructure development is to achieve what we already have in application development: code reuse. Besides the benefit of reducing the burden of maintaining more code, it also reduces risk because we can concentrate our testing in one place and benefit from using code that has already been proven.

In Osoʼs case, because our migration involved creating a copy of our entire infrastructure with the new changes, code reuse was vital. Even the smallest reusable component meant fewer new bugs and less time fixing bugs we had already fixed in the old infrastructure.

In Pulumi, you can use the import and function system of the underlying programming language, so that part of managing libraries is not an issue. The real challenge lies in how to actually write the business logic of libraries in Pulumi because writing Pulumi code is not like writing normal code.

Take a look at this example of creating an ECS service:


let service = new aws.ecs.Serice("api",{ });

This line of code might seem trivial, but a lot is happening here, and it’s important to understand Pulumi’s programming model on a deeper level to know how to work with it properly.

The first thing to note is the first argument: api. This is the logical resource name in Pulumi. The subsequent arguments are more about how to actually create the resource, and we wonʼt talk about that here. This logical resource is how Pulumi identifies this resource in the internal state.

The internal state is the core of the Pulumi engine. It maps the “logical resource name" we provide to the actual resource that Pulumi either created or plans to create. The code example above, at some point deep inside the Pulumi engine, turns into something that roughly looks like this:

{
 "aws:ecs:Service": {
 "api": {...},
 ...
 },
 ...
}

This is the key to understanding why writing Pulumi code is not like writing normal code. There is one “state namespace” per Pulumi execution, which effectively means that there is one global object that we are adding resources to. So the new aws.ecs.Service function call wasn’t just creating a service object that we can pass around; it was registering the service with the global list of resources we plan to deploy.

After we understand that calls to the Pulumi library have side effects on this global state, we can start to write better Pulumi libraries. For example, say we have this function call:

deployWebStack("app",true,false,"enabled");

Do you know which resources this will create? Is it safe to call deployWebStack again with the same arguments? What would happen if we deleted this line of code? These are all the familiar perils of global state, and something we must proactively contend with to write a good Pulumi library.

Here are some rules that we recommend:

DO use Pulumi Component Resources in libraries that create resources.
- This encourages a more declarative object-based approach to resources, which fits better with Pulumi’s model and also causes these objects to show up in the Pulumi output.
Do NOT conditionally create resources inside library functions.
- If the function is called, all the resources should be created, and if it is not, the resources should not be created. Conditionals should be at the top level, because that’s where you are deciding what is deployed in a particular region or component.
Do NOT check the current infrastructure state inside library functions.
- This should be passed in as arguments; otherwise, the library function will have non-deterministic behavior.
Do NOT attempt to implement idempotent resource creation functions.
- See the previous rule. The only way to implement a function that creates a resource only if it hasn’t been created yet is to check if it has been created, leading to non-determinism.
DO write functional code as inputs to resources.
- You can still write normal code in some cases, as long as it doesn’t call into Pulumi. For example, the code that generates the list of environment variables for an app can be a normal function.
DO have a strategy for resource naming.
- Because any function might register new resources, and because you may even have naming conflicts on the cloud provider side, having a coherent strategy for naming resources is helpful. We don’t have anything prescriptive here, because it’s really a matter of preferences and specific situations.
DO favor assigning resources to named objects and variables over getter functions.
- Rather than implementing a function or some kind of “getter” to refer to another resource, save that resource object in a variable at some scope where it can be referenced by the other resources that need it. This can even be in the global scope to match the reality that all the resources end up in one resource namespace in the end.

There’s a lot of advice here, but it really all comes down to keeping in mind what Pulumi is actually doing with your code and remembering that calls into the Pulumi library are actually registering resources in the global state.

Infrastructure Snapshot Testing for Zero Diff Refactors

Now that we know what we want to change in our infrastructure, and we know how to write our Pulumi code in a reusable way, we’re faced with a familiar software engineering challenge: How can we safely refactor our existing code to get to where we want to be? To answer this question, we can step back from the Pulumi specifics for a moment and think about refactoring in general.

One method that is amazing for refactoring code is Snapshot Testing. With Snapshot Testing, you run the legacy code that you want to change, save the output, and create a unit test around that. Then, when you make a code change, you can detect if your code changes result in any unintended changes in behavior.

We can imitate this with Pulumi by taking advantage of the fact that we can see a preview of the changes Pulumi will make before we deploy them. While we don’t have direct access to Pulumi’s internal data model in the code itself to make this “snapshot,” the preview can give us this comparison. If we make changes to our infrastructure, as long as pulumi preview shows no changes, we are 100% sure that the refactor we made is safe and won’t cause any issues. Because of this, we at Oso internally call this method a “zero diff refactor.”

This is a relatively simple concept, but it is necessary to make this migration possible. In our original infrastructure, most of our resources were deployed in one monolithic Pulumi project, and resources were created at the top level rather than in libraries. The 12-region deployment also made running Pulumi on this project very slow, and we wanted a clean separation between the old and new infrastructure. Using snapshot testing to refactor allowed us to build and reuse our in-house infrastructure libraries with 100% confidence and let us focus on the parts that held the real risk.

Conclusion

This migration did not go smoothly. We constantly found problems and unexpected assumptions that our code was making about the underlying infrastructure.

But no one outside Oso noticed. We had a clear migration plan, factored out common Pulumi code, and used zero-diff refactors. Putting this all together meant we could benefit from the leverage that Pulumi has the potential to provide. This allowed us to make major changes with confidence that they wouldn’t break our system.

In the early days of Oso, people were often surprised by how much we were able to accomplish with such a small core team of engineers. Part of the secret to this is using the tools we have available to gain maximum leverage, but this isn’t the entire story. Knowing how to use them safely allows us to actually benefit from that leverage in practice.

If you’re interested in exploring Pulumi further, here are several ways to get involved:

Discover how Pulumi Crosswalk for AWS simplifies “day one” tasks in our AWS guide
Register for one of our upcoming workshops
But most importantly, try Pulumi today!

DevSecOps Game-Changer: Security Automation That Delivers Business Results

Sara Huddleston — Thu, 21 Nov 2024 07:16:30 +0000

Organizations are under constant pressure to deliver new products and features faster than ever. But speed alone isn’t enough—businesses must also navigate the complex challenges of ensuring security and managing infrastructure costs effectively.

Enter DevSecOps - the strategic integration of security practices into the DevOps workflow. By automating security processes, organizations can achieve improved speed, scalability, and business impact, all while ensuring security remains a priority.

Tivity Health, a leading health and fitness solutions provider, has embraced this DevSecOps approach using Pulumi, a modern infrastructure as code (IaC) platform. During PulumiUP 2024, David Giambruno, Tivity Health’s VP of Engineering and DevOps, shared how, by leveraging Pulumi, he led the transformation that continuously drives remarkable results in speed, cost savings, and security.

On this article:

The Beginning: From Data Center to the Cloud

Tivity Health’s journey began with a strategic decision to transition from a traditional data center environment to a cloud-native architecture. Rather than opting for a “lift and shift” approach, they made the bold choice to go directly to a cloud-native model, embracing the principles of DevSecOps along the way.

The need to automate infrastructure deployment and management was fundamental. Giambruno explained, “If you can’t automate it, we don’t need it.” This philosophy drove the team to seek out a solution that would not only streamline their operations but also empower their developers to focus on building innovative products for their customers.

Embracing Pulumi: The Power of Automation, Productivity, and Security

Tivity Health’s search for the right tool led them to Pulumi, a unified platform for all the infrastructure needs that allows teams to use general programming languages, such as TypeScript, Python, Java, and Go, to define and manage their cloud infrastructure. Giambruno and his team immediately recognized Pulumi’s ability to deliver on their key requirements:

Automation: Pulumi’s infrastructure as code (IaC) approach enabled Tivity Health to automate the deployment and management of its cloud environments, reducing the time and effort required for these tasks.
Productivity: using Pulumi’s general-purpose programming languages allowed developers to define, deploy, and manage infrastructure within their existing tools.
Security: Pulumi’s security features allowed Tivity Health to integrate security practices into its infrastructure deployment processes, reducing the risk of security breaches and ensuring compliance.
Cost Optimization: Pulumi’s ability to automate the spin-up and teardown of cloud environments on demand has led to significant cost reductions for Tivity Health.

Giambruno describes Pulumi as a “three-dimensional” tool, offering a versatile set of capabilities that have been instrumental in transforming Tivity Health’s operations. “The ability to use those dimensions in lots of different ways to do the automation is what really makes a difference to the teams,” he says.

Driving Business Impact Through Security Automation

By adopting Pulumi and DevSecOps automation, Tivity Health realized significant business benefits:

Speed and Agility: By automating its infrastructure deployment and management processes, Tivity Health has dramatically reduced the time required to spin up new environments or change existing ones. “We run it through automation and boom, it’s out, it’s done,” Giambruno says. This newfound speed and agility have empowered Tivity Health’s developers to focus on building products and features rather than getting bogged down in infrastructure-related tasks.
Cost Optimization: Tivity Health’s cloud-native approach and Pulumi’s automation capabilities have resulted in significant cost savings. The company estimates that its annual cloud spend has decreased from $9.5 million in its data center days to just $2 million—a staggering 79% reduction. These cost savings have allowed Tivity Health to redirect resources towards more strategic initiatives that drive business growth.
Improved Security and Compliance: Tivity Health’s DevSecOps strategy, anchored by Pulumi, has strengthened its security posture and compliance efforts. By integrating security directly into its infrastructure workflows, Tivity Health has improved its security posture. Automation ensures that security measures are enforced consistently across their cloud environments, reducing risks and improving compliance.

Fostering DevOps Culture Through Cross-Functional Collaboration

Using Pulumi provided more than technical benefits. It also enabled better collaboration between the development, security, and operations teams. By providing a common language and framework for infrastructure management, Pulumi has helped break down silos and align these traditionally disparate groups towards a shared goal of delivering secure, high-quality products faster.

Lessons Learned: Navigating the Cultural Shift

Transitioning to DevSecOps and cloud-native practices required a cultural shift at Tivity Health. Giambrono acknowledges that this cultural shift was not without its challenges. He emphasizes the importance of addressing the human element of change, offering the following advice for organizations embarking on a similar journey:

Start with a proof of concepts: Giambruno recommends beginning with a small-scale proof of concepts to demonstrate the capabilities and benefits of the new technologies and processes to help alleviate fears and build confidence among team members.
Empower developers and make them feel safe: By involving developers in the process and ensuring they feel comfortable with the new tools and workflows, Tivity Health gained user buy-in and overall support for the new DevSecOps approach.
Secure buy-in from business and financial stakeholders: Address the concerns of business and financial stakeholders early on, such as the impact on costs and the ability to deliver tangible results. This is crucial for securing the necessary support and resources.
Celebrate successes and build momentum: Giambruno highlights the importance of celebrating the team’s achievements along the way, even when there are bumps on the road, as this helps build enthusiasm and keep the momentum going.

The Future of DevSecOps and Pulumi at Tivity Health

As Tivity Health continues to refine its DevSecOps strategy, Giambruno is optimistic about the future. He envisions a world where the company’s ability to deploy new products and features rapidly will give Tivity Health a significant competitive edge, allowing it to better serve its customers and drive business growth.

Looking ahead, Giambruno is particularly enthusiastic about the potential of Pulumi’s AI-powered capabilities, which he believes will further streamline and optimize the company’s infrastructure management processes. “I’m super looking forward to the tests we’re going to do, like when we acquire someone and then taking them in, ‘Borg-ing’ them into our automation and seeing how much we can take out of their operating cost as fast as possible,” he says.

Tivity Health’s journey with Pulumi is a powerful example of how the right tool, DevSecOps strategy, and automation can drive tangible business results. By focusing on automation, security, and collaboration, organizations can achieve faster, cheaper, and better cloud deployments—putting them on the path to long-term success in an increasingly competitive, cloud-native world.

To learn more about Pulumi and how it can transform your software development and infrastructure management:

Get started with Pulumi Tutorials
Attend an upcoming workshop
Try out the Pulumi Neo code assistant to accelerate your infrastructure as code journey

Unified and Programmatic Approach to Infrastructure Management at BMW Using Pulumi

Sara Huddleston — Thu, 03 Oct 2024 09:52:10 +0000

In the ever-evolving world of automotive technology, BMW has been at the forefront of innovation, seamlessly integrating software into the heart of their vehicles. As cars become increasingly complex, with a growing emphasis on connectivity, over-the-air upgrades, and brand-specific user experiences, the need for a robust and scalable software development approach has become paramount.

Enter the BMW Software Factory, a platform that aims to empower the company’s developers and provide them with a superior development experience. At the core of this initiative is the adoption of Pulumi, a modern infrastructure as code (IaC) solution that has transformed the way BMW manages its software ecosystem.

On this article:

The Challenges of a Sprawling Software Landscape

BMW’s software journey has been a testament to the exponential growth of automotive technology. What started with simple engine controllers has evolved into a complex network of electronic control units (ECUs) scattered throughout the vehicle. As the software footprint continues to expand, BMW recognized the need for a unified and efficient approach to software development and deployment.

The company’s initial efforts involved introducing a platform called “Code Craft,” which provided a comprehensive stack of services to support the software development lifecycle. This stack included a GitHub Enterprise application for source code management, a Gerrit system for Android-based developments, a continuous integration (CI) pipeline, artifact stores, build caching, and various other tools and services.

However, as the demand for software-driven features grew, the complexity of managing this sprawling ecosystem became increasingly challenging. BMW found itself grappling with the need to scale its infrastructure, navigate network limitations across multiple data centers, and adapt to the ever-changing landscape of cloud computing.

The Evolution of BMW’s Software Development Toolchain

BMW’s journey to streamline its software development process has been gradual and iterative. The company’s initial approach involved using Ansible for deployment and a custom-built deployment scripting solution for its OpenShift cluster.

As the complexity of the platform increased, BMW turned to Helm and Kubernetes to manage its containerized services. However, as the company ventured into the public cloud, the limitations of these tools became apparent. The team recognized the need for a more comprehensive and scalable solution to manage their infrastructure as code.

At this critical juncture, BMW discovered Pulumi. This modern IaC solution offered a unique advantage – the ability to leverage a full-fledged programming language, Python, to define and manage their infrastructure. This shift proved to be a game-changer, allowing BMW to leverage its expertise in Python and benefit from the rich ecosystem of libraries and tools available in the Python community.

Embracing Pulumi: Streamlining Infrastructure Management

BMW’s adoption of Pulumi was a strategic move that aimed to address the growing complexity of its software ecosystem. By transitioning from a patchwork of tools to a unified IaC solution, the company was able to streamline its infrastructure management and improve developer productivity.

Shared Modules: Promoting Reusability and Best Practices

One of BMW’s key initiatives was developing a shared modules library, which allowed the team to abstract the complexity of various infrastructure components, such as databases, and provide a consistent and user-friendly interface for their developers.

By leveraging Pulumi’s Python bindings, BMW was able to create reusable modules that encapsulated best practices and sensible defaults, making it easier for developers to provision and manage infrastructure resources. BMW used Pydantic, a data validation library, to define schema-based configurations for its infrastructure. These were then integrated into its IDEs, providing developers with auto-completion and validation support.
This approach not only accelerated the development process but also ensured that the infrastructure deployed across the organization adhered to consistent security and compliance standards.

Transformation Features: Protecting Legacy Services

As part of their software factory, BMW also faced the challenge of integrating legacy services that did not natively support modern authentication and authorization mechanisms, such as OpenID Connect (OIDC).
To address this, BMW leveraged Pulumi’s transformation features to seamlessly inject an OAuth2 proxy into their deployments, providing a secure and consistent way to protect these services without requiring extensive modifications to the underlying applications.
By encapsulating this functionality within a shared module, BMW was able to apply the OAuth2 proxy to multiple services, ensuring a consistent and secure access control layer across their software ecosystem.

Policy Enforcement: Ensuring Compliance and Security

One key benefit of Pulumi’s IaC approach is the ability to define and enforce policies across the organization, ensuring that infrastructure deployments adhere to security and compliance standards. BMW has leveraged Pulumi’s policy-as-code capabilities to implement mandatory checks, such as ensuring that all S3 buckets are encrypted at rest, preventing the deployment of non-compliant resources.

By integrating these policy checks into their deployment workflows, BMW has shifted security and compliance concerns to the left, addressing issues early in the development process and reducing the risk of costly post-deployment remediations.

The Benefits of Pulumi: Accelerating Development and Improving Maintainability

BMW’s adoption of Pulumi has yielded significant benefits, transforming the way the company approaches software development and infrastructure management.

Accelerated Development with Shared Modules

The implementation of shared modules has been a game-changer for BMW. It allows developers to leverage pre-built and tested infrastructure components without having to reinvent the wheel. This has resulted in a significant acceleration of the development process, as teams can focus on building their applications rather than grappling with the complexities of infrastructure provisioning.

Improved Maintainability and Consistency

By centralizing infrastructure management within the shared modules, BMW has ensured that best practices and security standards are consistently applied across the organization. This has not only improved the overall maintainability of the software ecosystem but has also reduced the risk of security and compliance violations.

Leveraging Python’s Ecosystem

BMW’s decision to leverage Pulumi’s Python bindings has been a strategic advantage, as the company was able to tap into the rich ecosystem of Python libraries and tools. This has enabled the team to seamlessly integrate Pulumi with their existing Python-based toolchain, including dependency management, testing frameworks, and code quality tools, further enhancing the development experience.

Streamlined Cloud Migration

As BMW continues to expand its use of public cloud services, Pulumi has played a crucial role in simplifying the migration process. By providing a consistent IaC approach across on-premises and cloud environments, Pulumi has enabled BMW to manage its infrastructure in a unified manner, reducing the complexity and overhead associated with multi-cloud deployments.

The Future of BMW’s Software Factory: Embracing the Cloud

Looking ahead, BMW’s Software Factory is poised to take the next step in its evolution, with plans to transition away from the self-hosted backend and embrace cloud-native services. This strategic move aims to further improve developer productivity and reduce the internal effort required to maintain the underlying infrastructure.

By leveraging the expertise and reliability of cloud service providers, BMW can focus on delivering innovative software features to their customers while the cloud providers handle the operational aspects of running the necessary services.

As BMW continues to push the boundaries of automotive software, Pulumi’s role in its Software Factory will only become more crucial. By providing a scalable and flexible IaC solution, Pulumi empowers BMW’s developers to innovate with confidence and be secure in the knowledge that their infrastructure is managed consistently and in alignment with the company’s security and compliance standards.

Conclusion: Unlocking the Future of Automotive Software

BMW’s journey with Pulumi in the Software Factory showcases the power of modern IaC solutions in navigating the complexities of the automotive software landscape. By embracing a unified and programmatic approach to infrastructure management, BMW has accelerated development, improved maintainability, and ensured compliance across its sprawling software ecosystem.

As the automotive industry continues to evolve, with cars becoming increasingly software-driven, the lessons learned by BMW can serve as a blueprint for other organizations looking to streamline their software development and deployment processes. By leveraging the capabilities of Pulumi and other cutting-edge technologies, the future of automotive software is poised to be more efficient, secure, and responsive to the ever-changing needs of both manufacturers and consumers.

To learn more about Pulumi and how it can transform your software development and infrastructure management:

Get started with Pulumi Tutorials
Attend an upcoming workshop
Try out the Pulumi Neo code assistant to accelerate your infrastructure as code journey

AWS CDK vs Pulumi: Why SST Chose Pulumi

Sara Huddleston — Wed, 25 Sep 2024 07:32:40 +0000

Cloud computing tools evolve, and so must the frameworks developers rely on. For SST (Serverless Stack), AWS CDK was a great starting point—but it had limitations.

CDK tied infrastructure to AWS.
Debugging was frustrating due to CloudFormation templates.
Multi-cloud was nearly impossible.

The solution? Pulumi. In this post, we’ll explore why SST moved to Pulumi, what challenges they overcame, and what this means for developers building modern cloud applications.

TL;DR: Pulumi lets SST offer a faster, more flexible, and provider-agnostic infrastructure experience.

The Beginnings of SST

Application developers have witnessed the rapid evolution of cloud computing and the growing need for developers to have direct access to powerful cloud resources. However, the traditional tools and approaches to Infrastructure as Code (IaC) have often been geared more toward DevOps teams, leaving application developers feeling disconnected from the infrastructure side of their projects.

This was the driving force behind the creation of SST (Serverless Stack), a framework that aims to bridge the gap between application developers and infrastructure management. In the early days, SST was built on top of AWS’s Cloud Development Kit (CDK), which allowed developers to define their infrastructure using TypeScript or Python. While this was a step in the right direction, they soon realized that the limitations of CDK and the underlying AWS CloudFormation were holding them back from truly empowering application developers.

CDK and CloudFormation Limitations

As the SST’s team continued to work with CDK and CloudFormation, they encountered several challenges that led them to reevaluate their approach. One key issue was the disconnect between application developers’ thinking and working and traditional IaC tools’ operating methods.

With CDK and CloudFormation, the infrastructure code is essentially a code generator, producing an intermediary format (such as YAML or JSON) that is then executed to deploy the resources. This means that the actual code you write as a developer is not the same as the code running during the deployment process. This can lead to several problems, such as difficulty debugging, lack of visibility into the deployment process, and challenges in extending or customizing the deployment workflow.

Additionally, as the SST’s team expanded its focus beyond the AWS ecosystem and started exploring other cloud providers and even on-premises infrastructure, it found that the AWS-centric nature of CDK and CloudFormation was becoming a limitation. It needed a more flexible and provider-agnostic solution that would allow it to deploy and manage infrastructure across a wide range of platforms.

Feature	AWS CDK	Pulumi
Multi-Cloud Support	❌ AWS-Only	✅ AWS, Azure, GCP, On-Prem
Debugging	❌ Hard due to CloudFormation	✅ Real-time debugging
Language Support	⚠️ TypeScript, Python	✅ Any programming language
Deployment Speed	❌ Slower due to CloudFormation	✅ Faster direct execution
Visibility	❌ Hard to trace errors	✅ Clear deployment state
Extensibility	⚠️ Limited to AWS ecosystem	✅ Custom providers & workflows
State Management	❌ CloudFormation state file	✅ Pulumi-managed state
Secrets Management	❌ AWS Secrets Manager only	✅ Cross-cloud secret support

A Provider-agnostic Solution: Discovering Pulumi

It was during this time that the SST team discovered Pulumi, a modern IaC platform that takes a fundamentally different approach to infrastructure management. Instead of generating an intermediary format, Pulumi treats the infrastructure code as a first-class program executed directly during deployment.

This paradigm shift had several important implications for SST and its users:

Visibility and Extensibility: With Pulumi, the infrastructure code is the same code that is running during deployment, which means there is much greater visibility into the deployment process and the ability to extend or customize it as needed.
Multi-Cloud Capabilities: Pulumi’s provider-agnostic approach allows SST’s team to easily work with a wide range of cloud and on-premises platforms, giving their users the flexibility to deploy their infrastructure wherever it makes the most sense for their application.
Simplified Mental Model: For application developers, the Pulumi model of “your code is the deployment” aligns much more closely with their existing mental models and workflows, making it easier for them to adopt and work with IaC tools.

Transitioning to Pulumi

Transitioning SST from CDK to Pulumi was not a trivial undertaking. Still, they knew it was a necessary step to truly fulfill their mission of empowering application developers with powerful infrastructure management capabilities.

One key challenge they faced was re-implementing the higher-level components and abstractions they had built on top of CDK. These components were designed to simplify the infrastructure management experience for their users, and they wanted to ensure that they could provide a similar level of abstraction and ease of use with Pulumi.

Additionally, they had to carefully consider how to handle the various edge cases and complex deployment scenarios their users encountered with the CDK-based version of SST. They wanted to ensure that the Pulumi-based version would not only match the functionality of the previous version but also improve upon it and address some of the limitations they had encountered.

The Benefits of Pulumi

As they worked through the transition to Pulumi, they realized the significant benefits that Pulumi offered to both the SST team as the framework developers and the users.

Improved Visibility and Debugging

One of Pulumi’s most immediate and tangible benefits was the improved visibility and debugging capabilities it provided. With the infrastructure code being the same as the deployment code, they could easily trace issues back to the source and understand exactly what was happening during the deployment process.

This starkly contrasted the CDK/CloudFormation approach, where the intermediary format (CloudFormation templates) often obscured the underlying logic and made it much more difficult to diagnose and resolve problems.

Extensibility and Customization

Pulumi’s design also allowed the SST team to easily extend and customize the deployment process to meet the specific needs of their users. They could leverage Pulumi’s built-in extensibility features, such as custom providers and dynamic components, to integrate with a wide range of cloud and on-premises services and implement complex deployment workflows tailored to our users’ requirements.

This level of customization was much more challenging with the CDK/CloudFormation approach, where they often had to resort to hacky workarounds or custom Lambda functions to achieve the desired functionality.

Multi-Cloud Capabilities

As mentioned earlier, one key driver for their transition to Pulumi was the need to support a wider range of cloud and on-premises platforms. With Pulumi’s provider-agnostic approach, the SST team was able to easily add support for new providers, allowing their users to deploy and manage infrastructure across a diverse set of environments.

This flexibility has been particularly valuable for their users, who may have workloads or requirements that span multiple cloud providers or even on-premises infrastructure. With SST built on Pulumi, they can now manage all of their infrastructure through a single, consistent interface without having to juggle multiple tools or approaches.

Simplified Mental Model

One of Pulumi’s most significant benefits for SST users is the simplified mental model it provides. By treating the infrastructure code as a first-class program, Pulumi aligns much more closely with how application developers think and work.

Instead of having to navigate the complexities of intermediary formats, deployment pipelines, and the separation between infrastructure code and deployment code, SST users can now focus on writing their infrastructure logic in the same programming languages they use for their application code. This makes it much easier for them to understand, maintain, and extend their infrastructure as their needs evolve.

The Future of SST with Pulumi

With Pulumi’s foundation in place, they can now focus on further enhancing the developer experience and expanding the capabilities of their framework. Some of the key areas they are exploring include:

Deeper Integration with Application Frameworks: By leveraging Pulumi’s flexibility, they can create even tighter integrations between SST and the application frameworks and libraries that their users rely on, making managing infrastructure seamless alongside their application code.
Expanded Provider Support: the SST team will continue to add support for a wide range of cloud and on-premises providers, ensuring that their users can deploy and manage their infrastructure wherever it makes the most sense for their needs.
Improved Deployment Workflows: Building on Pulumi’s extensibility, they can create more advanced deployment workflows that address the specific needs of application developers, such as faster deployment times, better rollback capabilities, and more granular control over the deployment process.
Enhanced Observability and Monitoring: By treating the infrastructure code as a first-class program, they can provide their users with better visibility into the deployment process and more robust monitoring and observability capabilities, helping them to identify and resolve issues quickly.

As the SST team continues to evolve with Pulumi at its core, they aim to deliver an even more powerful and user-friendly infrastructure management experience for developers. This will empower teams to focus on building great applications while effortlessly managing the underlying infrastructure.

Conclusion

If you’re interested in exploring Pulumi further, here are several ways to get involved:

Discover how Pulumi Crosswalk for AWS simplifies “day one” tasks in our AWS guide
Watch our on-demand workshop Getting Stated with Infrastructure as Code on AWS
Register for one of our upcoming Platform Engineering or DevOps workshops
But most importantly, try Pulumi today!

Frequently Asked Questions

What is SST?

SST is a framework that makes building modern full-stack applications on your infrastructure easy.

What is SST’s Ion?

Ion is the code name for a new engine for deploying SST applications. The constructs (or components) are deployed using Pulumi instead of CDK and CloudFormation (CFN). Once Ion is stable, it will be released as SST v3.

Does SST use mostly Terraform or Pulumi?

SST leverages Pulumi behind the scenes for its providers and deployment engine while also bridging Terraform providers through Pulumi.

How does SST make money?

SST (Serverless Stack) makes money primarily through its managed service, called SST Console. The Console is optional and includes a free tier. In short, SST primarily relies on SaaS (Software as a Service) revenue through its managed platform and potentially supplemental income streams like enterprise services and consulting.

How Starburst Data Creates Infrastructure Automation Magic With Code

George Huang — Wed, 23 Aug 2023 00:00:00 +0000

This blog post summarizes a presentation by Matt Stephenson at PulumiUP 2023.

Matt Stephenson is Senior Principal Software Engineer for Starburst Data and a Puluminary member. He’s deeply involved in the Infrastructure as Code (IaC) space, having contributed to Ansible, been a core contributor to Apache jclouds, and has written many Terraform plugins. He leads infrastructure architecture at Starburst and originally introduced Pulumi to the company. Starburst provides a data lake analytics platform that’s powered by Trino - an open-source distributed SQL query engine designed for running fast analytic queries across large datasets in multiple data sources. At Starburst, Matt helped revamp and improve how the company manages its multi-cloud and cloud native infrastructure.

Watch Matt Stephenson’s full presentation with demos. 8:03 demo of CI/CD. 15:33 demo of infrastructure automation.

Technical and Business Problems to Solve

Starburst operates a managed Trino service that runs on Kubernetes clusters on multiple clouds. Their Kubernetes clusters needed frequent manual scaling based on workloads. Furthermore, upgrading Trino versions running on the clusters ran the risk of potential service disruptions for customers. The company had been using tools like Terraform for IaC, Argo CD for deploying Helm charts, Maven, Docker containers, and a Slack bot for infrastructure and deployments. Their CI/CD workflow was difficult to maintain because they needed to create wrappers for their previous IaC tooling, making it less approachable in CI/CD environments. The pipeline was tenuously bound together using GitHub Actions and a Slack bot, requiring specialized GHA runners, leading to a messy architecture.

Why Starburst Chose Pulumi

Recognizing opportunities for improvement, Matt brought in Pulumi and its Automation API. This unified Starburst’s software development and infrastructure management practices, especially when deploying and managing Trino clusters on Kubernetes. The top two reasons Matt chose Pulumi were:

Support for programming languages: Pulumi supports the main languages used by Starburst’s developers, Java and TypeScript. This enables all engineers to write infrastructure code and promotes a cohesive application and infrastructure codebase.
Pulumi Automation API: Automation API is a programmatic interface for the Pulumi CLI and Pulumi IaC engine, which enables Starburst engineers to build infrastructure automation into their applications and CI/CD processes, and orchestrate complex provisioning workflows.

Benefits of Using Pulumi

Adopting Pulumi and its Automation API provided several benefits that made Starburst’s engineers more efficient and productive, ultimately leading to faster and more reliable releases for Starburst’s customers. Matt summed up the benefits of Pulumi for Starburst’s engineers: “When I think about building an airplane, I think about precision and having the right tools for the job.” Pulumi, for Starburst, epitomized this precision and efficiency.

Increased deployment speeds by 3x for new customer releases: Starburst engineers used the Pulumi Automation API to triple the deployment speed of multi-cloud Kubernetes clusters with blue-green deployments that complete within minutes. They did this by orchestrating blue-green updates across their data plane running on the clusters, ensuring zero customer downtime. They built intricate integrations between Pulumi and internal APIs that enabled them to seamlessly transition customers to new clusters. Now they can update their customers’ runtimes, including VPCs and Kubernetes clusters, twice a week.

Unified cluster and infrastructure management: Rather than juggling multiple tools for managing Kubernetes and infrastructure, Starburst engineers use a single tool to manage over 50 Kubernetes clusters and deploy Helm charts into the clusters, spanning multiple regions. They also use Pulumi to manage SaaS platforms like Cloudflare, Confluent, and DataDog.

Consistent deployment lifecycles: Using Pulumi’s integration with GitHub Actions, Starburst engineers built a streamlined CI/CD process where infrastructure and application code builds within the same pipelines with automated test environments. They accomplished this using Pulumi Automation API to abstract the underlying runtime environments, circumventing the need for additional layers and wrappers.

Try Pulumi for Infrastructure as Code Automation

Sign up for a free account to try deploying infrastructure on any cloud with Pulumi Deployments, or register for an upcoming workshop to learn more about how Pulumi can help you ship cloud infrastructure faster and more safely.

Go to more case studies →

How a Bank Modernized Its Software Engineering With Infrastructure as Code Automation

George Huang — Thu, 03 Aug 2023 00:00:00 +0000

This blog post summarizes a presentation by Dennis Sauvé at PulumiUP 2023.

Washington Trust Bank, the largest independently-owned full-service commercial bank in the Northwest, has served personal, private, commercial and wealth management clients throughout the region since 1902. It has assets exceeding $11 billion and currently has 42 branches and offices in Idaho, Oregon, and Washington.

As an FDIC-governed financial institution, it is imperative for the bank to maintain secure, reliable, and compliant cloud resources to protect clients’ personal data. On the other hand, it also aimed to create more agile development teams as it modernized its software development and infrastructure. Dennis Sauvé, the bank’s first DevOps Engineer, recognized Infrastructure as Code (IaC) as the solution to these challenges.

Embracing an Infrastructure as Code approach would allow them to automate building and deploying their cloud infrastructure, eliminate infrastructure provisioning as a bottleneck, and empower developers to self-service infrastructure and increase productivity.

Background

The main challenge was managing Azure infrastructure in an efficient and secure way that kept pace with the bank’s rapid growth. Previously, they ran on-premise infrastructure and deployed with BAT and Powershell scripts. Now, they had migrated to Azure and wanted to increase development speed through CI/CD for cloud infrastructure.

In addition, cloud resources’ security and reliability were paramount. They needed a way for the infrastructure and information security teams to work together to implement security best practices for infrastructure, including how to prevent insecure resources from being provisioned.

The bank decided to adopt IaC to create reliable and rapid deployments while ensuring that its cloud infrastructure stayed compliant and secure.

Why Washington Trust Bank Chose Pulumi

The bank evaluated several options, including Terraform and Azure Bicep. However, Pulumi stood out due to its versatility, support for multiple clouds and native Azure support, and the ability to write IaC in the bank’s preferred language, TypeScript. The ease of using Pulumi with GitHub, its ability to be used in conjunction with Azure CLI, and its comprehensive support for CI/CD through GitHub Actions were also compelling factors.

Dennis praised Pulumi, saying, “Pulumi was exactly the Swiss Army Knife versatility we were looking for. We’ve only uncovered more of what we love about Pulumi as our relationship has evolved.”

Benefits of Using Pulumi

Pulumi helped Washington Trust Bank improve efficiency by maintaining its infrastructure in code, promoting the security of its infrastructure, and accelerating development speeds. It automated deployment of web services for its development teams, data management tools for its business insights team, and orchestrated its entire hub-and-spoke cloud architecture.

Improved Development Speed with Reusable Infrastructure Packages and CI/CD

Pulumi’s automation capabilities played a transformational role. By freeing the infrastructure team from handling manual deployments, it could focus on improving developer productivity. The team created reusable and repeatable infrastructure components built around strict IT security standards in the form of Frazure, an internal NPM package.

Frazure allows the bank’s developers to easily provision out-of-the-box resources for cloud services and covers the necessary Azure resource groups, vnets, and other resources. The package encapsulates best practices, such as how resource groups and their virtual networks can be created, including subnet IP spaces, peerings to and from the hub virtual network, user-defined routes and more.

To deploy, developers open a pull request which kicks off a CI/CD pipeline in GitHub Actions and provisions testing environments and tests. Once the PR is reviewed and approved, the changes are automatically deployed to staging and live environments. This automated process enabled the bank to accelerate the building, testing, and deployment of new applications, translating into rapid value delivery to their customers.

Safeguards for Infrastructure

Pulumi also gave the bank total confidence in being able to rapidly recover its cloud infrastructure in the case of a failure in Azure. The integration of Pulumi with GitHub Actions provided the team with a precise delta of changes to be made to the environment. Coupled with mandatory reviews from InfoSec and infrastructure, this significantly reduced the likelihood of deploying unapproved changes, thereby enhancing the reliability of their deployment processes.

Policy as Code Guardrails with Pulumi CrossGuard

Pulumi CrossGuard adds an extra layer of security and control and is used in conjunction with Azure Policies, which are used for auditing purposes. CrossGuard prevents the deployment of undesired, insecure, or expensive resources during the preview and deployment stage, thus preventing developers from even reaching Azure to provision resources. Custom error messages give developers context on why their deployment was not allowed.

Pulumi Cloud Policy Packs allow them to group and deploy many policies simultaneously. The Policy Packs prevent specified resources from being deployed into staging and live environments. For example, one policy requires all SQL databases to use TLS 1.2 by default and another ensures all storage buckets have public access disabled by default. These capabilities helped bolster the security of the bank’s cloud infrastructure.

Financial services cloud modernization with Pulumi

The DevOps and development teams at Washington Trust Bank rely on Pulumi as a key tool in their cloud modernization, while maintaining high security and quality standards. Pulumi played a vital role in streamlining their cloud infrastructure management, enabling their developers to be more productive, and improving the security and reliability of its cloud infrastructure.

Sign up for a free account to try deploying infrastructure on any cloud, or register for an upcoming workshop to learn more about how Pulumi can help you ship cloud infrastructure faster and more safely.

Go to more case studies

How Skai Migrated to Amazon Keyspaces with Pulumi

Danny Zalkind — Tue, 16 May 2023 00:00:00 +0000

Danny Zalkind is the Senior Director of Infrastructure Engineering for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

As Skai continues its journey towards fully migrating to the cloud using Pulumi, we’ve taken another large bite out of the migration pie, moving our most critical data to AWS on top of Amazon Keyspaces, an Apache Cassandra–compatible database service.

This blog post will dive into how we used Pulumi and AWS Lambda to perform the migration.

Offloading a piece of the pie to AWS

First, some background on what we do. Skai helps marketers work smarter and faster with a unified platform to reach and convert shoppers. An important service we provide to our clients is optimization of their digital marketing campaign efficiency. We store billions of events per day to enable clients to continuously improve the value they get from their ad spend.

Our on-premises setup included a 100-node Apache Cassandra cluster to be able to withstand the scale of billions of events, some stored for prolonged periods of time. Maintaining and scaling this setup was a high effort from our Data Infra teams. It was clear to us it was time to offload that piece of the pie to our cloud provider.

Why Amazon Keyspaces

When looking for a solution, we were looking for a service that would mark the following checkboxes:

Fully serverless to allow us focus on our business instead of server maintenance.
Scalability should be automatic and easy to perform.
High availability and multi-region built-in.
Cost effective pay-as-you-go and commitment-based model.
Extensive support and rich documentation.
Fully automatable using Pulumi, our main Infrastructure as Code framework.

Migration method

Let’s take a look at how we set up the migration. We decided to migrate the data using a dual-write method, writing new data to both our on-prem Cassandra cluster and Keyspaces. Any historical data needed could be fetched from Snowflake, our data lake solution, using our Apache Spark pipeline.

Finally, to make sure that the data is valid and we have no data discrepancies on the Keyspaces side, we added a Lambda function to periodically sample the data and compare it to the data in our on-prem cluster.

We made a conscious decision to decouple the application code from the provisioning of the infrastructure, letting Pulumi handle all resource creation and configuration, while the application team used our standard CI/CD pipeline to deliver their code to the service owning the Keyspaces data.

Let’s start provisioning

Keyspaces requires creating a logical entity called a keyspace, which groups related tables relevant for one or more applications. After that, we iterate over our config and create each table with the relevant columns and the preferred provisioning mode — either “on-demand” or “provisioned,” depending on our workload pattern. A provisioned workload in pre-allocated and thus significantly (x7) lower in hourly price, suitable mainly for use cases where the workload is quite predictable with no spikes.

Finally we populate AWS tags in order to be able track costs and performance metrics.

The following Python code illustrates the process:

import pulumi
import pulumi_aws as aws
from pulumi_komponents.account_provider.account_provider import get_account_provider

# Define resource tagging
def __populate_tags(aws_config: pulumi.Config, component_type: str, tags: dict):
 return {
 "Project": pulumi.get_project(),
 "Environment": aws_config.require("account"),
 "Type": component_type,
 **tags,
 }

def pulumi_program():
 aws_config = pulumi.Config("aws")
 keyspaces_config = pulumi.Config("keyspaces")
 tags = keyspaces_config.require_object("tags")
 provider = get_account_provider(aws_config)

 # Define keyspace
 keyspace = aws.keyspaces.Keyspace(
 resource_name=f'keyspaces-{keyspaces_config.require("name")}',
 name=keyspaces_config.require("name"),
 tags=__populate_tags(aws_config, "keyspace", tags),
 opts=pulumi.ResourceOptions(provider=provider, protect=True),
 )

 # Define and iterate over keyspace tables
 tables = keyspaces_config.require_object("tables")

 for table_name, table_ddl in tables.items():
 columns = []
 partition_keys = []
 clustering_keys = []
 static_columns = []
 for col in table_ddl.get("schema").get("columns"):
 name, type = col.popitem()
 columns.append(
 aws.keyspaces.TableSchemaDefinitionColumnArgs(name=name, type=type)
 )
 for par in table_ddl.get("schema").get("partition_keys"):
 partition_keys.append(
 aws.keyspaces.TableSchemaDefinitionPartitionKeyArgs(name=par)
 )
 for cl in table_ddl.get("schema").get("clustering_keys") or []:
 name, order_by = cl.popitem()
 clustering_keys.append(
 aws.keyspaces.TableSchemaDefinitionClusteringKeyArgs(
 name=name, order_by=order_by
 )
 )
 for s_col in table_ddl.get("schema").get("static_columns") or []:
 static_columns.append(
 aws.keyspaces.TableSchemaDefinitionStaticColumnArgs(name=s_col)
 )

 mode = table_ddl.get("throughput").get("mode")

 aws.keyspaces.Table(
 resource_name=f"keyspaces-table-{table_name}",
 keyspace_name=keyspace.name,
 table_name=table_name,
 schema_definition=aws.keyspaces.TableSchemaDefinitionArgs(
 columns=columns,
 partition_keys=partition_keys,
 clustering_keys=clustering_keys,
 static_columns=static_columns,
 ),

 # Set capacity mode and units for read and write per Table
 capacity_specification=aws.keyspaces.TableCapacitySpecificationArgs(
 throughput_mode=mode,
 read_capacity_units=table_ddl.get("throughput").get("rcu")
 if mode == "PROVISIONED"
 else None,
 write_capacity_units=table_ddl.get("throughput").get("wcu")
 if mode == "PROVISIONED"
 else None,
 ),

 # Set general table properties such as TTL and recovery method
 default_time_to_live=None,
 ttl=aws.keyspaces.TableTtlArgs(status="ENABLED"),
 point_in_time_recovery=aws.keyspaces.TablePointInTimeRecoveryArgs(
 status="ENABLED"
 ),

 # Populate AWS resource tagging
 tags=__populate_tags(
 aws_config,
 "table",
 {
 "Table-type": "unarchived"
 if table_name.endswith("_unarchived")
 else "standard",
 **tags,
 },
 ),
 opts=pulumi.ResourceOptions(
 parent=keyspace, provider=provider, protect=True
 ),
 )

Comparison Lambda

Next we define and create a Lambda function that samples and compares data between the data newly ingested into Keyspaces, our on-premise Cassandra cluster, and our data lake Snowflake.

We created an abstract class implementation of AWS Lambda to standardize our various Lambda implementations and keep the code readable. This class returns all needed arguments, configuration, layers, etc. Here is some Python code that shows our abstraction, named AwsFunction:

class AwsFunction(AbstractAwsFunction):
 def __init__(self, args: AwsFunctionArgs):
 super().__init__(resource_type="AwsLambda:Function", args=args)

 # Set Lambda zip artificate path
 def _create_function(self) -> aws.lambda_.Function:
 code_archive_path = f"/tmp/{self.args.name}.zip"
 with zipfile.ZipFile(code_archive_path, "w") as zip_obj:
 zip_sub_folders(
 root_source=self.args.functions_absolute_path,
 destination=code_archive_path,
 selected_sub_folders=[ZIP_COMMON_DIR, self.args.code_directory_name],
 zip_obj=zip_obj,
 )

 # Get iam role, kms keys and layers
 self.function_role = self._get_function_role()
 self.kms_key = self._create_kms()
 layers_arn, layers = self.__build_layers()

 # Return function outputs
 return aws.lambda_.Function(
 resource_name=generate_resource_name(self.args.name, "function"),
 name=self.args.function_name,
 role=self.function_role.arn,
 package_type=self.args.package_type,
 architectures=[self.args.architecture.lambda_value],
 handler=self.args.handler,
 memory_size=self.args.memory_size,
 reserved_concurrent_executions=self.args.concurrent_executions,
 runtime=self.args.runtime,
 code=pulumi.FileArchive(code_archive_path),
 timeout=self.args.timeout,
 layers=layers_arn,
 tags=self._populate_tags(component="lambda", resource_type="function"),
 kms_key_arn=self.kms_key.arn,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=self.args.environment_variables.to_dict()
 ),
 vpc_config=self.args.vpc_config,
 opts=ResourceOptions(
 provider=self.args.provider, depends_on=[self.kms_key] + layers
 ),
 )

Now we instantiate the abstract class as part of our “building block” component class, create the function, triggers and invocation:

class AwsLambda(ComponentResource):
 def __init__(self, args: AwsLambdaArgs):
 super().__init__(f"AwsLambda", args.lambda_name)

 # Define the function nased on the abstract class
 aws_function = (
 AwsFunction(args.aws_function_args)
 if args.aws_function_args.package_type == "Zip"
 else AwsFunctionContainer(args.aws_function_args)
 )

 # Add triggers
 triggers = {}
 for trigger, config in args.aws_function_args.lambda_triggers.items():
 triggers[trigger] = LAMBDA_TYPES_TO_DEF[trigger](
 config, aws_function.function, args.provider, args.tags
 )

 # Add async invocation config
 FunctionEventInvokeConfig(
 generate_resource_name(args.lambda_name, "event-invoke-config"),
 function_name=aws_function.function.name,
 destination_config=FunctionEventInvokeConfigDestinationConfigArgs(
 on_failure=FunctionEventInvokeConfigDestinationConfigOnFailureArgs(
 destination=args.aws_function_args.on_failure_arn,
 )
 if args.aws_function_args.on_failure_arn
 else None,
 on_success=FunctionEventInvokeConfigDestinationConfigOnSuccessArgs(
 destination=args.aws_function_args.on_success_arn,
 )
 if args.aws_function_args.on_success_arn
 else None,
 ),
 qualifier=args.aws_function_args.qualifier,
 maximum_event_age_in_seconds=args.aws_function_args.maximum_event_age_in_seconds,
 maximum_retry_attempts=args.aws_function_args.maximum_retry_attempts,
 opts=ResourceOptions(provider=args.provider),
 )

Now we’re ready to call our class to create our comparison Lambda:

from pulumi_komponents.awslambda import AwsLambda, AwsLambdaArgs
from pulumi import Config

config = Config()

AwsLambda(AwsLambdaArgs(config))

Note: This only a partial code example, meant to demonstrate the way we use object-oriented paradigms to create reusable building-blocks of code to be consumed by various platform and R&D teams to provision AWS resources.

Putting it all together

Once we have our Keyspaces setup in place and the confidence that our dual-write process is writing the data correctly, we are now able to wait enough time to accumulate the data on the AWS side, using a feature toggle in our application to switch our backend systems to work with AWS Keyspaces.

Outcome

Following the successful migration, we have gained 50% reduction in backend process runtime, increased event processing throughput by 400% and freed up 30% of our team’s time to do other things, instead of maintaining cluster infrastructure.

I hope this post provides readers with useful techniques and ideas for migrating large-scale databases to the public cloud.

Managing NOAA Open Data across Multiple Clouds with Pulumi

Denis Willett, PhD — Tue, 08 Nov 2022 21:01:21 +0000

Denis Willett is a software engineer at the North Carolina Institute of Climate Studies who works on the NOAA Open Data Dissemination Program. His work focuses on leveraging cloud technologies for the development of data processing and machine learning pipelines. Denis did his PhD in Entomology and Nematology at University of Florida and his undergraduate and masters work in Earth Systems at Stanford University. You can read his full bio here.

NOAA Open Data Dissemination (NODD) makes environmental data freely and publicly accessible across Amazon Web Services (AWS), Microsoft Azure (Azure), and Google Cloud Platform (GCP). These data include near real-time satellite imagery, weather models, radar feeds, drought information, ocean databases, and a suite of climate data records among many others. This program supports more than 220 datasets and over 24PB of open data. Since its inception, the program has been growing rapidly, almost doubling in size over the past year.

Availability of the petabyte scale environmental data on the cloud is only one aspect of NODD. The program places a premium of performant data delivery and access. NODD data pipelines provide resilient and low, in many cases near real-time, latencies to NOAA’s core data offerings. In addition, NODD provides performant, parallel access to these data supporting billions of requests per day and accessions of individual datasets exceeding 1PB in a single day.

To support this program, the North Carolina Institute for Climate Studies (NCICS) through the Cooperative Institute for Satellite Earth System Studies (CISESS) engineers infrastructure on all three major public clouds ranging from object stores to messaging platforms, data ingest and transformation pipelines, and monitoring systems. These systems are deployed and managed by a very small team that, by necessity, must be agile, resilient and facilitate low latency data transfers at petabyte scale. Given the scale and diversity of resources managed through this initiative, we began looking for tools to facilitate multi-cloud management using infrastructure as code (IAC).

Before Pulumi

Prior to beginning to transition to Pulumi, our team built infrastructure primarily using the Command Line Interface (CLIs) and consoles of each cloud service provider. While this worked well, it required extensive Cloud Service Partner (CSP) specific training to be able to not only navigate the nuances of each provider’s offerings, but also the unique interface to deploy resources. Changing and updating infrastructure in this manner resulted in long development cycles where upgrades were mostly manual and composed of repetitive tasks.

Because of the manual nature of these deployments across three different cloud service providers, infrastructure began to sprawl. A clear snapshot of what resources were deployed where became difficult to compile and management of legacy and deprecated systems depended on manual intervention for clean-up. Infrastructure also began to drift. What started as uniform deployment of resources across multiple cloud platforms morphed as manual updates to each instance resulted in unique environments that were hard to replicate if they had to be deleted and redeployed.

As an example, our program relies on Apache NiFi deployments on AWS, GCP, and Azure to coordinate and manage petabyte scale data transfers while managing data provenance. Drift in these deployments makes it especially difficult to redeploy identical configurations (even though we were using Ansible and Containerization) if they go down.

An additional consequence of this approach is knowledge silos. The team member who deployed a group of resources on AWS had trouble (even with extensive documentation) communicating reusable architectures to another team member who was working on similar situations in Azure. While there are obvious nuances relevant to each cloud environment, our need to replicate similar infrastructure in a multi-cloud environment at scale necessitated extensive detailed communication in a team environment. Questions like: “What is the ARN of the most recent version of this service?” and “What queues do we have on GCP?” required communication and time taken by the individual responsible to answer those questions.

How and why we chose Pulumi

As our team grew and our platform scaled (more than doubling in storage size over the past year by more than 10PB), we began to explore tooling to help us better navigate and manage our multi-cloud and multi-account environment. The nature of the NOAA Open Data Dissemination program agreements with each cloud service provider (CSP) meant that we had multiple accounts with different permissions, funding, and configuration on each CSP, resulting in flexible options.

We began looking at infrastructure as code (IAC) solutions and initially considered many of the cloud-specific options: CloudFormation on AWS, Deployment Manager on Google, and Resource Manager on Azure. While these were nice for automation, we quickly discarded these as options because each had an individual learning curve for that environment; they did not translate well in a multi-cloud environment. These tools also did not have the control features of a full-fledged programming language; looping and if-then statements were challenging to implement.

We also considered other Infrastructure as Code providers. While these solutions would meet a lot of our needs, and one of our team members had extensive experience with one of these tools, the ability to use a common tool seamlessly in a multi-cloud environment was attractive. However, the learning curve associated with a domain specific language was cause for reticence. In addition, we really wanted to be able to use the features of a full-fledged programming language.

After experimenting with a few IAC options, our research institute began using Pulumi in late 2021 and immediately began implementing it to expand our infrastructure.

Benefits of using Pulumi for multi-cloud infrastructure as code

Pulumi had all the benefits we were looking for in a multi-cloud infrastructure as code platform. In terms of automation, manual checklists could be converted to code reducing both deployment time and human errors in deployment. This automation with infrastructure as code (and accompanying state management) allowed us to adopt an immutable infrastructure paradigm where old resources are replaced with new updated infrastructure. While these benefits are accessible via other IAC tools, we particularly appreciated the Pulumi interface to be able to quickly access and inspect infrastructure deployments.

We saw immediate wins for our teams in the following areas:

Drift was eliminated. Single source of truth, version controlled (and tagged) code documented exactly what was deployed where. When updates were needed, the code was updated and redeployed without manually tweaking individual resources through the console.
Management of legacy and deprecated resources were obviated. With IAC managed deployment that would automatically clean up old resources, we saved many future hours cleaning up unused cloud resources.
Team collaboration was enhanced. With a single source of truth and a visual snapshot of resources and deployments, the number of ‘What was the ARN of that resource again?” type questions dropped considerably. Team members could simply look at the deployment in one location and see all of the resources across all three major clouds. This type of friction-reduction really enhances team productivity.

In addition to these IAC benefits, we saw a number of key advantages in implementing Pulumi. First and foremost was the ability to use full-featured programming languages to deploy IAC solutions in a multi-cloud environment. This not only allowed us to fully take advantage of programming control structures (e.g., need 100 different rules on AWS? Now you do not have to manually script each one, just put it in a loop), but also flattened the learning curve for adoption. Our team already used Python extensively in our day-to-day work. Adopting Pulumi was almost as simple as importing a package. Accessing Pulumi through Python made it much easier to embrace.

This accessibility through Python also meant that learning how to create, deploy, and share reusable templates was trivial as all team members already knew how to build, share, and import modules in Python. The ability to rapidly share these templates accelerated our development and allowed for standardization around best practices. For example, we use serverless capabilities on all three CSPs. Once we had a template for deploying our function as a service (FAAS) code on one platform, we did not have to reinvent the wheel or remember to go through a checklist; it was as simple as importing our internal FAAS module and pointing it to the correct code to deploy.

Next steps

Pulumi has become an automation and collaboration tool that is accelerating our ability to provide petabyte scale data to the public in a multi-cloud environment. As we continue to scale our program, Pulumi is allowing us to leverage a small team to manage increasingly larger, more complex deployments in a dynamic environment. In addition, it is rapidly becoming a key player in a cross-cloud fabric that we are building to facilitate working across multiple platforms. By providing a similar, intuitive interface through our language of choice, Python, it is allowing us to more quickly adapt and reproducibly deploy infrastructure across AWS, Azure, and GCP.

Please reach out to us with your thoughts, reactions, and comments at NODD@NOAA.GOV.

How Elkjøp Nordic enables self-service infrastructure for developers

George Huang — Fri, 01 Jul 2022 00:00:00 +0000

At PulumiUP 2022, Tomas Jansson, software architect at Elkjøp Nordic, gave a presentation on how to enable developers to self-service infrastructure by using Pulumi’s Automation API.

Elkjøp Nordic is the leading consumer electronics retailer in the Nordics. The company sells consumer electronics, mobile phones, computers, white goods, domestic appliances, and services linked to these products both directly to consumers and to businesses. It is an omnichannel retailer and serves customers both online and through more than 400 stores. Elkjøp has retail outlets in Norway, Sweden, Finland and Denmark, and franchise operations in Greenland and Faroe Islands.

To support the company’s modernization strategy, Tomas developed a self-service infrastructure platform that would enable any developer to easily provision cloud infrastructure that uses approved resources and security practices. The goal was to empower teams at Elkjøp to increase their development velocity by using the cloud, while maintaining guardrails that uphold security and compliance.

Watch the full presentation below or read on to learn the highlights of his presentation:

The Problem: Balancing Agility and Governance

To achieve his goals, Tomas knew he had to solve the problem of balancing freedom with control. On the one hand, he wanted to foster the creativity of his developers and encourage innovation. He wanted his team to feel they were trusted and had the room to create the best solutions they could. On the other hand, he knew he had to maintain some control. Elkjøp needed to keep its applications secure. It also needed good governance so that teams can keep track of who is making changes. Tomas also wanted to limit the impact of deploying untested code. Code that was still being developed needed to be confined to the developer’s own environment. Finding the right balance between independence and control was paramount. Tomas believed this would result in great software and in developers who would enjoy their work.

The Solution: A Self-Service Infrastructure Platform

Tomas’ solution was to build a self-service infrastructure platform. Self-service infrastructure makes it possible for developers to easily access the infrastructure they need while limiting access to resources they don’t need. In effect, self-service infrastructure is a walled garden. Within the garden, developers have all the freedom they need. However, nothing they do affects what’s outside the wall. Code that causes unintended changes only affects them and no one else.

Tomas built an application that provides a “walled garden” of self-service infrastructure. Developers can create the environments they need, including an Azure resource group and a Kubernetes namespace, all through a graphical user interface. Developers can define whether the environment is for development, test or production. They can also assign values, such as which team members should have admin privileges to the associated GitHub repositories or who can contribute. From there, the platform handles provisioning and configuring all of these resources.

Here is an example that shows how a developer can create a repository simply by selecting a few parameters from a drop-down list.

Tomas determined that the best way to implement self-service infrastructure was with Pulumi and its Automation API. By using Pulumi, he could use standard programming languages to define infrastructure while using familiar software tools like IDEs and test frameworks. Pulumi also natively supports Azure and Kubernetes with 100% coverage of each provider’s APIs and same-day access to new features and updates. Finally, the Pulumi Automation API would enable him to program infrastructure automation directly into his application.

The Pulumi Automation API is a programmatic interface for running Pulumi programs without the Pulumi CLI. Automation API encapsulates the functionality of the CLI (pulumi up, pulumi preview, pulumi destroy, pulumi stack init, etc.) but with more flexibility since infrastructure operations can be controlled programmatically.

How the self-service infrastructure platform was built

Here is the workflow for the Elkjøp self-service infrastructure platform.

The self-service platform is a web application that runs the Pulumi Automation API on the server side. The application uses Pulumi to provision and configure every environment resource such as Azure resource groups, Kubernetes namespaces, service principals, and Kubernetes service accounts. Each environment comes with a Git repository that’s used by developers to manage their deployments. The Git repositories are the locus of each environment because they store everything needed to deploy changes such as infrastructure code and environment credentials.

Using the Pulumi Automation API, the application creates an Azure resource group, a team in both Azure AD and GitHub, and a new GitHub repository. The application then grants administrative access to the GitHub repo to members of the AD group. In addition to creating a Git repository for each environment, the application uses stack references for each Azure environment to inject the associated credentials into the GitHub Secret Store. These credentials are scoped to each GitHub environment so that developers can only work in their own environment. If a team has multiple environments, each environment has its own credentials so developers can’t, for example, deploy from a test environment to a production environment. Secrets are managed through the Pulumi Service.

Every new repository comes with Pulumi programs that developers can use to deploy their application. GitHub Actions initiate the deployment. When an action triggers, Pulumi reads the credentials that were generated by the self-service application, allowing Pulumi to deploy new Kubernetes or Azure resources to the environment.

Watch the demo

To illustrate how Elkjøp’s self-service platform works, Tomas presented a simplified demo application that’s based on Elkjøp’s production platform. The code for the application is available at https://github.com/mastoj/pulumi-automation-demo.

The demo is a TypeScript application that includes an API that provisions Azure and GitHub resources. Pulumi Automation API code is used to deploy new stacks by calling handler code, which is a Pulumi program that defines resources and outputs. The handler code can create Azure resource groups or GitHub repositories, and stack references are used so that outputs generated by the Azure resources can be stored in GitHub (e.g., resource group credentials).

Ready to dive into how this works in detail? Watch the demo in the video below starting around 10:20.

Next steps

After watching the demo of a self-service infrastructure platform, you can try it out for yourself by checking out the demo code. You can also learn more about Pulumi Automation API in our documentation.

If you’re new to Pulumi, then follow our getting started guide.

Cloud engineering fuels the next chapter of startup innovation

George Huang — Thu, 07 Oct 2021 00:00:00 +0000

The story of how the cloud fuels startup innovation seems never ending. In the beginning, AWS birthed cloud computing with its first service, SQS, in 2004 and quickly released several additional services (like S3, EC2, and SimpleDB). From this innovation, startups flourished because they were able to build, experiment, and grow faster than before at much lower cost. Airbnb, Netflix, Zynga, and many more were born, and the rest is history.

Today, a new generation of startups is flourishing because of the cloud, but this time with modern cloud architectures that are distributed, API-driven, and more resilient and scalable than ever. Today’s startups have to get to market even faster and rapidly innovate in order to delight customers and carve out market share. Most startups understand the benefits of adopting the modern cloud to help them achieve this goal. However, their ability to reap these benefits for competitive advantage depends on how well they can harness the modern cloud.

Within the cloud engineering community, we see several common patterns for harnessing the modern cloud. Some startups have teams of full-stack developers who need to deploy cloud infrastructure and applications safely and at high velocity. Others might have a few infrastructure or platform engineers who need to enable other developers to use cloud infrastructure easily on a self-serve basis. Many of these teams started off using domain-specific languages (DSLs) to manage infrastructure as code and quickly found that these languages were the limiting factor in achieving faster velocity. DSLs are cumbersome to use and don’t support the logic and expressiveness needed to build and manage modern architectures that are more complex in nature. DSLs are also a barrier to entry to most developers.

In contrast, cloud engineering makes cloud infrastructure more accessible and easier to use for everyone with tried and true software engineering practices and tools. It embraces the reality that all software is now cloud software. A core practice is using a commonly used language like TypeScript, Python, Go, or C#. That’s one reason why more and more startups are choosing Pulumi to help them achieve competitive advantage through cloud infrastructure. It also doesn’t hurt that Pulumi was rated the “Top Paying Technology” by developers in 2021.

Cloud engineering is a central character in the next chapter of how the modern cloud fuels startup innovation. Read on for three mini-stories of cloud engineering in action with startups.

Panther Labs

Panther Labs helps modern security teams build world-class detection and response pipelines using code and automation, developer-friendly workflows, and big data primitives. Its Platform Team is responsible for a large, complex serverless architecture on AWS. Because of the limitations of its legacy Infrastructure-as-Code (IaC) tool, the team was unable to manage and scale its cloud infrastructure with the speed and automation that the company needed to support its fast-growing business. After comparing different alternatives, Panther Labs decided to migrate to the Pulumi Cloud Engineering Platform. Pulumi increased the company’s deployment speeds by up to 10x, reduced the size of its infrastructure codebase by >50%, and enabled its developers to adopt cloud engineering best practices to deliver its cloud applications faster and more reliably.

Results

Infrastructure teardowns were twice as fast, fresh deployments were four times as fast, and simple changes were more than ten times as fast.
The infrastructure codebase was reduced by more than 50%.
Developers could use a programming language and existing software development practices that they already knew.
The ability to create reusable components made it easy to replicate and share components that always followed cloud engineering best practices.
Central visibility across hundreds of single-tenant customer deployments, including a history of when changes were made and what changed for each account.

Read the full case study→

WhyLabs

WhyLabs helps organizations run their AI applications with certainty by monitoring them for problems such as data drift and model degradation. Founded by alumni of AWS, the company began building their platform on AWS and used AWS-provided tools to manage their infrastructure. However, the WhyLabs team needed a platform that could handle the complexity of their modern cloud architectures and would set them up for success when they’re ready to adopt multi-cloud deployments. The team also wanted a platform that integrated with the modern development workflows used by WhyLabs’ developers, who have a software engineering-driven culture of tight collaboration and shared responsibility for infrastructure. By migrating to the Pulumi Cloud Engineering Platform, WhyLabs engineers can continuously and reliably ship new features faster than before for improved time-to-market. Pulumi enabled WhyLabs to adopt cloud engineering best practices out-of-the-box, including building infrastructure as code with general-purpose languages like TypeScript, versioning and reviewing infrastructure code through a code review process, and delivering infrastructure through its existing GitLab CI/CD pipeline using Pulumi’s integrations.

Results

Pulumi enabled WhyLabs engineers to continuously and reliably ship new features faster than before for improved time-to-market.
Pulumi enabled WhyLabs to adopt standard programming languages (like TypeScript) to build infrastructure as code. It also allows them to use software engineering best practices like reusability and unit testing.
Pulumi lowered the barrier to entry for developers to start working with infrastructure. A standard programming language means that every developer can quickly become involved in creating and maintaining the WhyLabs infrastructure.
Pulumi enables WhyLabs to shift infrastructure left because testing and security controls are automated with Pulumi and GitLab integration. Peer reviews and accountability are built into the GitLab CI/CD pipeline.
Pulumi’s integration with GitLab enables WhyLabs to deliver infrastructure through CI/CD pipelines just like with application code, which increases iteration frequency and accelerates time to market.
Pulumi’s support for multiple cloud providers will help WhyLabs expand their platform in the future to meet customers’ needs.

Read the full case study→

Fauna

Fauna provides a flexible, developer-friendly, transactional database delivered as a secure and scalable cloud API with native GraphQL. It needed to provide a way for customers to use its databases while meeting data residency requirements, such as complying with GDPR or FIPS. This meant allowing customers to store data in specific geographies while maintaining low latency and high availability. To build this capability, Fauna’s engineers needed a repeatable, scalable, and efficient way to build, deploy, and manage databases on behalf of its customers. Every database is backed by a cluster of nodes across multiple regions and cloud providers. After evaluating CloudFormation and Terraform, Fauna found that these tools’ domain-specific languages and features would be insufficient for the project. Fauna chose Pulumi to build, deploy, and manage its multi-cloud infrastructure because Pulumi enabled cloud engineering best practices that met its requirements. This included the ability to use a familiar and powerful programming language like Python, integration with existing software development workflows and tools, and software engineering automation.

Results

Delivered its new multi-region, multi-cloud database feature by using familiar and existing software tools and practices that were enabled by Pulumi.
Built and managed its complex cloud architecture more efficiently by using familiar languages like Python.
Increased the reliability and quality of infrastructure using existing code review processes and running unit tests.
Controlled access to its infrastructure using Pulumi’s SSO integration with its identity provider service.

Read Fauna’s blog post→

Summary

Cloud engineering is enabling startups to build, deploy, and manage modern cloud infrastructure and applications faster and with more confidence. By applying standard software engineering languages, practices and tools to cloud infrastructure, startups can harness modern cloud architectures like containers or serverless to their full potential. To illustrate this, we shared three case studies of startups practicing cloud engineering. Pulumi’s Cloud Engineering Platform was built to enable any organization to adopt cloud engineering best practices out-of-the-box. If you’re starting your cloud engineering journey, try Pulumi for free today with this Getting Started guide.

Cloud Engineering on the Rise

George Huang — Tue, 03 Aug 2021 00:00:00 +0000

One of the most fulfilling aspects of working at Pulumi is learning how customers and the community practice cloud engineering in their teams. It’s exciting to see how they use cloud engineering and Pulumi to implement best practices that enable leveraging the cloud to accelerate innovation and enable better business outcomes.

Pulumi is born from the experiences and needs of teams practicing cloud engineering every day. When we announced the Pulumi Cloud Engineering Platform in April, CEO & Founder Joe Duffy talked about bringing cloud engineering to everyone. Over the past year we have seen significant growth in cloud engineering and usage of Pulumi across companies of all industries and sizes, and spanning a diverse spectrum of teams and engineering disciplines. We’re also seeing growing numbers of job postings with “cloud engineer” in the title or which have Pulumi as a requirement or desired skill set.

Recently, we published several case studies about how teams are applying cloud engineering best practices. Cloud engineers apply standard software engineering practices and tools uniformly across infrastructure management, application development, and security to tame the complexity of delivering and managing modern cloud applications. We’ve published on our website and in this blog some of the key cloud engineering best practices that we see broadly across the community, and encourage you to read further to see three stories of cloud engineering in action.

These three case studies highlight a small sampling of the many types of companies and teams representing cloud engineers. Some are platform teams at large organizations, like Atlassian, who are responsible for empowering the rest of their developer teams. We also see DevOps teams using cloud engineering, such as at Skai and SANS Institute. There are other types of teams, like full-stack engineers or SREs, which we will share in future posts. All of these teams share a software engineering mindset and make a leveraged impact on the business by increasing speed, automation, reliability, and agility through cloud infrastructure innovation.

Atlassian’s Bitbucket DevSpeed team is responsible for improving developer productivity through better workflows and tooling. It used Pulumi to make it easier and faster for its developers to use cloud infrastructure and reduced developers’ time spent on maintenance by 50%. DevSpeed originally needed a way to expand regional support for its deployment pipeline, which deploys instances on AWS for more than one hundred team members around the world. The team also wanted to build a self-service platform that provisions cloud infrastructure for Bitbucket’s developers, without having to learn a complex domain specific language (DSL). They moved from a legacy DSL-based infrastructure as code tool to the Pulumi Cloud Engineering Platform, which let them define and deploy infrastructure in general-purpose languages that Bitbucket developers already used, such as Python. Using a familiar language like Python also made it easy for them to add cross-regional support to their deployment pipeline. They also built a self-service dashboard that allows any developer to deploy and configure private Bitbucket instances for feature development.

Atlassian’s Key Practices

Builds infrastructure as code with Python.
Makes infrastructure code readable and accessible to all developers in the organization by using a common language like Python.
Created a self-service platform that deploys and configures private Bitbucket instances for any Bitbucket developer. Underneath, everything runs through a CI/CD pipeline with Pulumi, Bitbucket Pipelines, and Bamboo.
Manages and deploys updates to all developer instances in the organization using features like the Pulumi Automation API.

Atlassian’s Results

Allowed developers to use general-purpose programming languages like Python and familiar software tools to deliver and manage infrastructure.
Enabled Bitbucket developers to easily provision approved cloud infrastructure using a self-service dashboard built with Pulumi.
Reduced the time Bitbucket developers spent maintaining their instances from 8 hours per week to fewer than four.
Reduced the size and complexity of its infrastructure code compared to its previous infrastructure tool while increasing the code’s clarity.

Read the full case study→

SANS Institute is the global leader in cybersecurity training. Using Pulumi, the organization adopted cloud engineering practices that enabled it to reduce deployment times by 70%. Its DevOps team needed a more consistent and reliable way to build, deploy, and manage the SANS virtual training environments used in labs. It had previously used legacy infrastructure tools which require inflexible DSLs that aren’t suited to modern cloud applications. The limitations of the DSLs meant the team needed to use AWS Lambda functions and Bash scripts to fill in gaps. The team needed a cleaner solution built for the modern cloud that would be more scalable and easier to manage. By adopting Pulumi, SANS increased delivery speed, quality, and consistency using cloud engineering practices such as building infrastructure as code with TypeScript and software tools, and deploying cloud applications with Git and automated delivery pipelines. It plans to use Pulumi to help other SANS business units modernize their practices and tools.

SANS Institute’s Key Practices

Builds infrastructure as code with TypeScript.
Deploys infrastructure through a GitOps and CI/CD process while managing its Git branches using a GitFlow pattern.
Production deployments can be initiated through an API call through a self-service platform that deploys an application instance.
Manages infrastructure configuration through Policy as Code which checks for compliance.

SANS Institute’s Results

Reduced deployment times for game servers by up to 70%.
Simplified their deployment process by eliminating the need to “glue together” several infrastructure as code tools, scripts, and functions. This let them remove a manual ticketing step.
Enabled developers to use familiar programming languages and tools, which also helped make their code cleaner and - more uniform, resulting in streamlined pipelines.
Helped to identify opportunities in other SANS departments where Pulumi can streamline operations.

Read the full case study→

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers. Skai’s microservices run on AWS, but its core monolith service was hosted in a data center running private cloud infrastructure. Skai’s DevOps group needed to migrate the service to the cloud because demand increased. Skai’s DevOps team decided to transition from using its legacy infrastructure tool because its domain-specific language (DSL) wouldn’t be suited to the demands of the newly migrated modern cloud architectures. Instead, Skai sought a solution that enabled it to use familiar languages and software engineering best practices to manage its infrastructure. That’s why Skai chose the Pulumi Cloud Engineering Platform to build, deploy, and manage its infrastructure with Python.

Skai’s Key Practices

Builds infrastructure as code in Python.
Leverages reusable infrastructure components.
Multiple teams collaborate on cloud infrastructure, including an architect, application development teams, and a core platform/DevOps team.
Teams use a “loosely-coupled, tightly-aligned” approach to create component classes for different infrastructure elements, with each team working on a different element.
Uses Python code to integrate infrastructure processes with internal metadata and IT systems.

Skai’s Results

Migrated a core monolith service with hundreds of terabytes of data from a data center to AWS.
Refactored the service’s infrastructure from being managed with a legacy infrastructure configuration tool to Pulumi, using Python as the language of choice.
Created modular infrastructure components that enable different teams to collaborate and work on different parts of the infrastructure in parallel.
Streamlined the registration and deregistration process for new application instances by integrating Pulumi with its internal IT systems.

Read the full case study→

Summary

Cloud engineers are a fast-growing community made of tens of thousands of members and thousands of companies. They apply standard software engineering practices and tools uniformly across infrastructure management, application development, and security to tame the complexity of delivering and managing modern cloud applications. To illustrate cloud engineering in action, we shared three case studies of teams using cloud engineering in their organizations. Pulumi’s Cloud Engineering Platform was built to enable any organization to adopt cloud engineering best practices out-of-the-box. If you’re starting your cloud engineering journey, try Pulumi for free today with this Getting Started guide.

How Webiny Built a Serverless Application Framework

Adrian Smijulj — Thu, 03 Jun 2021 00:00:00 +0000

Building an open-source framework for building serverless applications has many challenges, one of which is deploying cloud infrastructure resources. In this article, learn how Webiny uses Pulumi to enable its users to easily deploy and develop applications built on top of serverless cloud technologies.

What is Webiny?

Webiny is an open-source framework for building serverless applications, completely written in TypeScript.

In other words, it is a framework that enables developers to architect, build, and deploy solutions on top of the serverless cloud infrastructure, like AWS Lambda, Amazon DynamoDB, Amazon S3, and others. Some examples of what you can build with Webiny today are:

full-stack applications
websites
REST and GraphQL APIs
microservices

With the framework, Webiny also built a content management system called the Webiny Serverless CMS. It is free, open-source, and by default, included with every new Webiny project.

Finally, it’s worth mentioning that, at the moment, Webiny exclusively works with AWS. But, bringing Webiny to other cloud providers like Microsoft Azure and Google Cloud Platform (GCP) is definitely something we’ll be looking at in the future.

Building the Framework - The Challenge

Building a serverless framework certainly has its challenges. One of which is the deployment of cloud infrastructure, which, in the world of serverless, is one of the fundamental operations developers need to do, even while the application is still in development.

Before the version 5 release, Webiny relied on an infrastructure provisioning technology called Serverless Components (not to be confused with Serverless Framework). Using a concept of components, it is a system that enables developers to deploy ready-made blocks of cloud infrastructure resources configured for a specific use-case. Some examples of components are Website, REST API, a Full Stack Application, and more.

At the time, it seemed that the overall developer experience and the simplicity of the Serverless Components product was the right choice for our users. So, we decided to go with it.

But, over time, we noticed several things that, unfortunately, made us question our decision. And if I were to choose the top four, I’d mention the following.

1. Hard to Customize

While the idea around components for different use-cases certainly sounded interesting, ultimately, it was not ideal for Webiny. We frequently received questions regarding further component configuration and customization, which was not easy to perform.

“How do I configure a different database?”, “How do I set a VPC?”, “How do I set up a specific configuration parameter for my S3 bucket?” were just some of the questions we received. The answer almost always included us having to implement support for a particular feature. To some degree, this worked, but it was really us patching the holes, which we knew would only get us so far.

2. Using YAML Instead of Code

At Webiny, we believe cloud engineering and infrastructure-as-code is the future, so, naturally, we wanted our users to use familiar code (ideally TypeScript) and development tools to define their cloud infrastructure. Within Serverless Components, the components are configured via YAML files, of which we were never really big fans, simply because of the fact that writing code instead of configurations gives more flexibility to developers.

3. Vendor Lock-In

Vendor lock-in was introduced at a later stage of the Serverless Components product development. Essentially, to deploy a component, the user is now forced to use a proprietary service that comes with it. And while, among other things, the service enables much faster deployments, from our perspective, we saw this as an additional point of friction for our users. Ideally, users should be able to set up Webiny with only an AWS account.

4. Lack of Support for Other Cloud Providers

At the time, Serverless Components’ support for different cloud providers just wasn’t there. This would be a major roadblock for us as we get to multi-cloud integration.

Discovering Pulumi

Once we realized Serverless Components wasn’t an ideal solution for Webiny, we started looking for an alternative.

The key features that we were looking for were the following:

open-source
use code (preferably TypeScript) to create cloud infrastructure resources
cloud infrastructure code should be flexible, meaning, users should be able to make adjustments to it
support for multiple cloud providers is a must
no vendor lock-in

So, the search began, and soon enough, we discovered Pulumi. And as it turned out, it not only checked all the boxes we had initially listed but much more:

different options when it comes to storing cloud infrastructure state files: a managed SaaS (pulumi.com) with a console and self-hosted, (for example Amazon S3)
ability to deploy cloud infrastructure into multiple environments using its concept of stacks
advanced features like Policy as Code and watch mode
great documentation
a vibrant community of developers and a responsive team behind the product

I probably forgot to mention a thing or two here, but most certainly, at this point, we were ready to make a move and start experimenting with Pulumi.

Integrating Pulumi with Webiny

For us, the integration of Pulumi with Webiny consisted of three steps:

figure out how to integrate Pulumi’s programming model with Webiny
integrate Pulumi CLI into Webiny CLI
figure out the optimal way of handling cloud infrastructure state files

Let’s see how we tackled each of these three steps.

Integrating Pulumi’s Programming Model With Webiny

In terms of project organization, every Webiny project consists of two key concepts: packages and project applications (or just applications).

Packages are just regular NPM packages, or in other words, folders with a package.json manifest file and some code. On the other hand, project applications are higher-level organizational units formed from one or more packages that form applications, as the name itself suggests. Applications consist of both application code and cloud infrastructure needed to run them.

A single Webiny project can contain multiple project applications, where each one has its own independent set of necessary cloud infrastructure resources that need to be deployed.

As an example, a default Webiny project includes three project applications:

API - essentially, represents your GraphQL HTTP API
Admin Area - the Admin Area (React) application
Website - the public website, a (React) application with static site generation (SSG) in the cloud

If we were to compare these by the complexity of the necessary cloud infrastructure to deploy, the Admin Area is the simplest. It only relies on a single Amazon S3 bucket and an Amazon Cloudfront distribution. On the other hand, the API project application is the most complex one as it needs to deploy multiple AWS Lambda functions, Amazon DynamoDB tables, Amazon S3 buckets, and more.

Ultimately, we decided that each project application should be a Pulumi project. With this approach, we gave developers the ability to both define and deploy respective cloud infrastructures independently. And, with the concept of stacks, they are also able to deploy them into multiple environments, which we’ll show in a moment.

Pulumi CLI and Webiny CLI

Once we understood how to use Pulumi concepts with Webiny’s project organization, the next step was integrating the Pulumi CLI with the Webiny CLI.

And although the Pulumi CLI is great, we still wanted to keep it super simple for the user and make the overall developer experience as straightforward and unified as possible. For starters, we didn’t want our users to install the Pulumi CLI manually. We wanted it to happen automatically.

We’ve created our version of the Pulumi SDK, which lets us use the Pulumi CLI more programmatically. It also enables us to make the Pulumi CLI download experience as smooth as possible. Essentially, whenever a user runs a deployment-related command, all of the necessary Pulumi CLI binaries and plugins are downloaded and stored inside the project’s node_modules folder.

Although we could’ve saved us some time by using Pulumi’s Automation API (instead of creating the mentioned Pulumi SDK), at the time, the Automation API was still in preview and not generally available. And since the setup we already had was working well, we decided to keep it and hopefully revisit the Automation API integration in the future.

Once we had that in place, we were ready for the next step, exposing a couple of fundamental deployment-related commands via the Webiny CLI. The following examples show us some of the commands users can use:

# Deploys a project application (internally runs "pulumi up")
webiny deploy {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

# Previews a project application deployment (internally runs "pulumi preview")
webiny deploy {PROJECT_APPLICATION_FOLDER} --env dev --{ENVIRONMENT}

# Destroys cloud infrastructure resources previously deployed within
# the specified project application (internally runs "pulumi destroy")
webiny destroy {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

# Starts watching specified project application and continuously deploys
# changes to the cloud (internally runs "pulumi watch")
webiny watch {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

Note the --env argument appended to each command. With it and Pulumi’s concept of stacks, users can easily deploy their project applications into multiple environments.

Cloud Infrastructure State Files

The last piece of the puzzle was storing cloud infrastructure state files. Here we went with the following approach.

For local development, users’ cloud infrastructure state files are stored locally within their Webiny project using the Local Filesystem Backend, which we’ve seen worked great for developers.

On the other hand, for ephemeral environments spawned in CI/CD or long-lived environments like staging or production, through our documentation, we advise our users to use centralized and remote storage by using backends like Amazon S3 and even Pulumi Cloud (pulumi.com). Both backends have their pros and cons, and we let the users choose the one they want to use.

Show Me the Code

Let’s take a look at how it all works in the actual code.

As mentioned, by default, every Webiny project comes with three project applications: API, Admin Area, and Website, which are located in the api, apps/admin, and apps/website folders, respectively:

.
├── api # API
│ ├── code
│ ├── pulumi
│ ├── Pulumi.yaml
│ ├── tsconfig.json
│ └── webiny.application.ts
├── apps
│ ├── admin # Admin Area
│ │ ├── code
│ │ ├── pulumi
│ │ ├── Pulumi.yaml
│ │ ├── tsconfig.json
│ │ └── webiny.application.ts
│ └── website # Website
│ ├── code
│ ├── pulumi
│ ├── Pulumi.yaml
│ ├── tsconfig.json
│ └── webiny.application.ts
└── (...)

As we can see, every project application follows the same general organization. The two folders in each project are:

code - contains application code (one or more packages)
pulumi - contains cloud infrastructure (Pulumi) code

Furthermore, if we opened each of these pulumi folders, we’d see different cloud infrastructure resources clearly defined via multiple TypeScript classes. As a simple example, if we were to open the apps/admin/pulumi (Admin Area) folder, we’d find the following three files:

app.ts - deploys an Amazon S3 bucket that hosts the Admin Area (React) application
cloudfront.ts - deploys an Amazon CloudFront distribution for improved availability
index.ts - the Pulumi entrypoint file, imports classes defined within separate files

This code organization makes it much easier for developers to grasp the overall cloud infrastructure. It also makes it easier for them to adjust the code to their needs eventually.

A more complex example is the cloud infrastructure code in the API project application (api/pulumi), which deploys many different cloud infrastructure resources, like AWS Lambda functions, DynamoDB tables, ElasticSearch clusters, VPCs, and more.

But I will leave this up to you to check out, as pasting multiple chunks of code here might not be that productive.

Instead, I wanted to focus on a couple of other interesting and useful features that Pulumi and the infrastructure-as-code approach enabled us to create.

API Project Application - Different `dev` and `prod` Stacks

As mentioned, the API project application contains many different cloud infrastructure resources. But, as it turns out, when it comes to active development, we don’t have to unleash the full potential of the cloud, like we’re doing in staging and production environments.

For example, it probably makes no sense to deploy an Amazon ElasticSearch cluster into multiple availability zones (AZs) in most cases. A single AZ is enough for development purposes.

Another good example are VPCs and potentially NAT Gateways. Again, for development purposes, in most cases, we could probably get away without these too.

These extra resources make it slower to perform deployments from developers’ machines and incur extra costs for an organization because developers are deploying more resources, some of which are charged hourly.

So, with that in mind, we split the API project application’s cloud infrastructure into two stacks - the dev and prod. And, as you might’ve already guessed, only the prod variant will deploy absolutely all necessary cloud infrastructure resources.

What is even more interesting is the fact that this can be achieved with a simple if statement, which we placed in the index.ts entrypoint file:

// (...)

export = async () => {
 // (...)

 // WEBINY_ENV represents the environment which was passed upon running
 // the `webiny deploy` command. This is inserted automatically by the
 // Webiny CLI.
 if (process.env.WEBINY_ENV === "prod" || process.env.WEBINY_ENV === "staging") {
 // Import "prod" resources config and initialize resources.
 return await import("./prod").then(module => module.default());
 }

 // Import "dev" resources config and initialize resources.
 return await import("./dev").then(module => module.default());
};

Automatic Tagging of Cloud Infrastructure Resources

Another useful feature is the automatic tagging of the deployed cloud infrastructure resources. In other words, every taggable cloud infrastructure resource will be tagged with WbyProjectName and WbyEnvironment tags. For developers, this makes it much easier to see all of the deployed resources within their Webiny project.

We created a tagResources function, which essentially registers a global stack transformation via pulumi.runtime.registerStackTransformation function to achieve this.

It is also applied in the same index.ts entrypoint file we saw in the previous section:

import { tagResources } from "@webiny/cli-plugin-deploy-pulumi/utils";

export = async () => {
 // Add tags to all resources that support tagging.
 tagResources({
 WbyProjectName: process.env.WEBINY_PROJECT_NAME as string,
 WbyEnvironment: process.env.WEBINY_ENV as string
 });

 // (...)
};

Protect Feature

Finally, to protect our users from accidental deletions of mission-critical cloud infrastructure resources, we’ve used Pulumi’s protect feature:

The protect option marks a resource as protected. A protected resource cannot be deleted directly. Instead, you must first set protect: false and run pulumi up. Then you can delete the resource by removing the line of code or by running pulumi destroy. The default is to inherit this value from the parent resource and false for resources without a parent.

So, for Webiny users, we’ve made sure that this feature is automatically enabled for resources like DynamoDB tables, Cognito User Pools, Elasticsearch Cluster, and similar.

Future Plans

With the Webiny v5, we managed to introduce many significant improvements to both the Webiny Serverless Application Framework and Webiny Serverless CMS. However, we still have a lot of new cool things to unravel.

Webiny aims to be a multi-cloud compatible solution, and this is something Pulumi will significantly assist us with because deploying to multiple cloud providers is supported.

With that, we would also like to create various project templates, which would, for example, enable users to quickly start working on a new GraphQL or REST HTTP API, a simple React application, a microservice, and more. All of these require specific cloud infrastructure resources to be deployed, and again, this is where Pulumi will make our users’ lives much easier.

Finally, I already mentioned the Automation API as something that we might look into in the future, as it seems it’s something that can be seamlessly integrated with Webiny.

Conclusion

For Webiny, choosing Pulumi as the default infrastructure as code framework was certainly a big leap in the right direction.

It enabled us to leave the complexities of cloud infrastructure deployments to a product and company whose primary focus is exactly that. And this is what ultimately helped us completely focus on what we care about the most - creating an awesome serverless application framework.

Skai Migrates to AWS with Pulumi

Danny Zalkind — Tue, 04 May 2021 00:00:00 +0000

Danny Zalkind is the DevOps group manager for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai’s solution provides data-driven insights and optimization technology to help companies make informed decisions and scale performance across critical publishers.

Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers.

Why Skai is moving to AWS

The public cloud is not entirely new to us. Skai’s microservices have been running on AWS for a few years now. Our core monolith service is hosted in a data center running our private cloud infrastructure with its hundreds of Terabytes of data.

As our business grew, we noticed that our private cloud infrastructure could not keep pace with our business needs. Adding new capacity to our private cloud could take months of purchasing hardware, preparation, shipping, and deployment.

We decided to adopt infrastructure elastic enough to support the rate of innovation and growth that the business demanded.

We sought out the best infrastructure provider to shift as many services as possible to the cloud. AWS is a clear choice because of its unmatched managed services: RDS for our MySQL databases, Lambda functions for our scheduled processes and various automation tasks, DMS for data migration from the datacenter, Keyspaces for our Cassandra cluster, and EKS for our container workloads.

How Pulumi helped with migration

Once the team selected a cloud platform, it was time to plan and execute the migration. Skai’s DevOps group was selected to lead the project.

Why Pulumi

Skai’s private cloud infrastructure was provisioned and deployed using a combination of Puppet and Fabric. The code was long, full of customizations, not readable, and challenging to troubleshoot.

We were looking for a new solution that would:

Support cloud-native and multi-cloud architectures
Allow adding complex logic to the deployment process
Support reusable code blocks for various use cases
Allow each team to author different parts of the infrastructure
Use a modern programming language
And not create bottlenecks in the release process between the teams

We immediately fell in love with Pulumi, with its programmatic design, extensive modules, and extensibility via dynamic providers.

Project and Stack design

When the Skai team decided to standardize on Pulumi, we designed our architecture to support our use case and development cycle.

We decided to create a project for each AWS organization OUs and a stack for each instance of our application service.

For example, a project would be “platform-production” and contain all stacks and resources of individual environments such as “app1234,” where the number is an environment identifier.

This provided the optimal mix of releasing new code versions per workload type and placing them in the development lifecycle while making granular configuration changes per each instance more manageable.

Programming language

Pulumi supports several programming languages. For us, Python was an obvious choice due to its highly extensible nature, simplicity, and the team’s considerable amount of prior experience with Python. Since most existing automations were written in Python, this made adopting Pulumi very easy for the team.

Multi-team collaboration

Our organization is structured in multiple teams, with each team responsible for different areas of our infrastructure:

The Core team in charge of shared infrastructure such as provisioning AWS accounts
Multiple application teams are responsible for provisioning specific applications, and
An architect who leads designs, builds proof of concepts, and writes code.

It was essential to build our IaC projects with minimal coupling between the teams while keeping each team aligned with the same overarching project goals.

To enable this “loosely-coupled, tightly-aligned” approach, we created component classes for different infrastructure elements, which we later referenced in different places in our end-to-end code.

import pulumi
from pulumi_aws import ec2
from pulumi_aws import ebs

class KSAppServer(pulumi.ComponentResource):

 def __init__(self,
 name: str,
 args: KSAppServerArgs,
 opts: pulumi.ResourceOptions = None):
 super().__init__('kenshoo:KS:KSAppServer', name, None, opts)
 self.name = name
 self.app_instance = ec2.Instance("ks-aws-ue-" + args.ks_id + "-app",
 instance_type=args.instance_size,
 vpc_security_group_ids=args.security_groups_id,
 subnet_id=args.subnet,
 ami=args.ami,
 user_data=args.user_data_content,
 key_name=args.key_pair,
 tags={
 'Name': "ks-aws-ue-" + args.ks_id + "-app"
 },
 opts=pulumi.ResourceOptions(parent=self, delete_before_replace=True))

This practice enabled the application team to work on classes without being entirely dependent on the progress of the Core infrastructure team, which worked on classes of more low-level infrastructure such as VPCs and networking objects.

What about those internal metadata systems?

One more challenge we faced is registering each new instance of our application with internal metadata and IT systems. In our data center setup, we relied on a set of Jenkins jobs, but now, we wanted to integrate these processes with our code and make sure that we always have a consistent state of registration. We decided to use Pulumi’s dynamic providers to achieve this:

from requests import post, get

URL = 'https://krem.ourdomain.com/api/'

class KremMetadataRegistration:
 def create(self, ks_id):
 post(url=f'{URL}/onboarding/init_new_ks',
 params={'ks': f'ks{ks_id}'})

 def delete(self, ks_id):
 response_id = get(url=f'{URL}/config', params={'ks': ks_id}).json()[
 'id']
 if response_id:
 post(url=f'{URL}/onboarding/offboarding',
 params={'id': response_id})

Next, we reference the dynamic provider in another class that we call from the end-to-end code of the whole provisioning flow.

from pulumi.dynamic import *
from .metadata.krem_metadata_registration import KremMetadataRegistration

class RegistrationSchemaInputs(object):
 ks_id: Input[str]
 registered_service: Input[bool]

 def __init__(self, ks, registered_service):
 self.ks = ks
 self.ks_id = ks.lstrip('ks')
 self.registered_service = registered_service

class SchemaProvider(ResourceProvider):
 def __init__(self):
 self.metadatas = {
 'krem': KremMetadataRegistration()
 }

We can streamline the registration and deregistration processes for multiple systems as part of our provisioning lifecycle, creating a more streamlined flow with this approach.

Summary

Moving a high-scale and mature application to the public cloud presented quite a few challenges. We have selected Pulumi as our IaC framework of choice due to its native programmatic nature, an endless number of modules, and the ability to customize the solution using our dynamic providers and custom logic.

This post gave you a glimpse into how we designed and implemented provisioning of our application on AWS. Join us at the second part of the blog that follows our journey of automating data migration.

How Pinpoint Manages Kubernetes Costs and Deployments

Andrew Kunzel — Mon, 17 Aug 2020 00:00:00 +0000

This guest blog was contributed by Andrew Kunzel and Michael Goode of Pinpoint. Andrew is the Director of Backend Engineering, and Michael is a Platform Operations Engineer.

At Pinpoint, Kubernetes is the most powerful tool in our arsenal. It allows us to deploy and rapidly scale our applications with speed and efficiency that continues to delight our customers. In recent years, managed services like AWS EKS have made it easier than ever to leverage the power of Kubernetes in even the smallest of organizations. Yet even with these new conveniences, managing all of this infrastructure can be a daunting task. Right out of the gate, we knew that we wanted to avoid the burden of maintaining repositories full of home-brewed deployment scripts and domain-specific languages like YAML.

Pulumi affords us the ability to streamline the development of our cloud infrastructure with open source libraries that integrate deeply into AWS service APIs and support many popular languages. This introduces powerful, high-level constructs like for-loops and strong type-checking alongside the more rigid declarative approach mandated by other available tools. And it’s not just limited to hardware! We also use Pulumi to substantially reduce the overhead required to deploy critical services that run on Kubernetes itself, which helps to consolidate our deployment configuration even further and makes updating these services a breeze.

In this blog post, we’ll share some simple and intuitive ways in which Pulumi has simplified our approach to infrastructure-as-code.

Spot Instances

As a startup, we are always budget-conscious. Every tool we use has to justify either the up-front subscription cost or development time, or both. To tamp down our AWS EC2 costs, we decided to leverage EC2 spot instances as worker nodes in our development Kubernetes cluster. Spot instances are highly cost-efficient because they use spare compute capacity within a region at a fraction of the on-demand instance price. When you request a spot instance, you must offer the maximum price you are willing to pay. This price can fluctuate based on demand in a particular region, but will always be less than or equal to the current on-demand price. Knowing this, we could use Pulumi’s JavaScript AWS library to get the current on-demand price for a given instance type and plug that directly into the spotPrice property of the node group configuration.

function getOnDemandPrice(instanceType: aws.ec2.InstanceType): pulumi.Output<any> {
 return pulumi.output(aws.pricing.getProduct({
 filters: [
 {
 field: "instanceType",
 value: `${instanceType}`,
 },
 {
 field: "operatingSystem",
 value: "Linux",
 },
 {
 field: "location",
 value: "US East (N. Virginia)",
 },
 {
 field: "preInstalledSw",
 value: "NA",
 },
 {
 field: "usageType",
 value: `BoxUsage:${instanceType}`,
 },
 {
 field: "termType",
 value: "OnDemand",
 },
 ],
 serviceCode: "AmazonEC2",
 }, { async: true })).apply(data => {
 let result = JSON.parse(data.result)
 let typeOnDemand = result.terms.OnDemand
 let skuTermCode = Object.keys(typeOnDemand)[0]
 let rateCode = Object.keys(typeOnDemand[skuTermCode].priceDimensions)[0]
 let price = typeOnDemand[skuTermCode].priceDimensions[rateCode].pricePerUnit.USD
 return price
 })
 }

 cluster.createNodeGroup(`${stack.env}-${name}`, {
 amiId: ami,
 version: kubernetesVersion,
 instanceProfile: nodeGroupInstanceProfile,
 minSize: 1,
 maxSize: 10,
 desiredCapacity: 5,
 nodeRootVolumeSize: 100,
 nodeAssociatePublicIpAddress: false,
 nodeSubnetIds: stack.privateSubnets,
 instanceType: instanceType,
 spotPrice: getOnDemandPrice(instanceType)
 })

And just like that, we can quickly provision spot instances on our cluster! Better yet, we don’t ever have to worry about setting a valid spot price or managing this process via custom scripts in a CI/CD pipeline. The entire deployment is easily managed within the same file.

Managing Third-Party Services

Pulumi also helps us to manage the services that we run on our Kubernetes clusters by allowing us to efficiently configure these services consistently that avoid drift and minimizes complexity. Pulumi is extremely flexible in this regard because it offers native support for de-facto tools like Helm and Kustomize as well as individual Kubernetes components like pods and deployments. You can even render raw YAML, which greatly simplifies integration with legacy systems or custom components.

Transformations with Helm

We prefer to use Helm for most of our deployments. It is an excellent tool for deploying third party services and customizing them with a useful but complicated templating system. These templates are great when they work, but if the chart’s maintainer doesn’t make something configurable, you’re usually unable to alter the manifest when it comes time to deploy your chart. You’d likely have to fork and manage a custom version of the chart yourself in the worst case. Luckily, Pulumi can override just about anything in a manifest by using a Transformation. This property allows you to manipulate a manifest’s content before it is rendered and applied to the cluster.

As an example, we recently discovered that some versions of the Traefik Helm chart do not include an explicit namespace definition on certain templates. This caused a few resources to install into the default namespace; without Pulumi, we would have been unable to fix this issue without forking the chart or writing a post-install script to wrangle them back. Luckily, all we really had to do was create a function to set the object metadata and then pass that function directly into the ‘transformations’ property of the Traefik Helm chart:

 function setNamespace(obj: any) {
 if (obj.metadata.namespace === undefined) {
 obj.metadata.namespace = 'platform-services';
 }
 }

 new k8s.helm.v3.Chart("traefik", {
 repo: "traefik",
 chart: "traefik",
 version: "6.0.0",
 namespace: 'platform-services',
 transformations: [setNamespace],
 values: {
 deployment: {
 replicas: 3
 },
 },
 }, { provider: provider });

This will run the transformation on every manifest used by the chart, fixing our issue in only a few code lines.

Being able to manage all of our infrastructure as code and enhance the behavior of standard Kubernetes deployment tools has dramatically improved our ability to deploy services and maintain our configuration. We look forward to expanding our use of Pulumi as we grow as an organization and empower our developers further.

Test and Optimize Elasticsearch with Pulumi

Itamar Syn-Hershko — Thu, 05 Mar 2020 00:00:00 +0000

Guest Article: Itamar Syn-Hershko, Founder and CTO of BigData Boutique shows how they use Pulumi to benchmark Elasticsearch configurations across cloud providers. Pulumi enables BigData Boutique to test deployments in parallel and gather metrics to produce performant and cost-effective solutions for its customers.

BigData Boutique can rapidly iterate through infrastructure configurations based on client requirements. Familiar programming languages significantly shortens the inner dev loop because developers and operators can use modern programming tools such as IDEs, test frameworks, and CI/CD. By increasing the velocity of infrastructure deployment across many clouds with repeatable and versioned code, Pulumi enables companies to build efficient revenue centers for their business.

How Pulumi Drives Our Elasticsearch Capacity Planning and Cost Optimization Service

Capacity Planning and Cost Optimization of Elasticsearch clusters requires a special level of expertise and automation. Here is how we use Pulumi to launch long-running benchmarks to correctly identify the right configuration for our customers’ Big Data clusters.

At BigData Boutique, we are continually challenged by our customers—whether it’s complex Big Data challenges we are asked to solve, reducing costs for sophisticated systems holding mountains of data, improving performance for high-demand systems, or challenges faced in building a data-driven system.

One of the main challenges our customers are facing with Big Data systems is deploying the right hardware at the right scale. They need to know how to correctly estimate the number of servers required to run their compute workloads and databases, so they don’t pay more than they should, while still keeping those services running without interruption and within the defined KPIs.

To support our customers and help them succeed in their Big Data projects, we provide an array of advanced services, and our unique cost-optimization service addresses this challenge directly. This article outlines how we use long-running benchmarks to correctly identify the right configuration for our customers’ BigData clusters by using the infrastructure as code tool Pulumi for automating the creation of complex infrastructures with conditionals and highly tunable configurations to orchestrate those benchmarks at scale.

Elasticsearch Cost Optimization

Let’s take Elasticsearch as an example. Elasticsearch is a real-time search and analytics engine. It is built to ingest huge amounts of data and to allow one to search through and visualize them in real-time. But to be able to sustain the required amount of data on ingestion and the desired number of queries per second, the right hardware has to be used.

To do this, we have to make some decisions: which disks to use (SSD disks? local, or network storage?), the number of servers (scaling up or out?), CPU and RAM ratios, index mappings, various cluster configurations, and more. These are not easy questions to answer, and that is where our Elasticsearch expertise comes into play. But answering those questions correctly with a high level of certainty also requires rigorous testing for validating assumptions.

Whether our customer is running on-prem or on a cloud, our Elasticsearch Capacity Planning Service exists for one purpose: to find the hardware solution with the optimum balance between cost and performance, and do it scientifically, so the answers are as accurate and as precise as possible.

Elasticsearch Sizing Process

Our process begins with a sample data request. We ask the customer for their data (or a large sample of it), existing index mappings, expected queries, and finally known KPIs and SLAs. Our goal in the initial discovery phase is to reduce the number of free variables as much as possible because the more such variables, the more permutations we will have to test. Ideally, we want very few dimensions of freedom — usually in terms of hardware and configurations—so the number of permutations to test is on the order of dozens and not thousands.

Once the initial framework has been decided, we run fully automated benchmarks to test some of the permutations our experience tells us are the most promising to deliver good results. The benchmarking process is smart enough to omit long tails and outliers, so the results are not going to lie as to the true nature of the configuration that was selected and tested.

Between benchmark iterations, our team uses its years of experience to create more test configurations that optimize various configurations we decide are important based on the results of the previous benchmarks. After several iterations of benchmarking on various configurations, we send the preliminary findings to confirm our results and proposed direction to the customer. The purpose is to communicate the results and validate the trade-offs we decide on during the sizing procedure. Finally, we provide a summary session with our team of Elasticsearch experts and customer technical and business contacts to present the results and recommendations, and to explain the reasoning behind the analysis.

Finding the best Elasticsearch configuration requires careful benchmarking and tech expertise. Yet trade-offs will come into play. For instance, it might seem that the only way to gain another 100ms of performance improvement for a certain specific query would be to spend another $500 per month on hardware. But there may be other options. Our Sizing Service takes the guesswork out of the process. We automate the entire benchmarking procedure in order to make it an exact science.

To fully automate the benchmarking process, we use Pulumi.

Using Pulumi for State-of-the-Art Infrastructure Automation

Pulumi is an infrastructure-as-code tool that allows us to define and run infrastructures using code, as opposed to doing this manually or using other tools that use configuration-based syntax. This lets us create complex infrastructures with conditionals and highly tunable configurations. The Pulumi program we created can be run numerous times with different parameters. Each result is a cluster that is built and configured differently, sometimes even on completely entirely different clouds.

At BigData Boutique, we use that concept to launch multiple infrastructures in parallel, benchmark them, and show a report that highlights which config is better along with the various trade-offs for each. After the benchmark process, the cluster is destroyed, and the “provisioner” instance reports all logs, metrics, and the benchmark report, which we then analyze and visualize.

Generating, visualizing, and analyzing dozens of reports in parallel gives us, the Elasticsearch Engineers, the ability to understand which knobs to turn and what configurations to change to make our customers succeed in their project without spending more than they are willing to.

Conclusion

In sum, leveraging Pulumi helps us to create highly configurable infrastructures to orchestrate benchmarks at scale. As one of many advanced services that we offer, the Elasticsearch Capacity Planning and Cost Optimization Service continues to meet our customers’ most important planning and optimizations needs in an as scientifically rigorous manner as possible. You can read more about it here.

The original article was published on their company blog.

Mapbox IOT-as-code with Pulumi Crosswalk for AWS

Chris Toomey — Wed, 12 Jun 2019 00:00:00 +0000

Guest Author: Chris Toomey, Solution Architect Lead @ Mapbox

With 8 billion+ connected IoT devices and 2 billion GPS-equipped smartphones already online, logistics businesses are tracking assets at every step in the supply chain. At this scale and complexity, it is imperative to have a flexible way to ingest, process, and act upon this data, without sacrificing security or best practices.

To meet this need, Mapbox has created an Asset Tracking Solution that uses Pulumi’s open source JavaScript libraries (AWS, AWSX) available with multi-language support with Pulumi Crosswalk for AWS. Pulumi Crosswalk for AWS is an open source framework that streamlines creation, deployment and management of AWS services with built-in AWS Best Practices and minimal lines of code in common programming languages.

In this blog, we will show snippets of the Javascript code that embraces the power of Pulumi to program AWS service APIs to create the Mapbox solution. To see the full architecture in action with a live bike race across America, please refer to this webinar recorded on June 13th 2019 and the Mapbox asset tracking solution. Also refer to this blog of the Race across America showcased live during the webinar tomorrow.

Prerequisites: Install Pulumi; Install Node.js and Setup AWS

The diagram represents how Mapbox’s solution design on AWS services is built with Pulumi AWS and AWSX libraries in Javascript. Data is ingested in the Asset tracking IOT solution with a REST API. Mapbox uses its stream processor to perform enrichments such as geofencing, traffic-aware ETA calculations, and high-prevision elevation. This data, exposed by the API is backed by a high performance database (DynamoDB) to enable visualization in realtime on the map client.

The fun part – Pulumi Service console helps you map your cloud architecture shown above as a connected set of DAG resources so you need not remember what you deployed as your cloud environments scale:

Lets now work through the solution flow with some sample snippets of code. To get access to the complete piece of code, please connect with Mapbox solution team here and Pulumi team here. We will be happy to help you sample test this solution on AWS!

STEP 1: Create a Pulumi project using an AWS JavaScript template

$ brew install pulumi/tap/pulumi
$ mkdir asset-tracking && cd asset-tracking
$ pulumi new aws-javascript
$ yarn install
$ ls
Pulumi.dev.yaml Pulumi.yaml index.js node_modules package-lock.json package.json

STEP 2: Create the ingestion REST API with AWS Services

To visualize data the system has to accept large volumes of data from multiple sources into the rest of the system. Mapbox uses Pulumi’s AWS & AWSX libraries to build an IOT ingestion rule and forwards it to Kinesis streams. A sample index.js looks like this:

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");
const awsx = require("@pulumi/awsx");

// Create Kinesis stream for ingestion
const ingestStream = new aws.kinesis.Stream("ingestAssets", {
 shardCount: 1,
 retentionPeriod: 72
});

// Create IoT Rule to push into Kinesis stream
const iotRole = new aws.iam.Role("iotRole", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 Service: "iot.amazonaws.com"
 },
 Action: "sts:AssumeRole"
 }
 ]
 })
});

const iotRolePolicy = new aws.iam.RolePolicy("iotRolePolicy", {
 policy: pulumi.interpolate`{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "kinesis:*"
 ],
 "Resource": "${ingestStream.arn}"
 }
 ]
 }`,
 role: iotRole.id
});

const iotRule = new aws.iot.TopicRule("iotTrigger", {
 description: "Pass from IoT Core to Asset Tracking",
 name: "iotAssetIngest",
 enabled: true,
 kinesis: {
 partitionKey: "id",
 roleArn: iotRole.arn,
 streamName: ingestStream.name
 },
 sql: "SELECT * FROM 'topic'",
 sqlVersion: "2015-10-08"
});

Kinesis streams are used to ingest data into multiple AWS services defined as IOT rules, Firehose and Lambda functions. Each ingest stream has the right IAM role and policy to allow Kinesis to send the data into these AWS services.

The code above does not shown the Javascript code for Lambda functions used by Mapbox streamprocessor and Firehose all created with Pulumi AWS and AWSX libraries.

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

STEP 3: Create a consumption API for the map client to consume from a data source

Once, data is flowing through the system we build another API to query DynamoDB, transform that data, and provide it to our mapping client in geojson. With Pulumi AWSX, we create a APIGateway endpoint API that can query this data out of DynamoDB. The sample code in index.js will look like this:

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

// Create DynamoDB Table
const assetTable = new aws.dynamodb.Table("assetTable", {
 attributes: [
 {
 name: "id",
 type: "S"
 },
 {
 name: "ts",
 type: "N"
 }
 ],
 hashKey: "id",
 rangeKey: "ts",
 ttl: {
 attributeName: "expiration",
 enabled: true
 },
 billingMode: "PAY_PER_REQUEST"
 });

// Create API to read DynamoDB
const endpoint = new awsx.apigateway.API("assetQuery", {
 routes: [
 {
 path: "/",
 method: "GET",
 eventHandler: (request, ctx, cb) => {
 const AWS = require("aws-sdk");
 const ddb = new AWS.DynamoDB.DocumentClient({
 apiVersion: "2012-10-08"
 });
 const tableName = assetTable.name.value;
 const params = {
 TableName: tableName
 };
 ddb.scan(params, (err, data) => {
 const features = data.Items.map(item => {
 const point = turf.point([item.longitude, item.latitude], {
 id: item.id,
 speed: item.speed
 });
 return point;
 });
 const featureCollection = turf.featureCollection(features);
 cb(undefined, {
 statusCode: 200,
 body: Buffer.from(
 JSON.stringify(featureCollection),
 "utf8"
 ).toString("base64"),
 isBase64Encoded: true,
 headers: { "content-type": "application/json" }
 });
 });
 }
 }
 ],
 stageName: "dev"
});

Once we’ve created the API, we start defining our routes and then in-line we can define a Lambda function. It can use NPM modules and Pulumi will handle bundling all the dependencies.

This endpoint assumes we are querying Dynamo - and so we are going to use the DynamoDB Document Client. Using that straightforward scan syntax, we query the table and then start to parse our results. Since every data point has a latitude and longitude as GeoJSON, the data is transformed using Turf.js before sending to the client. Even more beautiful is how Pulumi handles all the heavy lifting of packaging, deploying, and giving you the endpoint to query immediately.

STEP 4: Hook up a map!

Now we have data coming and going from our Solution, we just have to hook it up to a Mapbox map. This is actually quite simple now that Pulumi has built us a REST API.

Following this example, we paste in the endpoint from Pulumi and pick our interval.

Additional Resources

For more information from Mapbox check out the Mapbox Blog. For more solutions from Pulumi please visit our Blog or head over to the Pulumi Examples repository. For AWS solutions visit the AWS partner blog site.

Pulumi Blog: Customers

Seven Rules for Building an AI-Native Software Factory

1. Transform, don’t enhance

2. Remove the problem, don’t solve it

3. Pick tools your agents can drive

4. Don’t let one agent do everything

5. Measure human hours per unit of value

6. Design for convergence, not one-shot correctness

7. Run the factory in the cloud, not on a laptop

How to build an AI-native software factory

Closing thought

Frequently asked questions

Platform Engineering Buffet at SEITENBAU

The Buffet Philosophy

The Challenge: Beyond Microservices

Building the Kitchen: Architecture with Pulumi

The Multi-Stack Approach

From Ingredients to Ready-Made Dishes: Reusable Components

Example: The Kubernetes Storage Component

Two Years Later: The Buffet is Open

How We Used Pulumi to Safely Migrate Oso's Global Infrastructure

Osoʼs Migration

How the Migration Actually Worked

Writing Reusable Pulumi Code

Infrastructure Snapshot Testing for Zero Diff Refactors

Conclusion

DevSecOps Game-Changer: Security Automation That Delivers Business Results

On this article:

The Beginning: From Data Center to the Cloud

Embracing Pulumi: The Power of Automation, Productivity, and Security

Driving Business Impact Through Security Automation

Fostering DevOps Culture Through Cross-Functional Collaboration

Lessons Learned: Navigating the Cultural Shift

The Future of DevSecOps and Pulumi at Tivity Health

Unified and Programmatic Approach to Infrastructure Management at BMW Using Pulumi

On this article:

The Challenges of a Sprawling Software Landscape

The Evolution of BMW’s Software Development Toolchain

Embracing Pulumi: Streamlining Infrastructure Management

Shared Modules: Promoting Reusability and Best Practices

Transformation Features: Protecting Legacy Services

Policy Enforcement: Ensuring Compliance and Security

The Benefits of Pulumi: Accelerating Development and Improving Maintainability

Accelerated Development with Shared Modules

Improved Maintainability and Consistency

Leveraging Python’s Ecosystem

Streamlined Cloud Migration

The Future of BMW’s Software Factory: Embracing the Cloud

Conclusion: Unlocking the Future of Automotive Software

AWS CDK vs Pulumi: Why SST Chose Pulumi

In This Article:

The Beginnings of SST

CDK and CloudFormation Limitations

A Provider-agnostic Solution: Discovering Pulumi

Transitioning to Pulumi

The Benefits of Pulumi

Improved Visibility and Debugging

Extensibility and Customization

Multi-Cloud Capabilities

Simplified Mental Model

The Future of SST with Pulumi

Conclusion

Frequently Asked Questions

What is SST?

What is SST’s Ion?

Does SST use mostly Terraform or Pulumi?

How does SST make money?

How Starburst Data Creates Infrastructure Automation Magic With Code

Technical and Business Problems to Solve

Why Starburst Chose Pulumi

Benefits of Using Pulumi

Try Pulumi for Infrastructure as Code Automation

How a Bank Modernized Its Software Engineering With Infrastructure as Code Automation

Background

Why Washington Trust Bank Chose Pulumi

Benefits of Using Pulumi

Improved Development Speed with Reusable Infrastructure Packages and CI/CD

Safeguards for Infrastructure

Policy as Code Guardrails with Pulumi CrossGuard

Financial services cloud modernization with Pulumi

API Project Application - Different `dev` and `prod` Stacks