This blog post summarizes a presentation by Matt Stephenson at PulumiUP 2023. Matt Stephenson is Senior Principal Software Engineer for Starburst Data and a Puluminary member. He’s deeply involved in the Infrastructure as Code (IaC) space, having contributed to Ansible, been a core contributor to Apache jclouds, and has written many Terraform plugins. He leads infrastructure architecture at Starburst and originally introduced Pulumi to the company. Starburst provides a data lake analytics platform that’s powered by Trino - an open-source distributed SQL query engine designed for running fast analytic queries across large datasets in multiple data sources.
Welcome to the sixth post in our series of blog posts focused on infrastructure as code (IaC) recommended practices. So far in this series, you’ve seen how Zephyr Archaeotech Emporium—the fictional company at the center of this series—uses Pulumi to manage their online retail store. You read how Zephyr’s initial use of Pulumi changed to use short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, incorporated Stack References, and used Pulumi Cloud’s role-based access control (RBAC) functionality to control access to their stacks. This post focuses on how Zephyr takes advantage of the Pulumi Automation API to bring an even greater level of orchestration to the stacks that represent their online store.
As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:
Today we’re excited to announce Review Stacks – dedicated cloud environments that get created automatically every time a pull request is opened, all powered by Pulumi Deployments. Open a pull request, and Pulumi Deployments will stand up a stack with your changes and the Pulumi GitHub App will add a PR comment with the outputs from your deployment. Merge the PR and Pulumi Deployments will destroy the stack and free up the associated resources.
Delivering software has been conventionally driven by CI/CD workflows. A single commit is merged into a codebase, and a small, and static set of workflow runs are triggered by the CI system to update the appropriate environments (Dev -> Staging -> Production). This may have been fine when the only product a company had to offer was a single stateless web service, but increasingly companies are called upon to deliver cloud infrastructure as a product.
What is FinOps? The FinOps Foundation eloquently defines FinOps as “an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.” Simply put, FinOps is the continuous effort to control cloud spend. Just as organizations have adopted operations-focused best practices into software development cycles and have considered how to best insert security best practices along the way, financial best practices may also be codified by developers writing cloud programs.
Adoption of modern cloud technologies and services is driving enormous value for organizations, but many companies are finding that deploying and managing cloud infrastructure is a bottleneck on how fast they can scale. When your tools can’t keep up, scaling your cloud footprint means proportionally scaling your headcount. Things quickly fall apart as demand for cloud infrastructure often outpaces the bandwidth of infrastructure teams. This leads to slow product releases, longer lead times to get new products to market, and burned-out operational teams.
At Pulumi, we want every engineer and organization to be able to take full advantage of the cloud. The cloud should be an accelerant to your business and not a bottleneck. Today we are excited to be launching Pulumi Deployments, a new collection of features to power infrastructure and platform automation and unlock the scale of the cloud.
At PulumiUP 2022, Tomas Jansson, software architect at Elkjøp Nordic, gave a presentation on how to enable developers to self-service infrastructure by using Pulumi’s Automation API. Elkjøp Nordic is the leading consumer electronics retailer in the Nordics. The company sells consumer electronics, mobile phones, computers, white goods, domestic appliances, and services linked to these products both directly to consumers and to businesses. It is an omnichannel retailer and serves customers both online and through more than 400 stores.
Pulumi community member Kay Plößer spent some time digging into setting up observability of a Pulumi deployment using Honeycomb. Read more to find out all the details on configuring Honeycomb and Pulumi together, with a side dish of Automation API!
Using Pulumi is more than just writing code and components. In addition to common software development practices, there are also a number of success patterns related to how your company or team builds and deploys Pulumi programs to successfully build, deploy, and manage your infrastructure and applications. In this continuation of a series, I will explore one of these patterns - using the Pulumi Automation API to create a developer portal.