Pulumi Blog: George Huang

How Starburst Data Creates Infrastructure Automation Magic With Code

George Huang — Wed, 23 Aug 2023 00:00:00 +0000

This blog post summarizes a presentation by Matt Stephenson at PulumiUP 2023.

Matt Stephenson is Senior Principal Software Engineer for Starburst Data and a Puluminary member. He’s deeply involved in the Infrastructure as Code (IaC) space, having contributed to Ansible, been a core contributor to Apache jclouds, and has written many Terraform plugins. He leads infrastructure architecture at Starburst and originally introduced Pulumi to the company. Starburst provides a data lake analytics platform that’s powered by Trino - an open-source distributed SQL query engine designed for running fast analytic queries across large datasets in multiple data sources. At Starburst, Matt helped revamp and improve how the company manages its multi-cloud and cloud native infrastructure.

Watch Matt Stephenson’s full presentation with demos. 8:03 demo of CI/CD. 15:33 demo of infrastructure automation.

Technical and Business Problems to Solve

Starburst operates a managed Trino service that runs on Kubernetes clusters on multiple clouds. Their Kubernetes clusters needed frequent manual scaling based on workloads. Furthermore, upgrading Trino versions running on the clusters ran the risk of potential service disruptions for customers. The company had been using tools like Terraform for IaC, Argo CD for deploying Helm charts, Maven, Docker containers, and a Slack bot for infrastructure and deployments. Their CI/CD workflow was difficult to maintain because they needed to create wrappers for their previous IaC tooling, making it less approachable in CI/CD environments. The pipeline was tenuously bound together using GitHub Actions and a Slack bot, requiring specialized GHA runners, leading to a messy architecture.

Why Starburst Chose Pulumi

Recognizing opportunities for improvement, Matt brought in Pulumi and its Automation API. This unified Starburst’s software development and infrastructure management practices, especially when deploying and managing Trino clusters on Kubernetes. The top two reasons Matt chose Pulumi were:

Support for programming languages: Pulumi supports the main languages used by Starburst’s developers, Java and TypeScript. This enables all engineers to write infrastructure code and promotes a cohesive application and infrastructure codebase.
Pulumi Automation API: Automation API is a programmatic interface for the Pulumi CLI and Pulumi IaC engine, which enables Starburst engineers to build infrastructure automation into their applications and CI/CD processes, and orchestrate complex provisioning workflows.

Benefits of Using Pulumi

Adopting Pulumi and its Automation API provided several benefits that made Starburst’s engineers more efficient and productive, ultimately leading to faster and more reliable releases for Starburst’s customers. Matt summed up the benefits of Pulumi for Starburst’s engineers: “When I think about building an airplane, I think about precision and having the right tools for the job.” Pulumi, for Starburst, epitomized this precision and efficiency.

Increased deployment speeds by 3x for new customer releases: Starburst engineers used the Pulumi Automation API to triple the deployment speed of multi-cloud Kubernetes clusters with blue-green deployments that complete within minutes. They did this by orchestrating blue-green updates across their data plane running on the clusters, ensuring zero customer downtime. They built intricate integrations between Pulumi and internal APIs that enabled them to seamlessly transition customers to new clusters. Now they can update their customers’ runtimes, including VPCs and Kubernetes clusters, twice a week.

Unified cluster and infrastructure management: Rather than juggling multiple tools for managing Kubernetes and infrastructure, Starburst engineers use a single tool to manage over 50 Kubernetes clusters and deploy Helm charts into the clusters, spanning multiple regions. They also use Pulumi to manage SaaS platforms like Cloudflare, Confluent, and DataDog.

Consistent deployment lifecycles: Using Pulumi’s integration with GitHub Actions, Starburst engineers built a streamlined CI/CD process where infrastructure and application code builds within the same pipelines with automated test environments. They accomplished this using Pulumi Automation API to abstract the underlying runtime environments, circumventing the need for additional layers and wrappers.

Try Pulumi for Infrastructure as Code Automation

Sign up for a free account to try deploying infrastructure on any cloud with Pulumi Deployments, or register for an upcoming workshop to learn more about how Pulumi can help you ship cloud infrastructure faster and more safely.

Go to more case studies →

How a Bank Modernized Its Software Engineering With Infrastructure as Code Automation

George Huang — Thu, 03 Aug 2023 00:00:00 +0000

This blog post summarizes a presentation by Dennis Sauvé at PulumiUP 2023.

Washington Trust Bank, the largest independently-owned full-service commercial bank in the Northwest, has served personal, private, commercial and wealth management clients throughout the region since 1902. It has assets exceeding $11 billion and currently has 42 branches and offices in Idaho, Oregon, and Washington.

As an FDIC-governed financial institution, it is imperative for the bank to maintain secure, reliable, and compliant cloud resources to protect clients’ personal data. On the other hand, it also aimed to create more agile development teams as it modernized its software development and infrastructure. Dennis Sauvé, the bank’s first DevOps Engineer, recognized Infrastructure as Code (IaC) as the solution to these challenges.

Embracing an Infrastructure as Code approach would allow them to automate building and deploying their cloud infrastructure, eliminate infrastructure provisioning as a bottleneck, and empower developers to self-service infrastructure and increase productivity.

Background

The main challenge was managing Azure infrastructure in an efficient and secure way that kept pace with the bank’s rapid growth. Previously, they ran on-premise infrastructure and deployed with BAT and Powershell scripts. Now, they had migrated to Azure and wanted to increase development speed through CI/CD for cloud infrastructure.

In addition, cloud resources’ security and reliability were paramount. They needed a way for the infrastructure and information security teams to work together to implement security best practices for infrastructure, including how to prevent insecure resources from being provisioned.

The bank decided to adopt IaC to create reliable and rapid deployments while ensuring that its cloud infrastructure stayed compliant and secure.

Why Washington Trust Bank Chose Pulumi

The bank evaluated several options, including Terraform and Azure Bicep. However, Pulumi stood out due to its versatility, support for multiple clouds and native Azure support, and the ability to write IaC in the bank’s preferred language, TypeScript. The ease of using Pulumi with GitHub, its ability to be used in conjunction with Azure CLI, and its comprehensive support for CI/CD through GitHub Actions were also compelling factors.

Dennis praised Pulumi, saying, “Pulumi was exactly the Swiss Army Knife versatility we were looking for. We’ve only uncovered more of what we love about Pulumi as our relationship has evolved.”

Benefits of Using Pulumi

Pulumi helped Washington Trust Bank improve efficiency by maintaining its infrastructure in code, promoting the security of its infrastructure, and accelerating development speeds. It automated deployment of web services for its development teams, data management tools for its business insights team, and orchestrated its entire hub-and-spoke cloud architecture.

Improved Development Speed with Reusable Infrastructure Packages and CI/CD

Pulumi’s automation capabilities played a transformational role. By freeing the infrastructure team from handling manual deployments, it could focus on improving developer productivity. The team created reusable and repeatable infrastructure components built around strict IT security standards in the form of Frazure, an internal NPM package.

Frazure allows the bank’s developers to easily provision out-of-the-box resources for cloud services and covers the necessary Azure resource groups, vnets, and other resources. The package encapsulates best practices, such as how resource groups and their virtual networks can be created, including subnet IP spaces, peerings to and from the hub virtual network, user-defined routes and more.

To deploy, developers open a pull request which kicks off a CI/CD pipeline in GitHub Actions and provisions testing environments and tests. Once the PR is reviewed and approved, the changes are automatically deployed to staging and live environments. This automated process enabled the bank to accelerate the building, testing, and deployment of new applications, translating into rapid value delivery to their customers.

Safeguards for Infrastructure

Pulumi also gave the bank total confidence in being able to rapidly recover its cloud infrastructure in the case of a failure in Azure. The integration of Pulumi with GitHub Actions provided the team with a precise delta of changes to be made to the environment. Coupled with mandatory reviews from InfoSec and infrastructure, this significantly reduced the likelihood of deploying unapproved changes, thereby enhancing the reliability of their deployment processes.

Policy as Code Guardrails with Pulumi CrossGuard

Pulumi CrossGuard adds an extra layer of security and control and is used in conjunction with Azure Policies, which are used for auditing purposes. CrossGuard prevents the deployment of undesired, insecure, or expensive resources during the preview and deployment stage, thus preventing developers from even reaching Azure to provision resources. Custom error messages give developers context on why their deployment was not allowed.

Pulumi Cloud Policy Packs allow them to group and deploy many policies simultaneously. The Policy Packs prevent specified resources from being deployed into staging and live environments. For example, one policy requires all SQL databases to use TLS 1.2 by default and another ensures all storage buckets have public access disabled by default. These capabilities helped bolster the security of the bank’s cloud infrastructure.

Financial services cloud modernization with Pulumi

The DevOps and development teams at Washington Trust Bank rely on Pulumi as a key tool in their cloud modernization, while maintaining high security and quality standards. Pulumi played a vital role in streamlining their cloud infrastructure management, enabling their developers to be more productive, and improving the security and reliability of its cloud infrastructure.

Sign up for a free account to try deploying infrastructure on any cloud, or register for an upcoming workshop to learn more about how Pulumi can help you ship cloud infrastructure faster and more safely.

Go to more case studies

Announcing the Speaker Lineup for PulumiUP 2023

George Huang — Wed, 24 May 2023 00:00:00 +0000

PulumiUP is our virtual user conference for the Pulumi community and anyone interested in Infrastructure as Code. Every year, we strive to build a program packed with technical talks, demos, and best practices with the goal of leaving you with new learnings and inspiration that will help you become a better cloud engineer. You’ll hear from industry leaders and experts about IaC, software engineering, DevOps, Platform engineering, and AI.

We hope you’ll join us virtually and take this opportunity to grow your knowledge and become infrastructure as code stars.

Get Ready for Amazing Launches and Demos

Learn about Pulumi’s newest features across the core IaC SDKs and Pulumi Cloud. And watch hands-on demos by the Pulumi team:

Joe Duffy, Co-founder and CEO
Luke Hoban, Co-creator of TypeScript and CTO
Meagan Cojocar, Principal Product Manager
Evan Boyle, Engineering Manager
Zaid Ajaj, Software Engineer

Final Speakers Announced

You’ll hear from innovators, leaders, and experts from multiple industries and learn about infrastructure as code, software engineering, and cloud engineering.

Sam Cogan - Solution Architect, WTW

Talk: Keeping your Infrastructure Code DRY (Don’t Repeat Yourself)
Sam is a Solution Architect with WTW and a Microsoft Azure MVP. Sam is focused on delivering applications into the cloud using IaaS, PaaS, Serverless, Containers, and Kubernetes. Sam is particularly focused on automation and DevOps, including Infrastructure as Code, Cloud automation tooling, PowerShell, and CI/CD tooling.

Dennis Sauvé - DevOps Engineer, Washington Trust Bank

Talk: How a Bank Modernized its Software Engineering with Infrastructure as Code Automation
Dennis is a development operations engineer at Washington Trust Bank in Spokane, WA. With years of experience in software development and cloud engineering, he works to streamline deployment pipelines and create solutions for developers.

Ala Shiban - Co-Founder & CEO, Klotho

Talk: Creating a ‘Cloud-Aware’ Code Solution
Ala is the co-founder of Klotho, a startup that transforms plain code to cloud-native code. He started his journey on Commodore64 with cassette tapes and (much) more recently worked on distributed systems, developer tools, and large-scale web, cloud, and microservice backends.

He’s led smaller teams and larger organizations at Microsoft’s Developer Division and Riot Games’ Infrastructure Platform organization.

Matt Stephenson - Sr. Principal Software Engineer, Starburst Data

Talk: Creating Infrastructure Automation Magic with Code
Matt is a Sr. Principal Software Engineer at Starburst Data. He’s responsible for Starburst’s infrastructure architecture. He’s worked at many tech companies, including Square, Oracle, Google, and Amazon. Aside from technology, Matt is a pilot and helps build The Man for Burning Man each year.

Tim Holm - Co-Founder & CTO, Nitric

Talk: Engineering a Multi-cloud Platform
Tim Holm is the Co-founder and CTO at Nitric, with 12 years of experience in software engineering and cloud development. In disciplines ranging from Server and Storage Virtualisation and Automation, Mobile Application development, Full-stack application development, and Cloud infrastructure automation. Before founding Nitric, Tim worked to deliver critical infrastructure projects in Government and Fintech.

In his spare time, Tim enjoys video games, building Lego, and mountain biking.

Tyler Scheuble - Head of Platform, People Data Labs

Talk: Using Monorepos to Level-up your Infrastructure as Code and Software Delivery
Tyler possesses a genuine enthusiasm for technology. At People Data Labs, he has designed petabyte-scale data pipelines and high-performance APIs with demanding availability requirements, which informed his approach to leading the development of the next generation of the startup’s development pipelines and environments.

Tyler is passionate about finding simple solutions to cross-cutting concerns such as security, observability, and development velocity.

Sefi Genis - Co-Founder & CTO, Firefly

Talk: Lessons Learned from Writing Thousands of Lines of IaC
Sefi is the CTO and Co-Founder at Firefly. He started his career in the 8200 Unit of the Israeli Intelligence Corps and then continued on to security and developer tooling companies.

Sefi was a founding engineer at Dome9 Security (later acquired by Checkpoint), where he led the Backend Infrastructure of the product post-acquisition. Before founding Firefly, he became Head of Engineering in an Israeli cybersecurity startup company. Today he is building Firefly to help engineers manage the new reality of cloud-native operations.

Panel Discussion - AI and the Future of Cloud Development

Prepare yourself for the future of cloud development! Gain valuable insights from industry experts in our panel discussion, where we explore the transformative impact of AI on the cloud infrastructure and software engineering industries. The panel discussion will be moderated by CEO Joe Duffy. These are the panelists:

Amanda Silver - CVP Developer Tools, Microsoft

Amanda is the CVP of PM for Microsoft’s Developer Division, including the Visual Studio family of products, .NET, TypeScript, and our developer platforms. She has been key to Microsoft’s transformation to contribute to open source with the introduction of TypeScript, Visual Studio Code, and the acquisition of both Xamarin and GitHub. She believes that a tight digital feedback loop with zero distance between end-users and engineering teams is a critical element of great product development.

Beyang Liu - Co-Founder & CTO, Sourcegraph

Beyang is the CTO and co-founder of Sourcegraph. Prior to Sourcegraph, Beyang was a software engineer at Palantir Technologies, where he developed new data analysis software on a small, customer-facing team working with Fortune 500 companies. Beyang studied Computer Science at Stanford, where he published research in probabilistic graphical models and computer vision at the Stanford AI Lab and thoroughly enjoyed his compilers course.

Luke Hoban - TypeScript Co-Creator and CTO, Pulumi

Luke is CTO at Pulumi. He has worked on developer tools his entire career. He started out on Visual Studio, C#, and .NET in the early 2000s, later joined the ECMAScript standards body as a representative of Microsoft and then became one of the co-founders of the TypeScript programming language.

Featured Partners

Thank you to our amazing partners for supporting PulumiUP! Visit these booths for tech talks, demos and Q&A about your favorite cloud platforms and developer tools:

AWS - Solutions Architects will be on hand to demo the latest AWS and Pulumi capabilities and help you get the most from the AWS platform.
Google Cloud - Customer Engineers will help you accelerate your cloud journey with helpful tips and Q&A.
Microsoft - Cloud Advocates will be available to show you how to kickstart your projects on Azure.
Equinix - Stop by the Equinix booth to learn all about the new Equinix Pulumi provider.
CircleCI - Stop by the CircleCI booth to learn best practices for adopting CI/CD in your organization.
New Relic - The New Relic team will demonstrate how to get started with Observability as Code with Pulumi and New Relic.
Firefly - If you have existing cloud resources you can convert them to Pulumi code easily with Firefly. Stop by the Firefly booth for a demo.
Nitric - The Nitric team will be on site to show you how to create cloud-agnostic infrastructure code from your application code.
Klotho - The Klotho team will show you how devops teams an application teams can easily collaborate to moderize and scale infrastructure without disruptions.
Port - Join the Port team to learn how to setup a self-service portal for your developers.

Want to attend PulumiUP 2023? Register now!

How Elkjøp Nordic enables self-service infrastructure for developers

George Huang — Fri, 01 Jul 2022 00:00:00 +0000

At PulumiUP 2022, Tomas Jansson, software architect at Elkjøp Nordic, gave a presentation on how to enable developers to self-service infrastructure by using Pulumi’s Automation API.

Elkjøp Nordic is the leading consumer electronics retailer in the Nordics. The company sells consumer electronics, mobile phones, computers, white goods, domestic appliances, and services linked to these products both directly to consumers and to businesses. It is an omnichannel retailer and serves customers both online and through more than 400 stores. Elkjøp has retail outlets in Norway, Sweden, Finland and Denmark, and franchise operations in Greenland and Faroe Islands.

To support the company’s modernization strategy, Tomas developed a self-service infrastructure platform that would enable any developer to easily provision cloud infrastructure that uses approved resources and security practices. The goal was to empower teams at Elkjøp to increase their development velocity by using the cloud, while maintaining guardrails that uphold security and compliance.

Watch the full presentation below or read on to learn the highlights of his presentation:

The Problem: Balancing Agility and Governance

To achieve his goals, Tomas knew he had to solve the problem of balancing freedom with control. On the one hand, he wanted to foster the creativity of his developers and encourage innovation. He wanted his team to feel they were trusted and had the room to create the best solutions they could. On the other hand, he knew he had to maintain some control. Elkjøp needed to keep its applications secure. It also needed good governance so that teams can keep track of who is making changes. Tomas also wanted to limit the impact of deploying untested code. Code that was still being developed needed to be confined to the developer’s own environment. Finding the right balance between independence and control was paramount. Tomas believed this would result in great software and in developers who would enjoy their work.

The Solution: A Self-Service Infrastructure Platform

Tomas’ solution was to build a self-service infrastructure platform. Self-service infrastructure makes it possible for developers to easily access the infrastructure they need while limiting access to resources they don’t need. In effect, self-service infrastructure is a walled garden. Within the garden, developers have all the freedom they need. However, nothing they do affects what’s outside the wall. Code that causes unintended changes only affects them and no one else.

Tomas built an application that provides a “walled garden” of self-service infrastructure. Developers can create the environments they need, including an Azure resource group and a Kubernetes namespace, all through a graphical user interface. Developers can define whether the environment is for development, test or production. They can also assign values, such as which team members should have admin privileges to the associated GitHub repositories or who can contribute. From there, the platform handles provisioning and configuring all of these resources.

Here is an example that shows how a developer can create a repository simply by selecting a few parameters from a drop-down list.

Tomas determined that the best way to implement self-service infrastructure was with Pulumi and its Automation API. By using Pulumi, he could use standard programming languages to define infrastructure while using familiar software tools like IDEs and test frameworks. Pulumi also natively supports Azure and Kubernetes with 100% coverage of each provider’s APIs and same-day access to new features and updates. Finally, the Pulumi Automation API would enable him to program infrastructure automation directly into his application.

The Pulumi Automation API is a programmatic interface for running Pulumi programs without the Pulumi CLI. Automation API encapsulates the functionality of the CLI (pulumi up, pulumi preview, pulumi destroy, pulumi stack init, etc.) but with more flexibility since infrastructure operations can be controlled programmatically.

How the self-service infrastructure platform was built

Here is the workflow for the Elkjøp self-service infrastructure platform.

The self-service platform is a web application that runs the Pulumi Automation API on the server side. The application uses Pulumi to provision and configure every environment resource such as Azure resource groups, Kubernetes namespaces, service principals, and Kubernetes service accounts. Each environment comes with a Git repository that’s used by developers to manage their deployments. The Git repositories are the locus of each environment because they store everything needed to deploy changes such as infrastructure code and environment credentials.

Using the Pulumi Automation API, the application creates an Azure resource group, a team in both Azure AD and GitHub, and a new GitHub repository. The application then grants administrative access to the GitHub repo to members of the AD group. In addition to creating a Git repository for each environment, the application uses stack references for each Azure environment to inject the associated credentials into the GitHub Secret Store. These credentials are scoped to each GitHub environment so that developers can only work in their own environment. If a team has multiple environments, each environment has its own credentials so developers can’t, for example, deploy from a test environment to a production environment. Secrets are managed through the Pulumi Service.

Every new repository comes with Pulumi programs that developers can use to deploy their application. GitHub Actions initiate the deployment. When an action triggers, Pulumi reads the credentials that were generated by the self-service application, allowing Pulumi to deploy new Kubernetes or Azure resources to the environment.

Watch the demo

To illustrate how Elkjøp’s self-service platform works, Tomas presented a simplified demo application that’s based on Elkjøp’s production platform. The code for the application is available at https://github.com/mastoj/pulumi-automation-demo.

The demo is a TypeScript application that includes an API that provisions Azure and GitHub resources. Pulumi Automation API code is used to deploy new stacks by calling handler code, which is a Pulumi program that defines resources and outputs. The handler code can create Azure resource groups or GitHub repositories, and stack references are used so that outputs generated by the Azure resources can be stored in GitHub (e.g., resource group credentials).

Ready to dive into how this works in detail? Watch the demo in the video below starting around 10:20.

Next steps

After watching the demo of a self-service infrastructure platform, you can try it out for yourself by checking out the demo code. You can also learn more about Pulumi Automation API in our documentation.

If you’re new to Pulumi, then follow our getting started guide.

Pulumi and RedMonk on developer-first infrastructure and why it matters

George Huang — Tue, 26 Apr 2022 00:00:00 +0000

What do assembly languages and the cloud have in common? Are abstractions the future of cloud computing? What does “infrastructure” really mean? And why do these questions matter to the platform engineers, infrastructure engineers, and developers who are building modern cloud applications today?

Joe Duffy (Founder & CEO, Pulumi) and James Governor (Co-founder, RedMonk) recently answered these questions and more in a conversation about developer-first infrastructure. Developer-first infrastructure means empowering developers to build and deploy modern cloud applications and infrastructure through the use of software engineering practices that tame modern cloud complexity.

If you’re interested in learning how the worlds of software engineering and cloud infrastructure are intersecting in addition to principles and best practices for building in the cloud, then watch the 28-minute video below.

Highlights of Pulumi and RedMonk’s discussion on developer-first infrastructure

In the first half of the discussion, Joe and James discuss the complexities of building on the modern cloud as application architectures have evolved from simple three-tier apps to the modern, distributed applications of today. Joe traces the evolution of modern computing in which complexity has been tamed over time through abstractions: from low-level assembly languages to higher-level languages and runtimes that make software more accessible, and finally to operating systems. They then discuss how an analogous transformation has yet to occur in the cloud on a mainstream basis.

“The level of abstraction [that most people are doing] with cloud infrastructure reminds me of assembly languages. You look at writing these YAML templates that over-specify every detail of the underlying architecture. I think [the industry] can do better. I think we can be inspired by what we’ve done in programming languages.” — Joe Duffy

In the second half of the discussion, the two discuss what developer-first infrastructure is, and how companies can use this philosophy to tame modern cloud complexity and accelerate development velocity. Companies that execute this well have a “developer-first multiple” — that is, the most valuable companies in the world today are the ones who’ve thought hard about the developer experience for their own employees or the communities that interact with them.

Joe describes five principles of developer-first infrastructure that make developers’ lives easier when building on the cloud.

Cloud Engineering
Harmonized Applications and Infrastructure
Beyond “Lift and Shift”
Secure and Observable By-Construction
An API for Everything

Finally, Joe dives into the best practices of developer-first infrastructure:

Unified Engineering: Software engineering tools and practices for applications and infrastructure.
APIs: Making everything programmable and composable with APIs.
Everything as Code: Use “as code” techniques like infrastructure as code and policy as code.
Polyglot: Embrace the best language ecosystems for the job at hand.
Modularity: Tackle complexity with real sharing and reuse, not copy & paste.
Continuous Delivery: Deliver and scale applications and infrastructure with automated techniques.
Continuous Verification: Verify and enforce guardrails at all times.
Environment Parity: Share as much as possible between dev, test, and prod environments.
Securely Configured: Loosely couple and securely configure environments.
Distributed Architectures: Go beyond “lift and shift” to building truly cloud-native applications and infrastructure.
Observable: Instrument applications and infrastructure to log high cardinality events, early and often.

Next steps

Developer-first infrastructure means empowering developers to build and deploy modern cloud applications and infrastructure through the use of software engineering practices that tame modern cloud complexity. If you feel inspired after viewing the Pulumi/RedMonk discussion video, then consider applying developer-first principles and practices for your next greenfield project or application using an infrastructure as code tool like Pulumi. You can also watch Joe Duffy’s presentation on developer-first infrastructure from the Cloud Engineering Summit to learn more about developer-first.

Announcing PulumiUP 2022: The Cloud Awakens on May 4th

George Huang — Thu, 31 Mar 2022 00:00:00 +0000

Calling all cloud engineers! Today we announce the second annual PulumiUP virtual conference taking place on May 4, 2022. We will also host the inaugural Pulumi Community Summit on May 5 to virtually assemble the world’s cloud, infrastructure, and Pulumi practitioners of all experience levels (hint: that’s you—all are welcome)! Register for both events here.

We’ve worked hard to curate a half-day of learning and expert insights from the people that are forming the best practices of the modern cloud and cloud native infrastructure era. On May 4, join us for four hours of exciting Pulumi product news and demos, technical talks, case studies of modern cloud innovators, and conversations on cloud topics that you won’t hear anywhere else. Hear from Pulumi’s CEO and CTO, pioneering engineers and engineering leaders from different industries, and Pulumi’s engineers. We are also introducing two tracks designed for Practitioners and Engineering Leaders.

Who’s speaking at PulumiUP 2022?

Aaron Friel, Staff Software Engineer, Pulumi
Elad Ben-Israel, Principal Software Development Engineer, Amazon Web Services
Joe Duffy, CEO & Founder, Pulumi
Justin Fitzhugh, VP of Cloud Engineering, Snowflake
Kat Cosgrove, Staff Developer Advocate, Pulumi
Luke Hoban, CTO, Pulumi
Meagan Cojocar, Senior Product Manager, Pulumi
Tomas Jansson, Software Architect, Elkjøp Nordic AS

In the coming weeks, we will be announcing even more speakers. Stay tuned!

What’s in store at PulumiUP 2022?

Stay tuned as we will announce the full agenda and speakers soon on our website and in an email to registrants. Here’s what you can expect at this year’s event:

Keynotes: Hear from Pulumi leaders Joe Duffy (Founder & CEO) and Luke Hoban (CTO) as they share new product announcements, demos, and thoughts on trends in the modern cloud.

Technical Talks: Dive deep with Pulumi’s engineers and guest speakers from the industry to learn best practices for infrastructure as code with popular programming languages and see demos.

Case Studies: Learn from engineers and leaders about how they have accelerated development velocity and increased reliability for their modern cloud applications with Pulumi.

Expert Conversations: Listen in on lively debates and discussions about the intersection of software development and cloud infrastructure from panels made up of industry leaders and practitioners.

Two Tracks: Choose from two session tracks that are designed for Practitioners and Engineering Leaders, in addition to the keynote and main stage presentations.

Inaugural Community Summit: Connect and engage with Pulumi practitioners from around the world—from the comfort of your home. We are hosting a Community Summit on May 5 featuring lightning talks and discussions.

Workshops: Get hands-on with Pulumi and cloud engineering in 1-hour workshops that will cover the fundamentals of Infrastructure as Code as well as more advanced subjects for experienced Pulumi users.

Join us on May 4 and 5

Register for PulumiUP and the Community Summit now! We will send you updates on the final agenda and speakers and post-event updates such as session recordings and upcoming events.

Cloud engineering fuels the next chapter of startup innovation

George Huang — Thu, 07 Oct 2021 00:00:00 +0000

The story of how the cloud fuels startup innovation seems never ending. In the beginning, AWS birthed cloud computing with its first service, SQS, in 2004 and quickly released several additional services (like S3, EC2, and SimpleDB). From this innovation, startups flourished because they were able to build, experiment, and grow faster than before at much lower cost. Airbnb, Netflix, Zynga, and many more were born, and the rest is history.

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

Today, a new generation of startups is flourishing because of the cloud, but this time with modern cloud architectures that are distributed, API-driven, and more resilient and scalable than ever. Today’s startups have to get to market even faster and rapidly innovate in order to delight customers and carve out market share. Most startups understand the benefits of adopting the modern cloud to help them achieve this goal. However, their ability to reap these benefits for competitive advantage depends on how well they can harness the modern cloud.

Within the cloud engineering community, we see several common patterns for harnessing the modern cloud. Some startups have teams of full-stack developers who need to deploy cloud infrastructure and applications safely and at high velocity. Others might have a few infrastructure or platform engineers who need to enable other developers to use cloud infrastructure easily on a self-serve basis. Many of these teams started off using domain-specific languages (DSLs) to manage infrastructure as code and quickly found that these languages were the limiting factor in achieving faster velocity. DSLs are cumbersome to use and don’t support the logic and expressiveness needed to build and manage modern architectures that are more complex in nature. DSLs are also a barrier to entry to most developers.

In contrast, cloud engineering makes cloud infrastructure more accessible and easier to use for everyone with tried and true software engineering practices and tools. It embraces the reality that all software is now cloud software. A core practice is using a commonly used language like TypeScript, Python, Go, or C#. That’s one reason why more and more startups are choosing Pulumi to help them achieve competitive advantage through cloud infrastructure. It also doesn’t hurt that Pulumi was rated the “Top Paying Technology” by developers in 2021.

Cloud engineering is a central character in the next chapter of how the modern cloud fuels startup innovation. Read on for three mini-stories of cloud engineering in action with startups.

Panther Labs

Panther Labs helps modern security teams build world-class detection and response pipelines using code and automation, developer-friendly workflows, and big data primitives. Its Platform Team is responsible for a large, complex serverless architecture on AWS. Because of the limitations of its legacy Infrastructure-as-Code (IaC) tool, the team was unable to manage and scale its cloud infrastructure with the speed and automation that the company needed to support its fast-growing business. After comparing different alternatives, Panther Labs decided to migrate to the Pulumi Cloud Engineering Platform. Pulumi increased the company’s deployment speeds by up to 10x, reduced the size of its infrastructure codebase by >50%, and enabled its developers to adopt cloud engineering best practices to deliver its cloud applications faster and more reliably.

Results

Infrastructure teardowns were twice as fast, fresh deployments were four times as fast, and simple changes were more than ten times as fast.
The infrastructure codebase was reduced by more than 50%.
Developers could use a programming language and existing software development practices that they already knew.
The ability to create reusable components made it easy to replicate and share components that always followed cloud engineering best practices.
Central visibility across hundreds of single-tenant customer deployments, including a history of when changes were made and what changed for each account.

Read the full case study→

WhyLabs

WhyLabs helps organizations run their AI applications with certainty by monitoring them for problems such as data drift and model degradation. Founded by alumni of AWS, the company began building their platform on AWS and used AWS-provided tools to manage their infrastructure. However, the WhyLabs team needed a platform that could handle the complexity of their modern cloud architectures and would set them up for success when they’re ready to adopt multi-cloud deployments. The team also wanted a platform that integrated with the modern development workflows used by WhyLabs’ developers, who have a software engineering-driven culture of tight collaboration and shared responsibility for infrastructure. By migrating to the Pulumi Cloud Engineering Platform, WhyLabs engineers can continuously and reliably ship new features faster than before for improved time-to-market. Pulumi enabled WhyLabs to adopt cloud engineering best practices out-of-the-box, including building infrastructure as code with general-purpose languages like TypeScript, versioning and reviewing infrastructure code through a code review process, and delivering infrastructure through its existing GitLab CI/CD pipeline using Pulumi’s integrations.

Results

Pulumi enabled WhyLabs engineers to continuously and reliably ship new features faster than before for improved time-to-market.
Pulumi enabled WhyLabs to adopt standard programming languages (like TypeScript) to build infrastructure as code. It also allows them to use software engineering best practices like reusability and unit testing.
Pulumi lowered the barrier to entry for developers to start working with infrastructure. A standard programming language means that every developer can quickly become involved in creating and maintaining the WhyLabs infrastructure.
Pulumi enables WhyLabs to shift infrastructure left because testing and security controls are automated with Pulumi and GitLab integration. Peer reviews and accountability are built into the GitLab CI/CD pipeline.
Pulumi’s integration with GitLab enables WhyLabs to deliver infrastructure through CI/CD pipelines just like with application code, which increases iteration frequency and accelerates time to market.
Pulumi’s support for multiple cloud providers will help WhyLabs expand their platform in the future to meet customers’ needs.

Read the full case study→

Fauna

Fauna provides a flexible, developer-friendly, transactional database delivered as a secure and scalable cloud API with native GraphQL. It needed to provide a way for customers to use its databases while meeting data residency requirements, such as complying with GDPR or FIPS. This meant allowing customers to store data in specific geographies while maintaining low latency and high availability. To build this capability, Fauna’s engineers needed a repeatable, scalable, and efficient way to build, deploy, and manage databases on behalf of its customers. Every database is backed by a cluster of nodes across multiple regions and cloud providers. After evaluating CloudFormation and Terraform, Fauna found that these tools’ domain-specific languages and features would be insufficient for the project. Fauna chose Pulumi to build, deploy, and manage its multi-cloud infrastructure because Pulumi enabled cloud engineering best practices that met its requirements. This included the ability to use a familiar and powerful programming language like Python, integration with existing software development workflows and tools, and software engineering automation.

Results

Delivered its new multi-region, multi-cloud database feature by using familiar and existing software tools and practices that were enabled by Pulumi.
Built and managed its complex cloud architecture more efficiently by using familiar languages like Python.
Increased the reliability and quality of infrastructure using existing code review processes and running unit tests.
Controlled access to its infrastructure using Pulumi’s SSO integration with its identity provider service.

Read Fauna’s blog post→

Summary

Cloud engineering is enabling startups to build, deploy, and manage modern cloud infrastructure and applications faster and with more confidence. By applying standard software engineering languages, practices and tools to cloud infrastructure, startups can harness modern cloud architectures like containers or serverless to their full potential. To illustrate this, we shared three case studies of startups practicing cloud engineering. Pulumi’s Cloud Engineering Platform was built to enable any organization to adopt cloud engineering best practices out-of-the-box. If you’re starting your cloud engineering journey, try Pulumi for free today with this Getting Started guide.

Cloud Engineering on the Rise

George Huang — Tue, 03 Aug 2021 00:00:00 +0000

One of the most fulfilling aspects of working at Pulumi is learning how customers and the community practice cloud engineering in their teams. It’s exciting to see how they use cloud engineering and Pulumi to implement best practices that enable leveraging the cloud to accelerate innovation and enable better business outcomes.

Pulumi is born from the experiences and needs of teams practicing cloud engineering every day. When we announced the Pulumi Cloud Engineering Platform in April, CEO & Founder Joe Duffy talked about bringing cloud engineering to everyone. Over the past year we have seen significant growth in cloud engineering and usage of Pulumi across companies of all industries and sizes, and spanning a diverse spectrum of teams and engineering disciplines. We’re also seeing growing numbers of job postings with “cloud engineer” in the title or which have Pulumi as a requirement or desired skill set.

Recently, we published several case studies about how teams are applying cloud engineering best practices. Cloud engineers apply standard software engineering practices and tools uniformly across infrastructure management, application development, and security to tame the complexity of delivering and managing modern cloud applications. We’ve published on our website and in this blog some of the key cloud engineering best practices that we see broadly across the community, and encourage you to read further to see three stories of cloud engineering in action.

These three case studies highlight a small sampling of the many types of companies and teams representing cloud engineers. Some are platform teams at large organizations, like Atlassian, who are responsible for empowering the rest of their developer teams. We also see DevOps teams using cloud engineering, such as at Skai and SANS Institute. There are other types of teams, like full-stack engineers or SREs, which we will share in future posts. All of these teams share a software engineering mindset and make a leveraged impact on the business by increasing speed, automation, reliability, and agility through cloud infrastructure innovation.

Atlassian’s Bitbucket DevSpeed team is responsible for improving developer productivity through better workflows and tooling. It used Pulumi to make it easier and faster for its developers to use cloud infrastructure and reduced developers’ time spent on maintenance by 50%. DevSpeed originally needed a way to expand regional support for its deployment pipeline, which deploys instances on AWS for more than one hundred team members around the world. The team also wanted to build a self-service platform that provisions cloud infrastructure for Bitbucket’s developers, without having to learn a complex domain specific language (DSL). They moved from a legacy DSL-based infrastructure as code tool to the Pulumi Cloud Engineering Platform, which let them define and deploy infrastructure in general-purpose languages that Bitbucket developers already used, such as Python. Using a familiar language like Python also made it easy for them to add cross-regional support to their deployment pipeline. They also built a self-service dashboard that allows any developer to deploy and configure private Bitbucket instances for feature development.

Atlassian’s Key Practices

Builds infrastructure as code with Python.
Makes infrastructure code readable and accessible to all developers in the organization by using a common language like Python.
Created a self-service platform that deploys and configures private Bitbucket instances for any Bitbucket developer. Underneath, everything runs through a CI/CD pipeline with Pulumi, Bitbucket Pipelines, and Bamboo.
Manages and deploys updates to all developer instances in the organization using features like the Pulumi Automation API.

Atlassian’s Results

Allowed developers to use general-purpose programming languages like Python and familiar software tools to deliver and manage infrastructure.
Enabled Bitbucket developers to easily provision approved cloud infrastructure using a self-service dashboard built with Pulumi.
Reduced the time Bitbucket developers spent maintaining their instances from 8 hours per week to fewer than four.
Reduced the size and complexity of its infrastructure code compared to its previous infrastructure tool while increasing the code’s clarity.

Read the full case study→

SANS Institute is the global leader in cybersecurity training. Using Pulumi, the organization adopted cloud engineering practices that enabled it to reduce deployment times by 70%. Its DevOps team needed a more consistent and reliable way to build, deploy, and manage the SANS virtual training environments used in labs. It had previously used legacy infrastructure tools which require inflexible DSLs that aren’t suited to modern cloud applications. The limitations of the DSLs meant the team needed to use AWS Lambda functions and Bash scripts to fill in gaps. The team needed a cleaner solution built for the modern cloud that would be more scalable and easier to manage. By adopting Pulumi, SANS increased delivery speed, quality, and consistency using cloud engineering practices such as building infrastructure as code with TypeScript and software tools, and deploying cloud applications with Git and automated delivery pipelines. It plans to use Pulumi to help other SANS business units modernize their practices and tools.

SANS Institute’s Key Practices

Builds infrastructure as code with TypeScript.
Deploys infrastructure through a GitOps and CI/CD process while managing its Git branches using a GitFlow pattern.
Production deployments can be initiated through an API call through a self-service platform that deploys an application instance.
Manages infrastructure configuration through Policy as Code which checks for compliance.

SANS Institute’s Results

Reduced deployment times for game servers by up to 70%.
Simplified their deployment process by eliminating the need to “glue together” several infrastructure as code tools, scripts, and functions. This let them remove a manual ticketing step.
Enabled developers to use familiar programming languages and tools, which also helped make their code cleaner and - more uniform, resulting in streamlined pipelines.
Helped to identify opportunities in other SANS departments where Pulumi can streamline operations.

Read the full case study→

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers. Skai’s microservices run on AWS, but its core monolith service was hosted in a data center running private cloud infrastructure. Skai’s DevOps group needed to migrate the service to the cloud because demand increased. Skai’s DevOps team decided to transition from using its legacy infrastructure tool because its domain-specific language (DSL) wouldn’t be suited to the demands of the newly migrated modern cloud architectures. Instead, Skai sought a solution that enabled it to use familiar languages and software engineering best practices to manage its infrastructure. That’s why Skai chose the Pulumi Cloud Engineering Platform to build, deploy, and manage its infrastructure with Python.

Skai’s Key Practices

Builds infrastructure as code in Python.
Leverages reusable infrastructure components.
Multiple teams collaborate on cloud infrastructure, including an architect, application development teams, and a core platform/DevOps team.
Teams use a “loosely-coupled, tightly-aligned” approach to create component classes for different infrastructure elements, with each team working on a different element.
Uses Python code to integrate infrastructure processes with internal metadata and IT systems.

Skai’s Results

Migrated a core monolith service with hundreds of terabytes of data from a data center to AWS.
Refactored the service’s infrastructure from being managed with a legacy infrastructure configuration tool to Pulumi, using Python as the language of choice.
Created modular infrastructure components that enable different teams to collaborate and work on different parts of the infrastructure in parallel.
Streamlined the registration and deregistration process for new application instances by integrating Pulumi with its internal IT systems.

Read the full case study→

Summary

Cloud engineers are a fast-growing community made of tens of thousands of members and thousands of companies. They apply standard software engineering practices and tools uniformly across infrastructure management, application development, and security to tame the complexity of delivering and managing modern cloud applications. To illustrate cloud engineering in action, we shared three case studies of teams using cloud engineering in their organizations. Pulumi’s Cloud Engineering Platform was built to enable any organization to adopt cloud engineering best practices out-of-the-box. If you’re starting your cloud engineering journey, try Pulumi for free today with this Getting Started guide.