Pulumi Blog: Community

Adding PostgreSQL State Backend Support to Pulumi: A Community Contribution Journey

Matan Baruch — Thu, 19 Jun 2025 14:46:07 +0300

When managing infrastructure as code at scale, reliable state storage is essential. Pulumi Cloud provides a fully managed, secure, and scalable solution out of the box. For teams that choose to build and maintain their own backend, Pulumi now offers support for PostgreSQL as a DIY state storage option—though this requires additional operational overhead and careful consideration around performance, security, and maintenance.

The Need for Database-Backed State Storage

Traditional DIY backends in Pulumi have relied on object storage systems like AWS S3, Google Cloud Storage, or Azure Blob Storage. While these work well for many use cases, they have limitations when it comes to handling very large state files, complex locking mechanisms, and transactional guarantees that some enterprise environments require.

PostgreSQL stood out as an excellent candidate for state storage due to its:

Large object support: Ability to handle substantial state files without size constraints
ACID compliance: Robust transactional guarantees for state consistency
Mature ecosystem: Well-established tooling and operational practices
Scalability options: From single instances to complex replication setups
Security features: Comprehensive authentication and authorization capabilities

The Community Contribution Process

This feature came to life through PR #19581, which addressed a long-standing community request tracked in issue #5632. The development process showcased the collaborative nature of open-source development, with multiple rounds of feedback, testing, and refinement.

Key Technical Challenges

The implementation required several technical considerations:

1. Integration with Go Cloud Development Kit (CDK)

The PostgreSQL backend needed to integrate seamlessly with Pulumi’s existing blob storage abstraction layer. This was achieved by implementing the blob.BucketURLOpener interface:

// URLHandler is a URL opener for PostgreSQL URLs.
type URLHandler struct{}

// OpenBucketURL implements blob.BucketURLOpener.
func (p URLHandler) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error) {
 pg, err := NewPostgresBucket(ctx, u)
 if err != nil {
 return nil, err
 }
 return pg.Bucket(), nil
}

2. Database Schema Design

The implementation uses a simple but effective schema for storing state data:

CREATE TABLE IF NOT EXISTS pulumi_state (
 key TEXT PRIMARY KEY,
 data JSONB NOT NULL,
 created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
 updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

This design allows for efficient key-based lookups while leveraging PostgreSQL’s native JSON support for flexible state data storage.

3. Cross-Platform Testing

One of the significant challenges was ensuring the implementation worked across different operating systems and CI environments. The solution involved using testcontainers-go to spin up PostgreSQL instances during testing, with appropriate skip conditions for environments where Docker isn’t available:

func skipIfDockerNotAvailable(t *testing.T) {
 if runtime.GOOS == "windows" || runtime.GOOS == "darwin" {
 t.Skip("Skipping test: Docker not available on this platform in CI")
 }

 // Additional Docker availability checks...
}

Performance and Limitations

PostgreSQL wins over a plain S3 bucket

ACID transactions with row-level locking – A postgres backend means strong consistency guarantees.
Faster for smaller state files – Community benchmarks show improvements on S3 backend speed for smaller state files.
Point-in-time restore – WAL-based backups allow for easier rollback.
Deep observability – plug pg-metrics into Grafana/APM for real-time insight.

Gaps you’ll still close only with Pulumi Cloud

Server-side policy guardrails (CrossGuard) – automatically enforced policies on every preview or deploy.
Always-on drift detection – automatic scans catch out-of-band changes.
Managed Deployments – fully managed runners, PR previews, and dependent stack updates.
Org-wide RBAC, SSO/SAML, immutable audit logs – turnkey for teams and auditors.
Search & analytics across all cloud resources (Insights) – not just those managed by Pulumi.
Enterprise Secrets & Configuration (ESC) - encrypted secrets and config with RBAC, versioning and audit trail.
SOC 2, high availability, and painless upgrades – Pulumi operates the control plane so you don’t have to.

Getting Started

Using the PostgreSQL backend is straightforward:

Login to your PostgreSQL backend:

pulumi login postgres://pulumi_user:secure_password@localhost:5432/pulumi_state?sslmode=require

Use Pulumi normally:

pulumi up
# Your state is now stored in PostgreSQL!

Community Impact

This contribution demonstrates the power of community-driven development in open source projects. It addresses a real need expressed by Pulumi users while maintaining the high quality standards expected from the project.

The implementation follows Pulumi’s architectural patterns and coding standards, making it a seamless addition to the existing codebase. The comprehensive documentation and testing ensure that future maintainers can easily understand and modify the code.

What’s Next?

The PostgreSQL backend opens up several possibilities for future enhancements:

High Availability: Support for PostgreSQL clustering and replication
Performance Optimizations: Caching strategies and connection pooling
Advanced Features: Custom backup strategies and state analytics
Multi-tenant Support: Isolation patterns for multiple teams or environments

Conclusion

The addition of PostgreSQL state backend support to Pulumi represents more than just a new feature—it’s a testament to the collaborative nature of open source development and the power of community contributions. By providing enterprise-grade state storage options while maintaining the simplicity that makes Pulumi great, this feature enables organizations to adopt infrastructure as code with confidence.

Whether you’re running a small startup or managing infrastructure for a large enterprise, having robust, reliable state storage options is crucial. The PostgreSQL backend provides an improved approach to Do-It-Yourself state management.

If you’re interested in contributing to Pulumi or have ideas for new features, the community welcomes your contributions. This PR serves as an excellent example of how community members can make meaningful improvements to the project while learning from experienced maintainers and following established best practices.

Introducing the Pulumi Puluminaries 2.0 Program

Engin Diri — Mon, 10 Feb 2025 12:15:00 -0800

We are excited to announce the Pulumi Puluminaries 2.0 Program. This is a fresh and revitalized way to celebrate and support Pulumi’s most passionate community members. Pulumi Puluminaries are individuals who demonstrate leadership in the Pulumi ecosystem by sharing best practices, creating valuable content, and helping fellow practitioners succeed.

Before we dive into what is new, we want to recognize and applaud the incredible achievements of our existing Pulumi Puluminaries. You can check out the great folks currently making a difference in our community on the Pulumi Puluminaries page. Their hard work and dedication have laid a strong foundation for what is next.

What is the Pulumi Puluminaries 2.0 Program?

The Pulumi Puluminaries 2.0 Program is designed to recognize and empower developers, architects, and technologists who are committed to driving innovation with Pulumi. This program offers exclusive opportunities to collaborate with Pulumi engineers, gain early access to new features, and share insights with the community.

Similar to how other ambassador programs bring together dedicated advocates around a specific focus, the Pulumi Puluminaries 2.0 Program unites a community of individuals who champion Pulumi’s approach to cloud engineering. Pulumi Puluminaries lead conversations, help new users adopt Pulumi effectively, and showcase best practices and use cases that highlight the power of Infrastructure as Code. They are also key participants in shaping Pulumi’s future by providing feedback to the product team and taking part in special initiatives.

What does it mean to be a Pulumi Puluminary?

Pulumi Puluminaries serve as champions of cloud engineering. They help users navigate the complexities of building, deploying, and managing applications with Pulumi. They also promote the benefits of Infrastructure as Code in modern organizations by creating thought leadership content, hosting or co-hosting events, and offering guidance for those just getting started.

Key Responsibilities for Pulumi Puluminaries:

As a Pulumi Puluminary, you will be expected to engage with the community at least once a quarter. Here are some ways you can contribute:

Community Engagement
- Actively share Pulumi-related articles, tutorials, and project stories.
- Present at conferences, meetups, or webinars on topics that highlight Pulumi’s value.
- Mentor new users and help them get started with Pulumi.
- Participate in Pulumi’s community forums, Slack channels, and social media discussions.
Thought Leadership
- Write blog posts (like Simen A. W. Olsen did in his latest post Your Perfect Infrastructure May Not Be So Perfect or Tyler Mulligan) or create videos showcasing innovative ways to use Pulumi.
- Contribute to discussions around best practices and patterns for cloud engineering with Pulumi.
Product Feedback
- Offer beta-testing support and provide product feedback directly to the Pulumi team.
- Participate in early-access programs and help guide product roadmap decisions.

Benefits of Joining the Program

Early Access: Get sneak peeks at new Pulumi features and releases before they are publicly announced.
Direct Collaboration: Collaborate one-on-one with Pulumi engineers and product managers, influencing future capabilities.
Recognition and Swag: Receive official Pulumi Puluminaries 2.0 digital badges, exclusive swag, and spotlight opportunities on Pulumi’s blog and at events.
Networking and Growth: Build relationships with peers in the Pulumi ecosystem, deepen your technical expertise, and grow your professional network.

How to Get Involved

We are looking for Pulumi enthusiasts who have shown dedication and passion for empowering others in the Pulumi community. Whether you have built cutting-edge projects, delivered insightful conference talks, or helped users overcome technical hurdles, we would love to hear your story.

Whether you want to apply for yourself or nominate someone else, we’d love to hear from you!

Apply or Nominate Now

2024 Year in Review: Growth, Innovation and Appreciation

Sara Huddleston — Fri, 20 Dec 2024 06:11:09 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

It’s the end of 2024, and like everyone, we’re counting down until 2025 while looking back at our year. We’ve had a very exciting year, from unveiling a bold new vision for Pulumi to delivering cutting-edge updates across our Pulumi Cloud.

Here’s a look at how Pulumi has grown, evolved, and continued empowering teams worldwide.

A Unified Vision: The Pulumi Platform

This year, Pulumi unveiled a bold new vision for Pulumi—a comprehensive product suite that extends far beyond infrastructure as code. The Pulumi platform now consists of three core products:

Pulumi IaC: Open source infrastructure as code in any programming language. With over 100M downloads, 167% growth in contributions, and industry-leading innovations, Pulumi IaC continues to redefine cloud automation.
Pulumi ESC: Centralized secrets management and orchestration that scales. General availability this year brought features like SDKs, versioning, tagging, and integrations with 1Password and Kubernetes secrets operators.
Pulumi Insights: Visibility, intelligence, and control over all infrastructure resources. Insights 2.0 expanded support for resources created outside Pulumi IaC, delivering advanced graph visualizations and policy enforcement powered by Pulumi CrossGuard.

Three core capabilities unite these products:

Pulumi Copilot: Generative AI is used to manage cloud resources.
Pulumi CrossGuard: A policy-as-code engine for compliance and best practices.
Pulumi Deployments: Infrastructure task orchestration.

Over 3,000 customers, including Nvidia, BMW, Unity Games, and Tivity Health, leveraged these solutions to enhance velocity, save costs, and secure their infrastructure.

Highlights of Releases and Enhancements

Pulumi’s product innovations this year have empowered teams to automate, secure, and manage cloud infrastructure with confidence:

Pulumi ESC General Availability, redefining secrets management with features like dynamic credentials and hierarchical environments. New capabilities include secrets syncing with external stores like AWS Secrets Manager and Kubernetes operators for runtime integration.
Pulumi Insights 2.0 delivers AI-powered search, a comprehensive inventory of resources created by Pulumi IaC as well as other tools like Terraform and CloudFormation, and automated remediation capabilities powered by CrossGuard.
Pulumi Copilot leverages Generative AI to transform how teams diagnose and resolve IaC issues, providing instant feedback and deployment recommendations.
Pulumi Kubernetes Operator 2.0 introduced scalable, secure deployments with dedicated workspace pods, enhanced customization options, and improved stability under dynamic conditions.
Drift Detection and Remediation in Pulumi Cloud now automates drift detection for all 180+ supported providers, ensuring infrastructure consistency, enhanced security, and reduced operational risks.
Azure Classic Provider v6.0.0 updates included the latest upstream changes and ensured compatibility with Pulumi Azure Native Provider.
Google Cloud Provider v8.0.0 provides full resource coverage for the latest Google Cloud updates.
AWS CDK 1.0 expands compatibility with AWS CDK features, bridging the CDK and Pulumi ecosystems and enabling integration with constructs from AWS’s construct hub.
Pulumi Visual Studio Code Extension empowers developers with tools for debugging Pulumi programs, YAML language support, and direct ESC management within the IDE.
Pulumi Docker Containers enhances CI/CD workflows with versioned images, pre-installed tools like Poetry and ppm, and runtime flexibility for Python, Node.js, and .NET.

By the Numbers

Pulumi’s growth and impact this year have been amazing:

75,000+ Pull Requests: Driven by 5,600+ contributors.
3,000+ Customers: Empowering industries like AI, gaming, finance, and healthcare.
100M+ Downloads: Pulumi IaC adoption soared globally.
167% Contribution Growth: Outpacing other IaC tools like Terraform and OpenTofu.
71% Cloud Cost Savings: Customers like Tivity Health saved millions with Pulumi.

Community Achievements

Pulumi’s vibrant community has been at the heart of everything we’ve accomplished:

Global Pulumi User Groups (PUGs): With 20 groups across 9 countries, including new meetups in Chicago, Columbus, São Paulo, and Tel Aviv, our community continues to grow. These meetups collectively hosted over 6,396 members.
PulumiUP 2024: Over 8,000 registrants globally, with nearly 1,000 for the first-ever PulumiUP Europe.
Workshops: 72 workshops with 14,500+ registrations from engineers wanting to learn Pulumi.
22,000 GitHub Stars: A testament to our community’s passion and dedication.

Looking Ahead to 2025

To our users, customers, partners, Puluminaries, and employees, “thank you!” Your support has made 2024 an amazing year, and we’re excited for what’s ahead in 2025. Together, we’ll continue to build the future of cloud infrastructure and security.

Stay tuned for more innovations, community events, and collaboration opportunities. Here’s to another year of growth, transformation, and success!

Explore Pulumi ➡️

PulumiUP 2024: Dive Into the Future of Cloud, Platform Engineering, and AI/ML

Sara Huddleston — Wed, 11 Sep 2024 07:04:40 +0000

PulumiUP 2024 is just around the corner! It will be held on September 18th, starting at 8 AM PT | 15:00 UTC +0, and with over 5,500 engineers from all over the world already registered, this is shaping up to be the must-attend event for cloud professionals, platform engineers, and AI/ML enthusiasts alike. From entry-level engineers to tech executives, this event brings together professionals from companies of all sizes to explore the latest innovations and best practices in Cloud and IaC, Platform Engineering & DevOps, and AI/ML.

If you haven’t registered yet, now’s the time! Start building your schedule today, select the talks you want to watch live and on-demand and add them to your schedule.

PulumiUP at Glance

Main Track: Keynote, Cloud Culture Talks, and More

In the Main Track, you’ll hear directly from the Pulumi leadership team about exciting product innovations. Don’t miss the Cloud Culture talks focused on the human side of technology and collaboration.

Keynote and New Product Announcements - Joe Duffy (Co-founder and CEO), Luke Hoban (CTO), and other key Pulumi leaders will unveil new products and innovations shaping the next wave of cloud development. Expect to see live demos showcasing how these tools can revolutionize your workflows, from cloud infrastructure to security automation.
"Why Care About Building a Cloud Center of Excellence?" - Alex Radu (VP Product & Marketing Manager, Public Cloud, J.P. Morgan) discusses why building a Cloud Center of Excellence is essential for long-term cloud strategy success.
"What Is ‘HugOps’ and Why Is It Important?" - Rees Pozzi (Senior Platform Engineer, Kainos) explores the meaning and importance of “HugOps,” a practice promoting empathy and collaboration in the tech world but also by all those impacted by it.

AI/ML Track Highlights

Panel: AI for Cloud Development

AI is transforming how we approach development and infrastructure, and PulumiUP offers an incredible lineup of talks and panels on the topic.

"Data Privacy Challenges with Large Language Models" - Aditi Godbole (Data Science, AI & ML Leader, SAP) will address the critical question of managing data privacy, addressing LLM basics, AI data privacy fundamentals, and specific privacy issues in LLMs.
"Pulumi Powered AI/ML on Kubernetes" - Jason Smith (App Modernization Specialist, Google Cloud) will show you how to serve an open-source LLM with a RAG on Kubernetes and set up a Kubernetes environment for AI/ML workloads with Pulumi.
"AI Tools for Developers" - Jim Clark (Principal Software Engineer, Docker) will explore how Tools can enhance the capabilities of AI assistants and show how the combination of AI, Tools, and containerized runtimes, can enhance interactions between developers and their tools.
"The AI Governance Challenge" - Patty O’Callaghan (Technical Director, Charles River) will explore the complexities of AI governance and how to navigate them.
Panel Discussion: “AI for Cloud Development” - Luke Hoban (CTO, Pulumi), Phillip Carter (Principal Product Manager, Honeycomb), Giri Sreenivas (CPO, Docker), Clare Liguori (Senior Principal Software Engineer, AWS) and Meagan Cojocar (Principal Product Manager, Pulumi) will provide diverse perspectives on how AI is shaping cloud infrastructure.

Platform Engineering and DevOps Track Highlights

Panel: Secrets and Policies—Automating Cybersecurity

The Platform Engineering & DevOps track at PulumiUP is perfect for anyone looking to automate infrastructure, secure their systems, and scale their environments with ease. This track features talks and panels on cutting-edge technologies and practices.

"Moving Mountains: How IaC Unlocks Massive Refactoring Possibilities" - Shaun Verch (Infrastructure Engineer, Oso) will show you how infrastructure as code can transform large-scale refactoring projects.
"Using Pulumi to Empower Kubernetes Fleet Management" - Blake Romano (Senior Software Engineer, Imagine Learning) will show you how Imagine Learning leverages Pulumi to manage a fleet of Kubernetes clusters and other platform resources quickly and reliably, and how it revolutionized their workflow.
"Streamline Your Infrastructure Deployment with GitOps and Pulumi Operator" - Sam Cogan (Solutions Architect & Azure MVP, WTW) will highlight how you can define your infrastructure in Git, and have Pulumi automatically reconcile it into deployed infrastructure.
"Is This a Platform? Platform Engineering Before PMF" - Jk Jensen (Team Lead, Staff Software Engineer, MystenLabs) will share how they built their service platform around Pulumi to help engineers and researchers move from prototypes to deployed environments, enabling their teams to iterate more quickly and develop an increasing number of systems without additional complexity.
"Security Automation—Faster. Cheaper. Better." - David Giambruno (VP of Engineering and DevOps, Tivity Health) and Joe Duffy (Founder/CEO, Pulumi) will discuss security automation and how these efforts impacted speed and agility, cost optimization and improved security and compliance.
"Meet Devs Where They Are: Why IaC Must Be Real Code" - Jeremy Adams (Head of Ecosystems, Dagger) will present why real code (that teams already use for applications) is essential for both IaC and CI/CD pipelines, and to meet modern DevOps best practices.
"Security of IaC Pipelines and Infrastructure Governance With Policies-As-Code" - Marina Novikova (Senior Partner Solutions Architect, AWS) and Andy Taylor (Senior Networking Specialist/ Solutions Architect, AWS) will explore security best practices for IaC pipelines and infrastructure governance using policies-as-code, cover best practices for handling sensitive information like access keys, tokens and encryption keys, and scaling and automating your DevSecOps.
Panel Discussion: “Secrets and Policies—Automating Cybersecurity” - Arun Loganathan (Senior Product Manager, Pulumi), Maya Kaczorowski (PM of Security), Jason Meller (VP, Product, 1Password), and Ofir Cohen (CTO, Container Security, Wiz) will explore the intersection of innovation and security, offering insights into how AI is reshaping the threat landscape and how organizations can stay ahead.

Cloud and IaC Track Highlights

Panel: Infrastructure as Code - Can we do better?

The Cloud and IaC track is ideal for those wanting to deepen their understanding of infrastructure as code and its impact on modern cloud computing. Learn from industry pioneers and see where the future of IaC is headed.

"From CDK to Pulumi: Evolution of SST" - Dax Raad (Founding Engineer, SST) and Jay V (Founder, SST) will share insights into the early development of their product, initially built on AWS Cloud Development Kit (CDK), the challenges they encountered and the groundbreaking decision to transition to Pulumi.
"Unified Software Development at BMW Software Factory with Pulumi" - Jan-Peter Alten (Expert Software Engineer, DevOps, BMW Group) will showcase how BMW uses Pulumi to unify their software development processes, resulting in an increase in developer productivity, streamlined infrastructure management, and enhanced scalability, security, and compliance across their complex software ecosystem.
"Stacking Accounts: Multi-Account Deployments in a Single Stack for Monitoring, Management, and More" - Denis Willett (Software Engineer, North Carolina Institute for Climate Studies) will explain how they use Pulumi for managing 60+PB of environmental data across AWS, GCP, and Azure and leverage multi-account deployments in a single stack via the Pulumi Automation API.
"Industrialize the Configuration of Your GitHub Repositories With IaC" - Alexandre Nédélec (Software Engineer, Avanade) will live code using the GitHub, Azure native, and Azure AD Pulumi providers to create a GitHub repository and configure its pipeline to deploy to Azure using OpenID Connect.
Panel Discussion: “Infrastructure as Code — Can We Do Better?” - Joe Duffy (Co-Founder and CEO, Pulumi), Brian Grant (CTO, Stealth), Elad Ben-Israel (Co-Founder and CEO, Winglang), Adam Jacob (Co-Founder and CEO, System Initiative), and Luke Hoban (CTO, Pulumi) will discuss the past, the present, and the future of Infrastructure as Code and the overall infrastructure technologies.

Workshops and Hands-On Learning

Want to get hands-on experience? Our live workshops are designed to deepen your understanding of trending topics, Pulumi, and its applications.

"Managing Team Secrets with 1Password & Pulumi ESC" on September 25, led by Phil Johnston (1Password) and Diana Esteves (Pulumi).
"Securing IaC Pipelines in Regulated Industries" on September 26, led by Josh Kodroff (Pulumi) and Marina Novikova (AWS).
"Deploy Tailscale infrastructure in AWS with Pulumi" on October 15, led by Diana Esteves (Pulumi) and Lee Briggs (Tailscale).
"Observability as Code for AI Apps with New Relic and Pulumi" on October 30, led by Diana Esteves (Pulumi) and Harry Kimpel (New Relic).

Amazing Partners and Sponsors

Thank you to our amazing customers, sponsors, and partners for supporting PulumiUP!

AWS The world’s most comprehensive and broadly adopted cloud, offering over 200 fully featured services from data centers globally.
Google Cloud Help developers build quickly, securely, and cost effectively with the next generation of modern infrastructure designed to meet specific workload and industry needs.
New Relic Data for engineers to monitor, debug, and improve their entire stack.
Docker Accelerate how you build, share, and run applications.
1Password Streamline how you manage SSH keys, API tokens, and other infrastructure secrets throughout the entire software development life cycle.
Tailscale Make creating software-defined networks easy: securely connecting users, services, and devices.
Pinecone Build remarkable GenAI applications fast, with lower cost, better performance, and greater ease of use at any scale.
Wiz Secure everything you build and run in the cloud.
Honeycomb Observability that helps solve problems you couldn’t before.
Mysten Labs Building critical infrastructure to enable a more decentralized internet.
Imagine Learning Empowering educators to inspire breakthrough moments in every student’s unique learning journey with digital-first, K–12 education solutions.
Dagger Powerful, programmable open source CI/CD engine that runs your pipelines in containers.
Oso Authorization as a service.

Build Your Schedule Now

With so many compelling topics across AI/ML, Platform Engineering, DevOps, and Cloud and IaC, now is the time to build your PulumiUP schedule. Select the sessions that match your goals and interests to maximize your experience. Plus, you’ll receive a link to watch your chosen panel discussions and tech talks on-demand!

Haven’t registered yet? It’s not too late! Join 5,500 engineers worldwide in this groundbreaking event and gain the insights you need to drive innovation at your company. Register now->

We can’t wait to see you there!

Platform Engineering: Cloud-Native, Maturity Models, and Platforms as Products

Sara Huddleston — Fri, 26 Jul 2024 17:40:17 +0000

The Platform Engineering & DevOps in-person series launched in Berlin with two great speakers. This blog article is an overview of Dominik Kress’s talk, “What the Heck is the CNCF Platform Working Group? Answers from a Member!” in which he discussed Cloud-Native Platforms, The Platform Maturity Model, and approaching Platforms as Products.

In this article, you’ll learn more about platform engineering and how to get involved with the CNCF Platform Working Group. Make sure to check our Pulumi User Groups (PUGs) to find a meetup near you.

On this platform engineering article:

What is a Platform?

According to CNCF, a platform in cloud-native computing is an integrated collection of capabilities designed to meet users’ needs. It serves as a cross-cutting layer, ensuring a consistent experience integrating and managing services and capabilities for various applications and use cases. A good platform provides consistent user experiences through portals, project templates, and self-service APIs.

Atlassian describes platform teams as creators of capabilities used by multiple product teams, reducing their overhead and cognitive load. Martin Fowler and Evan Bottcher define a digital platform as a foundation of self-service APIs, tools, services, and support arranged as an internal product. These platforms enable autonomous teams to deliver features faster with less coordination.

The specific capabilities of a platform are tailored to the needs of its stakeholders and users. While platform teams provide these capabilities, they do not always implement them directly. Managed service providers or dedicated internal teams can maintain the underlying implementations, ensuring consistency across the organization. Platforms are tailored to an enterprise’s internal users and are particularly relevant for cloud-native architectures, separating supporting capabilities from application-specific logic.

Defining Cloud Native Platforms

The core of the discussion at the CNCF Platform Working Group is: “What exactly is a cloud-native platform, and why do organizations need one?” A cloud-native platform is a strategic business initiative that enables organizations to deliver value more efficiently and effectively. It is characterized by:

Vendor-neutrality: A cloud-native platform should be open and agnostic, allowing organizations to leverage various technologies and services without being locked into a single vendor. Scalability and flexibility: The platform should be designed to accommodate the organization’s growing and changing needs, with the ability to add or remove components as needed easily.
Developer-centricity: The platform should empower developers to be more productive by providing self-service capabilities, streamlined workflows, and a consistent developer experience.
Operational efficiency: The platform should simplify and automate the management and maintenance of the underlying infrastructure, allowing the organization to focus on delivering value to customers.
Continuous improvement: The platform should be designed with a mindset of continuous evolution, with regular updates and enhancements to address emerging needs and technologies.

By aligning with these principles, organizations can achieve faster time-to-market, improved developer productivity, and reduced operational overhead.

The Platform Maturity Model: Charting the Path to Success

Dominik Kress at the Berlin Pulumi Group meetup explaining the platform maturity model

The Platform Working Group has developed a Platform Maturity Model. This framework can help assess the current state of a cloud native platform and identify steps to enhance its maturity. It defines five levels:

Ad-hoc: At the lowest level, the platform is ad-hoc, with no dedicated team or budget and a reliance on individual efforts and workarounds.
Operational: The platform has a dedicated team and budget, which focus primarily on operational tasks and maintenance rather than strategic initiatives.
Scalable: The platform is now viewed as a product with a clear roadmap and a focus on delivering value to internal customers. The platform team has more autonomy and the ability to drive innovation.
Enabled: The platform has become a strategic asset, with a thriving ecosystem of contributors and consumers. The platform team is empowered to make decisions and drive the platform’s evolution.
Optimized: The platform is continuously optimized, focusing on measuring and improving key performance indicators. The platform is deeply integrated into the organization’s technology strategy and decision-making processes.

By using the CNCF’s Platform Maturity Model as a guide, organizations can assess their current level of platform maturity, identify areas for improvement, and develop a roadmap to reach a higher level of platform success. This model has proven to be a valuable tool for many organizations, helping them to align their platform initiatives with their broader business objectives.

Platforms as Products: Driving Success

One of the key initiatives currently underway within the Platform Working Group is the “Platforms as Products” research project. This effort aims to explore the parallels between traditional product management and the management of cloud-native platforms to identify best practices and patterns that can help organizations unlock their platforms’ full potential.

The core premise of the “Platforms as Products” initiative is that successful cloud-native platforms should be treated and managed like products rather than just technology projects. This means applying principles of product management, such as:

Defining a clear vision and roadmap: Establishing a long-term vision for the platform and a roadmap to get there, focusing on delivering value to internal customers.
Measuring and optimizing for key metrics: Identifying and tracking the right performance indicators to ensure the platform delivers the desired outcomes.
Fostering a thriving ecosystem: Encouraging and enabling a community of contributors and consumers to drive the platform’s evolution and adoption.
Continuously iterating and improving: Embracing a continuous improvement mindset, with regular updates and enhancements to the platform based on user feedback and emerging needs.

By adopting a product management approach to their cloud native platforms, organizations can better align their platform initiatives with their broader business goals and ensure that the platform delivers tangible value to the organization.

Conclusion

This article and Dominik Kress’s talk highlighted the importance of platform engineering in cloud-native computing. Key points included defining cloud-native platforms, assessing platform maturity, and managing platforms with product management principles.

If you’re interested in learning more about platform engineering and getting involved in CNCF’s initiatives, there are several ways to participate:

Register for the Platform Engineering workshop series course.
Join the bi-weekly CNCF Platform Working Group meetings every other Tuesday at 5 PM Berlin time (11 AM ET).
Learn how to build Developer Portals with Pulumi. You can get off the ground faster with organization templates, a new project wizard, or leveraging the backstage plugin.

Frequently Asked Questions

What is the CNCF Platform Working Group?

Within the CNCF, the Platform Working Group is a collaborative effort focused on the critical topic of cloud native platforms. As a working group member, you can share insights and experiences from our discussions.

The Platform Working Group is part of the CNCF’s App Delivery Technical Advisory Group, which includes sustainability, runtime, observability, and security initiatives. The working group is open to anyone interested in contributing to the conversation around cloud native platforms, and its members come from diverse backgrounds, including product managers, engineers, and industry experts.

Who is Dominik Kress?

Dominik Kress is a product manager at Giant Swarm and a CNCF Platforms Working Group member.

He is on a mission to simplify developers’ lives by delivering intuitive developer platforms. He has been in the IT industry for over seven years, starting his journey as a Full Stack Software Engineer, falling in love with DevOps and Product Organisations to become a Product Manager for Cloud Technology later.

Dominik is passionate about Cloud Transformation, Product Management, and helping people. He loves contributing to the community with talks, articles, or publications.

How can I participate in the Platform As A Product research?

The “Platforms as Products” research project is currently underway, with the Platform Working Group conducting interviews and gathering insights from organizations that have successfully implemented cloud native platforms. The goal is to distill these learnings into a set of best practices and patterns that can be shared with the broader community.

To participate in this research, sign up here.

Platform Engineering & DevOps Series Kickoff Announcement

Sara Huddleston — Mon, 15 Jul 2024 14:38:46 +0000

We are excited to announce the kickoff of the Platform Engineering & DevOps Series, which will run from July 16 to October 31. This series will feature in-person events across various cities, including Berlin, London, Paris, Sydney, Boston, San Francisco, Seattle, Austin, Denver, and NYC. For those unable to attend an in-person event, we encourage you to participate in a virtual DevOps & Platform Engineering workshop.

Upcoming In-Person Events

Pulumi is hosting in-person meetups and virtual workshops for engineers interested in Platform Engineering and DevOps, covering topics from best practices to setup and implementation to advanced topics. Join us for tech talks, hands-on workshops, and opportunities to network with fellow engineers.

First Platform Engineering & DevOps in-person event: Berlin - July 16

Join Dominik Kress (Product Manager at Giant Swarm and member of the CNCF Platforms Working Group) and Guy Menahem (Solutions Architect at Komodor and community leader at The Platformers) at LOBE Block for an engaging discussion on platform engineering.

Dominik Kress will present “What the Heck is the CNCF Platform Working Group? Answers from a Member!”
Guy Menahem will present “For This Backstage, You Don’t Need a Special Pass: Platform Engineering in Practice.”

For more details and to RSVP, visit the Berlin Meetup page.

Second In-Person Event: Boston, July 24

Join Pulumi’s founder & CEO, Joe Duffy, and Jamie Baldanza, Director of DevOps at Relay Therapeutics, for the launch of our Boston meetup at The Foundry. This is a great opportunity for DevOps engineers, platform engineers, software engineers, and Cloud enthusiasts to connect and share insights.

RSVP is required due to limited capacity. Secure your spot, and RSVP at the Boston Meetup page.

More coming soon!

We expected Seattle, Paris, and London’s meetups to happen in early September. Details for the other in-person events will be coming soon, so stay tuned for more updates at Platform Engineering & DevOps Series.

No upcoming in-person events in your area? Attend one of our virtual workshops and keep up with the latest in DevOps & Platform Engineering!

Virtual Workshops

Getting Started with CI/CD for AWS using GitHub Actions: July 17. Learn more and register here.
From Zero to Production in Kubernetes: July 25. Learn more and register here.
Advanced CI/CD for AWS using Pulumi and GitHub Actions: August 14. Learn more and register here.
Mastering Platform Engineering: From Setup to Scaling Success: TBD.

Join us for tech talks, hands-on workshops, and opportunities to network with industry peers. We are excited to meet you in person or virtually. See you there!

How AI is Transforming DevOps: AI Talks for DevOps Insights

Sara Huddleston — Mon, 13 Nov 2023 22:05:34 +0000

The integration of artificial intelligence (AI) with DevOps signals a new era in software development. DevOps possesses unique characteristics and needs that make it exceptionally compatible with AI augmentation. Given that code fundamentally relies on language, and large language models (LLMs) serve as the core of GPT functionality, these models are particularly well-suited for tasks such as code generation. This article unwraps the topics addressed during our “AI: Friends or Foe | AI Talks for DevOps” event in San Francisco.

How will AI Impact the Future of DevOps

In the Fireside chat, Luke Hoban expressed his enthusiasm for incorporating AI into DevOps, emphasizing its potential utilization by both DevOps professionals and the development team. In contrast, Nana Janashia, TechWorld with Nana, adopted a more cautious stance, drawing attention to potential risks, particularly regarding reliability.

What is DevOps in AI

When considering the adoption of AI and AI engineering, such as prompt engineering and other specific disciplines, AI engineers rely heavily on the cloud. Many AI-based systems are built around cloud-native tools, practices, and managed services.

The extensive use of cloud-native primitives means that AI is fundamentally a cloud adoption story. Consequently, established DevOps and infrastructure as code tooling and best practices play an important role in AI engineering, in what is being done, and in leveraging its use.

What is AI in DevOps

Integrating AI in DevOps entails employing machine learning (ML) and other artificial intelligence technologies to automate and optimize various software development and delivery aspects, including tasks such as automating testing and deployment processes and refining resource management.

The incorporation of AI in DevOps can result in:

Enhanced speed
Improved resource management
Increased reliability throughout the software development lifecycle

Consequently, organizations may experience swifter deployments, reduced errors, and heightened overall productivity. By substituting manual processes with automation and AI-powered solutions, DevOps teams can elevate product quality and more efficiently oversee their systems.

How AI is Currently Used in DevOps

Luke Hoban at the San Francisco Pulumi Group meetup discussing the usage of AI, during the AI Talks for DevOps

DevOps significantly impacts all stages of the development and operations lifecycle. It has helped break silos and optimize to increase developer velocity and productivity, creating a positive impact on the ops side.

AI assists DevOps teams to refine their operations by detecting inefficiencies and triggering warnings as soon as issues appear. This technology also enables real-time observation of systems and applications, permitting prompt actions from operations personnel and minimizing downtime.

AI-driven automation can also help the software development process with code generation. The operations teams from both departments quickly pinpoint any issues in their workflows while speeding up decision-making processes.

In the long term, the AI may use cloud credentials to create and autonomously automate deployment and management and perform DevOps tasks on behalf of the DevOps or platform engineering team. In the near term, it is used to assist humans, from DevOps professionals to developers.

“In the near term, AI usage in DevOps is about how do you keep the human in the loop, how do you use it as a learning tool, as a code generation tool, and for auditing to provide actionable notifications, all of which are still very human-dependent. There is so much more it can be used for. Currently, it’s a tool serving humans. But it is fascinating to imagine it in the future when it can be trusted to be an autonomous agent.” - Luke Hoban, CTO of Pulumi and co-creator of TypeScript

So, how does this technology bring about these advancements? Two main ways are by streamlining development tasks and improving monitoring and security protocols.

Streamlining Development Workflows

AI has the potential to significantly change the software development life cycle by automating routine processes, using resources efficiently, and offering solutions for improved code. DevOps teams can leverage AI-driven tools to increase effectiveness when managing their assets while also improving code quality, leading to faster project deliveries.

For instance, AI can identify any slowdowns during progress and recommend improvements, analyze root causes, and help make judicious decisions regarding resource allocation, ultimately resulting in efficient delivery of software products.

Enhancing Monitoring and Security

The usage of AI significantly boosts DevOps monitoring and security. With the assistance of AI tools, operations teams can use machine learning algorithms to examine a massive amount of data to protect their applications from potential risks and enhance reliability. These algorithmic solutions provide valuable advice for development personnel, which helps strengthen protection measures while optimizing performance simultaneously.

How Can a DevOps Team Take Advantage of Artificial Intelligence

A DevOps team can leverage AI in several ways to enhance efficiency, automation, and overall productivity. Here are some key strategies for a DevOps team to leverage AI:

Code Generation and Learning

AI can be a powerful tool to make the development process much smoother. It can help engineers focus on what matters instead of searching for documentation on websites, developer forums, or search engines for answers to help solve specific problems. The perfect code snippet is often hard to find or may not even exist.

Generative AI, powered by LLMs, can help developers and DevOps engineers quickly discover, learn, and, for example, use new cloud infrastructure APIs.

Pulumi AI response to Run metabase in Azure, using TypeScript

You can see it in practice using Pulumi AI. You can use natural language to ask for a specific cloud infrastructure goal and obtain a corresponding Pulumi program in your preferred programming language. Moreover, Pulumi AI facilitates the iterative refinement of your cloud infrastructure, incorporating new features, enhancements to security and performance, resolution of correctness issues, and clarification of requirements. Try Pulumi Neo.

Intelligent Resource Management

Platform and developer teams are spinning up resources across many cloud accounts within multiple cloud providers. It can be almost impossible to aggregate and search your infrastructure across these accounts since every cloud and every account is (by design!) a walled garden.

Often, to answer key questions about their business, such as “Which team has the largest cloud footprint?”, teams need to combine data across various clouds, cloud accounts, and SaaS tools. This prevents organizations from getting quick answers to operational questions and barriers to building aggregations to derive insights. AI can provide actionable insights over infrastructure with natural language processing (NLP) queries.

“I can get developers using IaC immediately with Pulumi Deployments and its GitHub integration, while Pulumi Insights [Resource Search] makes it really easy to find idle developer environments that need to be shut down, which reduces our cloud costs” - Santosh Dornal, Head of Software Test & DevOps, Alkira.

When using Pulumi Cloud, you can leverage AI Assist. You can provide a plain English (Spanish, French, or Japanese) request to express queries where you might not know the exact syntax, type tokens, or package names. AI Assist makes it easier by providing answers to queries like:

show me all s3 buckets not tagged in production
show azure and azure native security groups
show all AWS VPCs

Remediation Tool: Continuous and Automatic Compliance

Defining and managing security rules, criteria, and conditions are key to enabling security at scale. Policy as Code, or Compliance as Code, has the ability to verify and spot problems before deploying the infrastructure. When policies are written with code, you can apply software development practices such as testing, automated deployment, and version control.

AI can be valuable not just in anomaly detection but also in remediation, which doesn’t just warn or error but can change the infrastructure directly to remediate compliance gaps. With Pulumi, Remediation policies can be configured and applied across any subset of stacks within an organization.

Consider the unfortunate incident at GitLab, marked by a disastrous sequence of events resulting in the loss of segments of production databases and substantial user data.

In this scenario, AI could have swiftly pinpointed the root cause and proposed a rapid solution, potentially mitigating the extent of damage before it escalated.

While the benefits of writing policies as code are evident for DevOps engineers, the organizational benefits are even more significant.

Enhance Collaboration and Communication

Nowadays, many professionals with DevOps in their titles are held accountable for the multiple aspects of Dev + Ops. At its core, DevOps is a cultural movement that removes barriers among teams and promotes collaboration and shared responsibility between dev and ops.

DevOps teams have seen great benefits from AI-driven tools, particularly in improving transparency, encouraging feedback, providing quick answers to common queries, and automating routine tasks. Many major companies, like Microsoft, use AI to build efficient workflows that make better decisions, leading to increased business outcomes. This includes smoother application delivery, a prime function of DevOps initiatives.

Improved Product Delivery

Companies incorporating AI into their DevOps strategy reap the rewards of accelerated product delivery. This has been made possible through improved workflows, automated testing, and strengthened security measures, leading to greater efficiency in software development by allowing development and operations teams more scope for dealing with complex tasks using machine learning models. They can make better decisions, resulting in faster product delivery and enhanced business outcomes.

By streamlining existing systems within this context, customers can benefit from superior products delivered faster than before while companies provide and reinforce their business value.

Planning and Cost Optimization

Implementing AI algorithms or AI tools that analyze usage patterns can assist in predicting future resource needs and optimize cloud infrastructure costs. This also ties in with intelligent resource management mentioned before, which can enhance cost efficiency, optimize resource allocation, and prevent unnecessary expenses.

Top 5 Skills for DevOps Engineers in the AI Era

Adora Nwodo with Nana discussing skills that DevOps engineers will need to learn to make AI work

DevOps engineers must open new skills, mainly in cloud services, data, and AI/ML.

Programming Languages: Although DevOps don’t usually need to know general programming languages in-depth, they should know what is most used by application teams. It’s a nice-to-have to find errors in CI/CD pipelines and fix them themselves as needed, but it is a must-have if the goal is to work MLOps.
Infrastructure as Code: Automation is important for DevOps and critical for MLOps. IaC, using familiar languages, has been used since the early days of MLOps and is a key component to solving the cloud orchestration challenge present in AI/ML. Dive into more detail in The Real AI Challenge is Cloud, not Code!
Continuous Integration and Delivery: CI/CD is mandatory knowledge, and it must be understood in detail to predict data preparation code, AI/ML code, CI/CD code, and infrastructure-related code.
Custom containers: As a DevOps professional, you are expected to understand Docker and/or Kubernetes concepts and best practices.
Orchestration: This is the holy grail of Ops and your key to unlocking AI.

How to Overcome AI for DevOps Challenges

The integration of AI in DevOps brings forth several challenges that organizations must navigate to leverage the full potential of these technologies. Here are some key challenges and potential strategies to overcome them:

1. Skills and Expertise

Challenge: AI & ML expertise is often scarce within DevOps teams, leading to a skill gap in implementing and managing AI-driven tools.

Overcoming Strategy: Organizations will need to hire professionals with AI and ML expertise, or existing team members will need training in machine learning, becoming proficient in MLOps combined to DevOps principles. Collaboration between DevOps and data science teams can also bridge the knowledge gap.

2. Integration Complexity

Challenge: Integrating AI seamlessly into existing DevOps workflows can be complex, given the diverse technologies and processes involved.

Overcoming Strategy: Prioritize a phased integration approach, gradually introducing AI capabilities into specific stages of the DevOps pipeline. This allows teams to adapt without disrupting established processes.

3. Scalability

Challenge: As AI applications within DevOps expand, ensuring scalability becomes crucial to accommodate growing workloads.

Overcoming Strategy: Design AI solutions with scalability in mind, utilizing cloud resources and containerization. Regularly assess performance and adjust infrastructure to meet evolving demands.

4. Data Quality and Availability

Challenge: AI models rely heavily on high-quality and readily available data. Inconsistent or inadequate data can lead to inaccurate predictions or suboptimal performance.

Overcoming Strategy: Prioritize data quality and establish robust data pipelines. Conduct regular audits to ensure data accuracy and implement strategies for handling missing or incomplete data.

5. Cultural Resistance

Challenge: Resistance to change or skepticism about the benefits of AI within DevOps and in software development can impede successful integration.

Overcoming Strategy: Foster a culture of innovation and continuous improvement. Provide clear communication about the benefits of AI, involve team members in the decision-making process, and showcase success stories to build confidence.

Navigating these challenges requires a thoughtful and strategic approach, focusing on collaboration, skill development, and the gradual incorporation of AI into existing DevOps practices. Regular evaluation and adaptation are essential to ensure a successful and sustainable integration of AI in the DevOps environment.

Conclusion

AI in DevOps can transform software development by making workflows more efficient, monitoring and security stronger, and overall effectiveness increased. Organizations that embrace AI/ML aim for better collaboration between teams, automating and intelligently managing infrastructure, reducing time-to-market, and increasing developer productivity and cost savings due to streamlining processes within their development lifecycle. AI and ML knowledge will become necessary skills expected to be applied to enhance and benefit the development, security, and operation teams as a whole. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Can AI replace DevOps?

DevOps has not been taken over by AI yet. Rather, its use has made a difference in code quality and accelerated certain processes. Improvements to the quality of coding have been seen as one of the main benefits brought on through integration with artificial intelligence.

What is an example of AI in DevOps?

AI can be utilized to make DevOps processes more efficient by creating test cases and pipelines, assessing infrastructure configurations for security concerns, and deploying applications. This technology includes machine learning algorithms, natural language processing systems, computer vision capabilities, chatbot programs, and virtual assistants. All are designed with one goal in mind – optimization of operations within development teams.

What is the future of DevOps?

Operations teams collaborating with developers to create quality products quickly and effectively is the goal for DevOps moving forward. The vision of this process involves both sides working together, resulting in an improved system and increased efficiency.

LangChain for DevOps: Learn LLM & GenAI for Dev, Sec & Ops

Sara Huddleston — Thu, 26 Oct 2023 00:00:00 +0000

The emergence of DevOps revolutionized software development. Now, with AI powered tools like LangChain, these transformations are being accelerated. Unsurprisingly, our distinguished speaker at the launch of Pulumi’s in-person AI Talks, Patrick Debois, who coined the term “DevOps,” has recently tuned into LLM and GenAI Ops using the Langchain framework.

This article unwraps the topics addressed during our “Generative AI, Apps, and DevOps | AI/ML Talks” event in Seattle, focusing on Langchain and GenAI for DevOps. You’ll be able to watch Patrick Debois teach LangChain to learn LLMs and GenAI, with code examples and lessons that are easily understood by traditional software engineering.

What is LangChain used for?

LangChain is an open-source Python framework designed to empower developers to construct robust generative AI applications. It facilitates the integration of advanced AI models such as OpenAI’s GPT, Google’s Gemini, and Meta’s LLaMA. This versatile tool finds application in developing retrieval-augmented chatbots and other personalized assistants, utilizing technologies like ChatGPT. Furthermore, it enables tasks such as question-answering (GQA), summarization, code comprehension, API interactions, data cleansing, and much more.

Why is LangChain becoming popular?

The landscape of AI development is rapidly evolving, and LangChain is packed with incredible features for building LLM applications and tools. Developers with a wide range of expertise use LangChain for tasks such as managing interactions with language models, establishing seamless connections between diverse components, and integrating external resources like APIs and databases effortlessly.

The LangChain platform hosts a diverse collection of APIs seamlessly integrated into applications. This enables developers to incorporate advanced language processing functionalities without the need for laborious construction from scratch.

LangChain Accelerating DevOps

Patrick Debois presenting the workshop Dev, Sec & Ops meet LangChain

LangChain can streamline the integration of large language models and data sources. By helping bring in AI capabilities, DevOps can revolutionize software development life cycles by making it easier for teams to handle challenges while optimizing resource allocation and usage through their cutting-edge, AI-powered solutions. This makes the automation of manual processes easier and the development of new resources far more accessible with faster utilization rates by leveraging artificial intelligence. Significantly important as “Developer Velocity and Productivity” become business performance metrics across various industries.

The features included in LangChain include chain interface memory functions for scalability within apps, control over dataflow, as storage – making code quality more reliable, plus seamless machine learning models integration, which allows them to build custom automated AI programs quickly. Thus providing users with an advanced but easy way of creating sophisticated solutions by combining their knowledge of coding and machine training models together.

How to Use LangChain to Learn LLMs and GenAI for Dev(Sec)Ops

Patrick teaches in the way he would like to be taught. The workshop mainly uses the Langchain framework and basic Python knowledge. OpenAI will also need to be installed. You’ll learn essential skills in multiple use cases, applying Langchain for LLM application Development.

You can see the breakdown of the lessons per role below, and find the code examples in the Learning LLMs and GenAI for DevSecOps repo.

Developer

Calling a simple LLM using OpenAI
Looking at debugging in Langchain
Chatting with OpenAI as a model
Using prompt templates
Use Docloader to read your local files and prepare them for the LLM
Explain the calculation and use of embeddings
Understand how splitting and chunking are important
Loading embeddings and documents in a vector database
Use a chain for Questions and Answers to implement the RAG pattern (Retrieval Augmented Generation)
Show the use of OpenAI documentation to have the llm generate calls to find real-time information
Implement an Agent and provide it with tools to get more real-time information

Operations

Find out how many tokens you are using and the cost
How to cache your calls to an LLM using exact matching or embeddings
How to cache the calculation of embeddings and run the calculation locally
Run your own local LLM (using Ollama)
Track your calls and log them to a file (using a callback handler)
Impose output structure (as JSON) and have the LLM retry if it’s not correct

Security

Explain the OWASP top 10 for LLMS
Show how simple prompt injection works and some mitigation strategies
How to detect prompt injection using a 3rd party model from Hugging Face
Detect project injection by using a prompt
Check the answer LLMs provide and reflect if it is ok
Use a Hugging Face model to detect if an LLM output was toxic
Show a simple prompt for asking the LLM’s opinion on Kubernetes and Trivy vulnerabilities

Real-World Applications of LangChain AI in DevOps

Vector similarity search in a RAG application. Credit to LangChain and Neo4J team

LangChain AI has practical applications in DevOps, particularly related to question-answering using documents as context, extraction, evaluation, and natural language processing tasks, all powered by AI tools. An interesting scenario is implementing a knowledge graph based RAG application with LangChain to support the DevOps team.

ChatGPT based on LLMs enabled with these AI tools is one of the foremost implementations at present. Tracking capabilities through LangChain enables developers to determine which prompts are more effective, thus supporting what artificial intelligence offers when integrated into development operations.

Monitoring and Anomaly Detection with LangChain AI

Monitoring LLM performance with LangChain and LangKit. Credit to WhyLabs

LangChain AI provides an efficient approach to DevOps through continuous monitoring, and anomaly detection. LangChain’s integration in DevOps allows for intelligent monitoring of various system parameters, application performance metrics, and log files. It can process large volumes of data generated by applications and infrastructure components. Through natural language processing (NLP) techniques, LangChain can interpret logs, error messages, and system alerts in real-time.

LangChain’s AI-driven anomaly detection capabilities are pivotal in identifying deviations from expected behavior in the monitored data. By analyzing historical data patterns, LangChain can learn the normal behavior of the system and its components. When the system encounters an unusual event or behavior, LangChain can promptly detect it as an anomaly. These anomalies could be security threats, performance bottlenecks, code quality, or any irregularities within the system.

Keep in mind that monitoring LLM performance is important to ensure you can rely on the model’s accuracy and relevance in real-world applications. Our customer WhyLabs has a great article showing the significance of monitoring LLMs performance with LangChain and how to get started with monitoring.

Future: AI, Apps, and DevOps

Platform engineering, DevOps, or MLOps may be the next steps in the AI-driven software development future. AI tools are the gateway to giving developers access to big language models that allow them to create highly sophisticated solutions.

Accelerating Development with AI-Powered Tools

AI-powered tools are revolutionizing the development process, enabling DevOps teams to access intelligent recommendations and improve code quality or obtain suggestions on new ways to accomplish goals. An example is the usage of Pulumi Neo.

Tools like LangChain, Amazon CodeGuru, and Pulumi AI provide actionable insights for efficient resource utilization while enforcing policy and compliance best practices. An IaC that leverages AI empowers teams to automate and more effectively manage infrastructure.

Developers can reap multiple benefits from higher quality assessment of code lines, instantaneous detection of policy and security issues, useful cues on how to resolve problems, and improving code quality, productivity, and time management.

AI has greatly impacted optimization during the review stage, providing developers with better insights into performance issues, code effectiveness, and resource utilization without any extra effort.

Security and Vulnerability Management with AI

The marriage of AI, machine learning, and DevOps/DevSecOps practices revolutionizes security and vulnerability management in software engineering. AI-enabled tools technologies not only strengthen the defenses of software systems but also enable organizations to stay one step ahead in ensuring successful deployments by removing manual processes through automation, using single sources of truth, enforcing least-privileged access through role-based access controls, and automated remediation to correct configuration violations like auto-tagging, removing Internet access, and enabling storage encryption.

DevSecOps can foster collaboration between developers, security, and operations and still enhance security and compliance best practices using Artificial Intelligence (AI) and Machine Learning techniques.

Leveraging Machine Learning in DevOps

Applying Machine Learning in DevOps is a game-changer. Not only does it automate and enhance various processes but also brings an intelligent, predictive, and adaptive dimension to development workflows. By embracing these technologies, development teams can achieve higher efficiency, improve code quality, optimize performance, and seamlessly integrate with artificial intelligence, thereby shaping the future of software engineering.

ML models can evaluate past outcomes from compilation/builds plus performance metrics within an operation so that developers may leverage these insights when writing code accordingly with machine learning algorithms, allowing them to identify vulnerabilities or issues faster through root cause analysis while ameliorating system security against outside attacks like hackers or DDOS activity thus boosting overall system productivity thanks to AI implementation into existing processes.

Implementing AI models into DevOps: Key Considerations

When considering AI, it’s essential to assess data quality, integration complexity, cost savings, and ethical/legal considerations:

Ensuring the data used for training these AI tools are precise, exhaustive, and up-to-date will help gain more reliable results.
An important aspect of the implementation process is integrating existing DevOps instruments and guaranteeing scalability, depending on the company’s necessities.
Ensuring that all laws applicable when working with such technology.

Conclusion

AI and ML are rapidly becoming a necessity, which can greatly enhance and benefit Development, Security, and Operation teams. At first glance, it may appear daunting or out of reach, but tools like Langchain and Pulumi can make it easier to get started. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Who is Patrick Debois?

Patrick Debois is a highly accomplished technologist whose expertise spans the domains of Development (Dev), Security (Sec), and Operations (Ops). Acknowledged as a respected confidant within the developer, security, and operations communities, Patrick is presently deeply engrossed in the realm of Artificial Intelligence and Machine Learning, continually pushing the boundaries of his technical acumen.

He was the organizer of the inaugural DevOpsDays in 2009. He is credited with coining “DevOps” and co-authoring the renowned DevOps Handbook. In the past, Patrick has collaborated with esteemed technology organizations such as Atlassian and Snyk. Currently, he wears dual hats as the Vice President of Engineering and a Distinguished Engineer at Showpad.

Is LangChain Free?

Yes, LangChain is an open-source framework, free to use and modify.

How Does LangChain Work?

LangChain is a meticulously designed open-source framework tailored to simplify the creation of applications powered by large language models (LLMs). First, you will need to have a language model or you can use a public language model or train your own.

Once you have a language model ready, you can start building applications. The framework offers an array of tools and APIs, simplifying the process of connecting language models to diverse data sources, interacting with their environment, and constructing intricate applications.

LangChain operates by linking together various components, referred to as links, to establish a workflow. Each link in the chain performs a task and is connected in a sequence, enabling the output of one link to serve as the input for the subsequent one. This sequential linking allows the chain to execute intricate tasks by combining simple tasks seamlessly.

See LangChain’s get started docs for more information.

Does ChatGPT use LangChain?

ChatGPT utilizes LangChain to create custom artificial intelligence (AI). This allows for an individualized experience.

Converting Bicep code to Pulumi

Zaid Ajaj — Fri, 29 Sep 2023 00:00:00 +0000

Bicep is a DSL developed by Microsoft to simplify the authoring of ARM templates and deploy resources to Azure. Today I will be sharing with you a new Pulumi converter plugin that I have been working on that converts Bicep code to any of the supported Pulumi languages.

The Bicep language is a domain-specific language for Infrastructure-As-Code built by Microsoft that uses declarative syntax to deploy Azure resources. It was made to address the shortcomings of authoring ARM templates that are written in JSON. Compared to ARM templates, Bicep is much more concise and readable. However, here at Pulumi we believe that using general-purpose programming languages that are familiar to developers is the best way to author Infrastructure-As-Code. Besides being able to use a general-purpose language, you and your team can benefit from using Pulumi Cloud as you scale to larger projects and teams with features such as Pulumi Deployments, Organization Access Tokens, Review Stacks and more.

That is why I have been working on a new Pulumi converter plugin that converts Bicep code to any of the supported Pulumi languages using the Pulumi CLI via pulumi convert.

Installing and using the plugin

First, install the converter plugin using the Pulumi CLI:

pulumi plugin install converter bicep

This command will install the plugin from the latest GitHub release of the repository.

Once installed, you can use it to convert Bicep code to Pulumi. For example, let’s say we have the following Bicep code in a file called example.bicep which deploys a storage account:

resource storage 'Microsoft.Storage/storageAccounts@2022-09-01' = {
 name: 'storageaccount'
 location: resourceGroup().location
 kind: 'StorageV2'
 sku: {
 name: 'Standard_LRS'
 }
}

Navigate to the directory where the file is located and run the following command. For example to generate TypeScript code from the Bicep code above, run:

pulumi convert --from bicep --language typescript --out output -- --entry example.bicep

This command will use the Bicep converter plugin we just installed to convert the code in example.bicep to TypeScript and place the generated code in the output directory.

You can change typescript to any of the supported languages such as csharp, python, go, java or yaml and you are good to go.

If you don’t specify the --entry flag, the plugin will look for the first file with the .bicep extension in the current directory.

Let us look at the generated code from the example above:

import * as pulumi from "@pulumi/pulumi";
import * as azure_native from "@pulumi/azure-native";

const config = new pulumi.Config();
// The name of the resource group to operate on
const resourceGroupName = config.require("resourceGroupName");
const currentResourceGroup = azure_native.resources.getResourceGroupOutput({
 resourceGroupName: resourceGroupName,
});
const storage = new azure_native.storage.StorageAccount("storageaccount", {
 accountName: "storageaccount",
 kind: "StorageV2",
 location: currentResourceGroup.apply(currentResourceGroup => currentResourceGroup.location),
 resourceGroupName: currentResourceGroup.apply(currentResourceGroup => currentResourceGroup.name),
 sku: {
 name: "Standard_LRS",
 },
});

import pulumi
import pulumi_azure_native as azure_native

config = pulumi.Config()
# The name of the resource group to operate on
resource_group_name = config.require("resourceGroupName")
current_resource_group = azure_native.resources.get_resource_group_output(resource_group_name=resource_group_name)
storage = azure_native.storage.StorageAccount("storageaccount",
 account_name="storageaccount",
 kind="StorageV2",
 location=current_resource_group.location,
 resource_group_name=current_resource_group.name,
 sku=azure_native.storage.SkuArgs(
 name="Standard_LRS",
 ))

package main

import (
 "github.com/pulumi/pulumi-azure-native-sdk/v2/resources"
 "github.com/pulumi/pulumi-azure-native-sdk/v2/storage"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cfg := config.New(ctx, "")
 // The name of the resource group to operate on
 resourceGroupName := cfg.Require("resourceGroupName")
 currentResourceGroup, err := resources.LookupResourceGroup(ctx, &resources.LookupResourceGroupArgs{
 ResourceGroupName: resourceGroupName,
 }, nil)
 if err != nil {
 return err
 }
 _, err = storage.NewStorageAccount(ctx, "storageaccount", &storage.StorageAccountArgs{
 AccountName: pulumi.String("storageaccount"),
 Kind: pulumi.String("StorageV2"),
 Location: *pulumi.String(currentResourceGroup.Location),
 ResourceGroupName: *pulumi.String(currentResourceGroup.Name),
 Sku: &storage.SkuArgs{
 Name: pulumi.String("Standard_LRS"),
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AzureNative = Pulumi.AzureNative;

return await Deployment.RunAsync(() =>
{
 var config = new Config();
 // The name of the resource group to operate on
 var resourceGroupName = config.Require("resourceGroupName");
 var currentResourceGroup = AzureNative.Resources.GetResourceGroup.Invoke(new()
 {
 ResourceGroupName = resourceGroupName,
 });

 var storage = new AzureNative.Storage.StorageAccount("storageaccount", new()
 {
 AccountName = "storageaccount",
 Kind = "StorageV2",
 Location = currentResourceGroup.Apply(getResourceGroupResult => getResourceGroupResult.Location),
 ResourceGroupName = currentResourceGroup.Apply(getResourceGroupResult => getResourceGroupResult.Name),
 Sku = new AzureNative.Storage.Inputs.SkuArgs
 {
 Name = "Standard_LRS",
 },
 });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.azurenative.resources.ResourcesFunctions;
import com.pulumi.azurenative.resources.inputs.GetResourceGroupArgs;
import com.pulumi.azurenative.storage.StorageAccount;
import com.pulumi.azurenative.storage.StorageAccountArgs;
import com.pulumi.azurenative.storage.inputs.SkuArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 final var config = ctx.config();
 final var resourceGroupName = config.get("resourceGroupName");
 final var currentResourceGroup = ResourcesFunctions.getResourceGroup(GetResourceGroupArgs.builder()
 .resourceGroupName(resourceGroupName)
 .build());

 var storage = new StorageAccount("storage", StorageAccountArgs.builder()
 .accountName("storageaccount")
 .kind("StorageV2")
 .location(currentResourceGroup.applyValue(getResourceGroupResult -> getResourceGroupResult.location()))
 .resourceGroupName(currentResourceGroup.applyValue(getResourceGroupResult -> getResourceGroupResult.name()))
 .sku(SkuArgs.builder()
 .name("Standard_LRS")
 .build())
 .build());

 }
}

configuration:
 resourceGroupName:
 type: string
resources:
 storageaccount:
 type: azure-native:storage:StorageAccount
 properties:
 accountName: storageaccount
 kind: StorageV2
 location: ${currentResourceGroup.location}
 resourceGroupName: ${currentResourceGroup.name}
 sku:
 name: Standard_LRS
variables:
 currentResourceGroup:
 fn::invoke:
 Function: azure-native:resources:getResourceGroup
 Arguments:
 resourceGroupName: ${resourceGroupName}

Notice how the in the original Bicep code, we reference the location of the implicit resource group via resourceGroup().location. When the converter generates code for the target language, it replaces this implicit resource group with an explicit one by parameterizing the program by a resource group name and using that resource group when deploying the resources.

Early days for the converter

Currently the converter supports most of the features of Bicep such as parameters, variables, modules, resources, outputs and a number of built-in functions. That said, the converter plugin is still in its early days and it’s not yet tested against many real-world Bicep programs. I am actively working on improving it and making it support most of the Bicep language features. The source code is available on this GitHub repository. If you find any issues or have any feedback, please open an issue on the repository.

How does it work?

The converter plugin is a native binary built using .NET in F# and new Pulumi Converter SDK. Using the core Pulumi nuget package, we shipped an experimental API that allows you to easily build converter plugins that can extend the functionality of pulumi convert.

Building a language converter plugin involves taking the source language, in this case Bicep, and converting it to our internal Pulumi Configuration Language, or PCL for short. The converter implements a function of the following shape:

let convertProgram (request: ConvertProgramRequest): ConvertProgramResponse =
 let sourceDirectory = request.SourceDirectory
 let targetDirectory = request.TargetDirectory
 let args = request.Args
 // conversion logic here

In our case, we read the Bicep code from the source directory, convert it to PCL and write out the PCL code in the target directory. Then, Pulumi uses the built-in program generation facilities to take care of the rest of the work of generating the target language code from the PCL code so that we don’t have to worry about language-specific details. The converter only needs to know how to generate PCL.

To build the actual transformation from Bicep to PCL, I made use the of the Azure.Bicep.Core package available for .NET which allowed me to parse the Bicep code and generate a typed Abstract Syntax Tree (AST) from it. Building an AST from the source language allows us to easily traverse the tree, analyze it and symbolically rewrite pieces of it. Working at the AST level also makes it easy to test source code transformation using structure rather than text. Once we have obtained the Bicep AST, we transform it into a Pulumi AST that represents a PCL program. Finally, we print out the Pulumi AST to a string and write it to the target directory.

If you are curious about the specification of the AST that represents Bicep programs, head over to this file from the source code. You will find types such as BicepProgram and BicepSyntax that model almost every aspect of Bicep code. As for the AST of PCL programs, you can find the type definitions here.

The F# language is great for almost everything, but is especially amazing for writing language converters because of its powerful pattern-matching capabilities and the ability to write code in a functional style. Many of us working at Pulumi are big fans of F#!

Here is a snippet of the transformation pipeline written in F# that takes the input Bicep program and converts it to the final PCL text

bicepProgram
|> BicepProgram.dropResourceUnknowns
|> BicepProgram.reduceScopeParameter
|> BicepProgram.parameterizeByTenantAndSubscriptionId
|> BicepProgram.addResourceGroupNameParameterToModules
|> BicepProgram.parameterizeByResourceGroup
|> Transform.bicepProgramToPulumi
|> Transform.modifyComponentPaths rewriteComponentPath
|> Printer.printProgram

You can learn more on the implementation of these functions from the source code on GitHub. I’ve written it in a way that is easy to follow, understand and contribute to. Most of these functions are fully unit-tested and I am working on adding more tests to cover more of the Bicep language features.

Bonus converter: ARM to Pulumi

Those who are familiar with the Bicep CLI, know that it has a built-in decompiler that converts ARM templates to Bicep. I thought it would be fun to build another converter that takes ARM templates to Pulumi but without writing any conversion logic for it. Instead, I would decompile the ARM template to Bicep, then use the logic from the Bicep converter to generate the final PCL code. Fortunately, the decompiler code is embedded in the nuget package Azure.Bicep.Core so we don’t have to rely on the existence of the Bicep CLI in order to make this converter work.

From all of this, I created another converter pulumi-converter-arm that works exactly like the Bicep converter but takes ARM templates as input. Head over to the repository to learn more about how to install it and how to use it.

Depending on how well this ARM converter does its job, we might consider deprecating arm2pulumi in favor of it. However, we first need to test it against more templates and iron out the rough edges. It would be the ideal situation because there are virtually zero maintenance costs, any improvements to the Bicep converter would automatically get picked up here. Also, improvements to the decompiler from the Azure.Bicep.Core nuget package would benefit the converter as well.

Building your own Pulumi language converter

Converter plugins are a great way to extend the Pulumi CLI with pulumi convert to support new languages. When we shipped support for Converting full Terraform Programs To Pulumi, we extended the pulumi convert command to allow installing and using converter plugins that are shipped independently from the Pulumi CLI.

In essence, those plugins are executable binaries that can be written in any language, which serves up a gRPC server that implements the Converter contract:

// Converter is a service for converting between other ecosystems and Pulumi.
// This is currently unstable and experimental.
service Converter {
 // ConvertState converts state from the target ecosystem into a form that can be imported into Pulumi.
 rpc ConvertState(ConvertStateRequest) returns (ConvertStateResponse) {}

 // ConvertProgram converts a program from the target ecosystem into a form that can be used with Pulumi.
 rpc ConvertProgram(ConvertProgramRequest) returns (ConvertProgramResponse) {}
}

This Converter interface is considered experimental and is subject to change in the future.

Now, learning how to setup gRPC servers and implement the Converter contract is not an easy task unless you are familiar with it. For .NET, we shipped the experimental Pulumi converter SDK to the main Pulumi nuget package that makes it extremely easy to build converter plugins.

A bare-bones converter plugin that does nothing looks like this in F#:

open Pulumi.Experimental.Converter
open Pulumi.Codegen

let convertProgram (request: ConvertProgramRequest): ConvertProgramResponse =
 // TODO: Implement the conversion logic here
 ConvertProgramResponse.Empty

convertProgram
|> Converter.CreateSimple
|> Converter.Serve
|> Async.AwaitTask
|> Async.RunSynchronously

This will spin up a gRPC server that implements the Converter contract and will serve it on a random port assigned during startup. The Pulumi converter SDK automatically implements gRPC reflection which allows tools such as Postman to discover the available gRPC services and methods and send example requests to them. See Postman with gRPC.

Publishing converter plugins

The easiest way to make your converter plugin available to others is to publish it on GitHub releases. The Pulumi CLI expects a naming convention pulumi-converter-<name> and will automatically download the plugin from the latest GitHub release of the repository when given the --server flag during pulumi plugin install converter <name> --server <url>.

Converter plugins written in .NET are simple console applications that are compiled to native binaries for each target platform. To see how this is done, refer to this createAndPublishArtifacts function written for the Bicep converter that does the heavy lifting of building the native binaries for each platform, putting them in an archive then publishing them to GitHub releases.

Note: when publishing native binaries for macOS, the Github Action executing the publishing process must be using a macOS runner so that the generated binary from dotnet publish gets signed with a developer certificate. This is required for the binary to be executable on macOS.

Final thoughts

I hope you enjoyed this article and learned something new about the Pulumi converter plugins and Pulumi internals. I am very excited about the Bicep converter plugin and I am looking forward to hearing your feedback on it from the community. It was a lot of fun writing this in F# and made writing complex pieces of code very trivial.

If you have any questions or feedback, don’t hesitate to reach out!

How Starburst Data Creates Infrastructure Automation Magic With Code

George Huang — Wed, 23 Aug 2023 00:00:00 +0000

This blog post summarizes a presentation by Matt Stephenson at PulumiUP 2023.

Matt Stephenson is Senior Principal Software Engineer for Starburst Data and a Puluminary member. He’s deeply involved in the Infrastructure as Code (IaC) space, having contributed to Ansible, been a core contributor to Apache jclouds, and has written many Terraform plugins. He leads infrastructure architecture at Starburst and originally introduced Pulumi to the company. Starburst provides a data lake analytics platform that’s powered by Trino - an open-source distributed SQL query engine designed for running fast analytic queries across large datasets in multiple data sources. At Starburst, Matt helped revamp and improve how the company manages its multi-cloud and cloud native infrastructure.

Watch Matt Stephenson’s full presentation with demos. 8:03 demo of CI/CD. 15:33 demo of infrastructure automation.

Technical and Business Problems to Solve

Starburst operates a managed Trino service that runs on Kubernetes clusters on multiple clouds. Their Kubernetes clusters needed frequent manual scaling based on workloads. Furthermore, upgrading Trino versions running on the clusters ran the risk of potential service disruptions for customers. The company had been using tools like Terraform for IaC, Argo CD for deploying Helm charts, Maven, Docker containers, and a Slack bot for infrastructure and deployments. Their CI/CD workflow was difficult to maintain because they needed to create wrappers for their previous IaC tooling, making it less approachable in CI/CD environments. The pipeline was tenuously bound together using GitHub Actions and a Slack bot, requiring specialized GHA runners, leading to a messy architecture.

Why Starburst Chose Pulumi

Recognizing opportunities for improvement, Matt brought in Pulumi and its Automation API. This unified Starburst’s software development and infrastructure management practices, especially when deploying and managing Trino clusters on Kubernetes. The top two reasons Matt chose Pulumi were:

Support for programming languages: Pulumi supports the main languages used by Starburst’s developers, Java and TypeScript. This enables all engineers to write infrastructure code and promotes a cohesive application and infrastructure codebase.
Pulumi Automation API: Automation API is a programmatic interface for the Pulumi CLI and Pulumi IaC engine, which enables Starburst engineers to build infrastructure automation into their applications and CI/CD processes, and orchestrate complex provisioning workflows.

Benefits of Using Pulumi

Adopting Pulumi and its Automation API provided several benefits that made Starburst’s engineers more efficient and productive, ultimately leading to faster and more reliable releases for Starburst’s customers. Matt summed up the benefits of Pulumi for Starburst’s engineers: “When I think about building an airplane, I think about precision and having the right tools for the job.” Pulumi, for Starburst, epitomized this precision and efficiency.

Increased deployment speeds by 3x for new customer releases: Starburst engineers used the Pulumi Automation API to triple the deployment speed of multi-cloud Kubernetes clusters with blue-green deployments that complete within minutes. They did this by orchestrating blue-green updates across their data plane running on the clusters, ensuring zero customer downtime. They built intricate integrations between Pulumi and internal APIs that enabled them to seamlessly transition customers to new clusters. Now they can update their customers’ runtimes, including VPCs and Kubernetes clusters, twice a week.

Unified cluster and infrastructure management: Rather than juggling multiple tools for managing Kubernetes and infrastructure, Starburst engineers use a single tool to manage over 50 Kubernetes clusters and deploy Helm charts into the clusters, spanning multiple regions. They also use Pulumi to manage SaaS platforms like Cloudflare, Confluent, and DataDog.

Consistent deployment lifecycles: Using Pulumi’s integration with GitHub Actions, Starburst engineers built a streamlined CI/CD process where infrastructure and application code builds within the same pipelines with automated test environments. They accomplished this using Pulumi Automation API to abstract the underlying runtime environments, circumventing the need for additional layers and wrappers.

Try Pulumi for Infrastructure as Code Automation

Sign up for a free account to try deploying infrastructure on any cloud with Pulumi Deployments, or register for an upcoming workshop to learn more about how Pulumi can help you ship cloud infrastructure faster and more safely.

Go to more case studies →

Join a Pulumi User Group (PUG) Meetup!

Sara Huddleston — Mon, 01 May 2023 00:00:00 +0000

Hi everyone,

After several great years on Meetup, the Pulumi Community is moving to a new home: Luma.

This wasn’t an easy decision. Together we’ve built 38 Pulumi User Groups across 14 countries, with nearly 10,000 members between them. Meetup is where that community took shape, and we’re grateful for everything that happened there.

We want to make it as easy as possible for you to show up. All upcoming events and workshops will be published on Luma going forward.

We have a global community made up of people from many different countries, but we also have many local communities. We want to enable our users to meet, share knowledge and ideas, answer questions, and learn about Pulumi and best practices. For that reason, we created the Pulumi User Groups, aka PUGs on Meetup.

Members of PUGs will have access to a global network of experts who can offer help, advice, and support on using Pulumi effectively. The PUGs will also feature regular in-person meetups, webinars, in-person and virtual workshops, and other events to help members stay up-to-date with the latest cloud infrastructure and Pulumi developments.

23 Pulumi User Groups (PUGs) Meetups in 10 countries

PUGs are led by Pulumi employees and Puluminaries, who are passionate community members that are experts in topics related to cloud infrastructure, Pulumi Cloud, cloud-native software development, and much more, and are willing to share their knowledge with the community.

United States Pulumi User Groups

South America Pulumi User Groups

São Paulo Pulumi User Group

European Pulumi User Groups

Middle East Pulumi User Groups

Tel Aviv Pulumi User Group

Asia Pulumi User Groups

Singapore Pulumi User Group

Oceania Pulumi User Groups

The First PUG in person

Our very first in-person PUG meetup took place in London, UK. The event was well-attended by developers, cloud architects, and other infrastructure enthusiasts. The attendees had the opportunity to network with other like-minded professionals and learn more about Pulumi’s tools and features from expert speakers.

During the meetup, attendees heard from several speakers, including Joe Duffy, Co-Founder & CEO at Pulumi, who was there to speak about Infrastructure as Code that writes itself after the release of Pulumi Insights, which happened that same day.

Other speakers included Liam Hampton, Sr. Regional Cloud Advocate at Microsoft, who spoke about Microsoft Azure and IaC, and Sam Cogan, Director of Solution Architecture at Willis Towers Watson, who spoke about building a library of reusable modules you can repeatedly reuse in your IaC projects.

Overall, the first PUG meetup was a huge success, and attendees left feeling inspired and energized about Pulumi’s tools and the community’s possibilities. With the launch of PUGs, Pulumi is demonstrating its commitment to building a strong and supportive community of professionals passionate about cloud infrastructure.

More PUGs and events to come

We are planning more in-person PUG events for the months to come. With PUGs, you’ll have the opportunity to connect with like-minded professionals, share knowledge and best practices, and stay up-to-date with the latest developments in cloud infrastructure and Pulumi.

Join the Pulumi User Groups in your local area, or if there isn’t one yet, you are welcome to join the Pulumi Slack community and meet people from all over the world.

International Women's Day: Celebrating our Women in Tech

Sara Huddleston — Wed, 08 Mar 2023 13:00:00 -0800

Today is International Women’s Day, and this year the theme is #EmbraceEquity - which means creating an equitable environment. An equitable work environment means understanding that everyone, regardless of gender, religion, ethnicity, background, or resources, brings strength to the workforce and that opportunities should be given to them based on their individual needs.

For Pulumi, it means a work environment where everyone can share ideas and respect them even when disagreeing. Women’s experiences - as well as men’s and nonbinary’s experiences - inform the direction of digital technology and innovation.

Being International Women’s Day, today we focus on women in tech.

Women in Tech Stats Awareness

Women earn an average of 17% (US) to 13% (EU) less than men, according to U.S. Census Bureau and European Commission.
60% of women in tech said a parent or teacher encouraged them. Logitech and Girls Who Code’s report on understanding the barriers women face in tech noted the importance of having role models and people encouraging them.
In small companies, 30.2% of employees are women, according to the report Top Companies for Women Technologists.
75% of women in tech are asked to handle more administrative tasks compared to their male colleagues, according to Navisite’s survey.

Below, you can meet some of Pulumi’s tech women - their experience, why they chose tech, and advice to thrive.

Pulumi’s Women in Tech

Sarah Hughes, Software Engineer

“I was lucky enough to go to high school with a ton of other girls who were into tech, so I took AP Computer Science in a class that was close to a 50/50 split. It really helped to start from scratch on equal footing. (Almost) everyone knew absolutely nothing, and it was obvious that our strengths and weaknesses were completely independent of gender.”

Lina Desloge, Account Executive

“I have always enjoyed problem-solving and working with people…which is what ultimately led me to a technical product. The advice I would give my younger self is don’t worry, just start!”

Kelsey Dirks, Customer Success Manager

“When I entered university, I chose Electrical Engineering as my major. I soon learned coding was not my specialty and switched to Communications, but I loved the people in the classes. It was clear both the professors and students were passionate about their craft. I knew that when I left university, I wanted to work with these types of people, but I had no idea in what capacity. Over the years, my experience led me to be a Customer Success Manager.

What I would tell my younger self is to not stress about trying to figure everything out right away. Every job will have its ups and downs, but when it comes to a career, trust that it will all work out. The biggest piece of advice I can give to women entering the tech field is don’t be afraid to lean on your female support system. If you aren’t able to advocate for yourself, your sisters in STEM will.”

Jonelle Boyd, UX Designer

“If I could give advice to women entering the tech field, I would tell them don’t be afraid of change. Whether that be a role change or a job change, because every new opportunity is a chance to learn and grow. I’ve changed roles as well as worked at a variety of different places during my career and I’ve learned something new in every position. There’s tremendous value in having a wide range of experience, so never be afraid of making that change.”

Casey Huang, Software Engineer

“To my younger self - At some point, you’ve heard people sell working in the tech industry by saying the hours are more flexible - you’re not chained to your desk 9 to 5, you can leave early as long as you get your work done. This is somewhat true, but you should know: there is always more work to do. Some of it is important enough to move the business needle. Some of it isn’t as important, but you just care a lot about it. It’ll be up to you to define your boundaries, and it will be up to you to defend them. But it’s okay; if your work environment is truly a good fit for you, they’ll follow your lead and support you. You can’t min/max life like you like to do in video games. What you’re doing is good enough. Don’t be so afraid of making mistakes. If anything, you’ll learn faster that way.”

Guinevere Saenger, Software Engineer

“Working in tech provides me with significant financial independence, professional respect, and a flexible work schedule. These are worthwhile for anyone to have, but especially so for women.”

We’re always looking for more amazing talent to join us in our community and as part of the Pulumi team.

Managing NOAA Open Data across Multiple Clouds with Pulumi

Denis Willett, PhD — Tue, 08 Nov 2022 21:01:21 +0000

Denis Willett is a software engineer at the North Carolina Institute of Climate Studies who works on the NOAA Open Data Dissemination Program. His work focuses on leveraging cloud technologies for the development of data processing and machine learning pipelines. Denis did his PhD in Entomology and Nematology at University of Florida and his undergraduate and masters work in Earth Systems at Stanford University. You can read his full bio here.

NOAA Open Data Dissemination (NODD) makes environmental data freely and publicly accessible across Amazon Web Services (AWS), Microsoft Azure (Azure), and Google Cloud Platform (GCP). These data include near real-time satellite imagery, weather models, radar feeds, drought information, ocean databases, and a suite of climate data records among many others. This program supports more than 220 datasets and over 24PB of open data. Since its inception, the program has been growing rapidly, almost doubling in size over the past year.

Availability of the petabyte scale environmental data on the cloud is only one aspect of NODD. The program places a premium of performant data delivery and access. NODD data pipelines provide resilient and low, in many cases near real-time, latencies to NOAA’s core data offerings. In addition, NODD provides performant, parallel access to these data supporting billions of requests per day and accessions of individual datasets exceeding 1PB in a single day.

To support this program, the North Carolina Institute for Climate Studies (NCICS) through the Cooperative Institute for Satellite Earth System Studies (CISESS) engineers infrastructure on all three major public clouds ranging from object stores to messaging platforms, data ingest and transformation pipelines, and monitoring systems. These systems are deployed and managed by a very small team that, by necessity, must be agile, resilient and facilitate low latency data transfers at petabyte scale. Given the scale and diversity of resources managed through this initiative, we began looking for tools to facilitate multi-cloud management using infrastructure as code (IAC).

Before Pulumi

Prior to beginning to transition to Pulumi, our team built infrastructure primarily using the Command Line Interface (CLIs) and consoles of each cloud service provider. While this worked well, it required extensive Cloud Service Partner (CSP) specific training to be able to not only navigate the nuances of each provider’s offerings, but also the unique interface to deploy resources. Changing and updating infrastructure in this manner resulted in long development cycles where upgrades were mostly manual and composed of repetitive tasks.

Because of the manual nature of these deployments across three different cloud service providers, infrastructure began to sprawl. A clear snapshot of what resources were deployed where became difficult to compile and management of legacy and deprecated systems depended on manual intervention for clean-up. Infrastructure also began to drift. What started as uniform deployment of resources across multiple cloud platforms morphed as manual updates to each instance resulted in unique environments that were hard to replicate if they had to be deleted and redeployed.

As an example, our program relies on Apache NiFi deployments on AWS, GCP, and Azure to coordinate and manage petabyte scale data transfers while managing data provenance. Drift in these deployments makes it especially difficult to redeploy identical configurations (even though we were using Ansible and Containerization) if they go down.

An additional consequence of this approach is knowledge silos. The team member who deployed a group of resources on AWS had trouble (even with extensive documentation) communicating reusable architectures to another team member who was working on similar situations in Azure. While there are obvious nuances relevant to each cloud environment, our need to replicate similar infrastructure in a multi-cloud environment at scale necessitated extensive detailed communication in a team environment. Questions like: “What is the ARN of the most recent version of this service?” and “What queues do we have on GCP?” required communication and time taken by the individual responsible to answer those questions.

How and why we chose Pulumi

As our team grew and our platform scaled (more than doubling in storage size over the past year by more than 10PB), we began to explore tooling to help us better navigate and manage our multi-cloud and multi-account environment. The nature of the NOAA Open Data Dissemination program agreements with each cloud service provider (CSP) meant that we had multiple accounts with different permissions, funding, and configuration on each CSP, resulting in flexible options.

We began looking at infrastructure as code (IAC) solutions and initially considered many of the cloud-specific options: CloudFormation on AWS, Deployment Manager on Google, and Resource Manager on Azure. While these were nice for automation, we quickly discarded these as options because each had an individual learning curve for that environment; they did not translate well in a multi-cloud environment. These tools also did not have the control features of a full-fledged programming language; looping and if-then statements were challenging to implement.

We also considered other Infrastructure as Code providers. While these solutions would meet a lot of our needs, and one of our team members had extensive experience with one of these tools, the ability to use a common tool seamlessly in a multi-cloud environment was attractive. However, the learning curve associated with a domain specific language was cause for reticence. In addition, we really wanted to be able to use the features of a full-fledged programming language.

After experimenting with a few IAC options, our research institute began using Pulumi in late 2021 and immediately began implementing it to expand our infrastructure.

Benefits of using Pulumi for multi-cloud infrastructure as code

Pulumi had all the benefits we were looking for in a multi-cloud infrastructure as code platform. In terms of automation, manual checklists could be converted to code reducing both deployment time and human errors in deployment. This automation with infrastructure as code (and accompanying state management) allowed us to adopt an immutable infrastructure paradigm where old resources are replaced with new updated infrastructure. While these benefits are accessible via other IAC tools, we particularly appreciated the Pulumi interface to be able to quickly access and inspect infrastructure deployments.

We saw immediate wins for our teams in the following areas:

Drift was eliminated. Single source of truth, version controlled (and tagged) code documented exactly what was deployed where. When updates were needed, the code was updated and redeployed without manually tweaking individual resources through the console.
Management of legacy and deprecated resources were obviated. With IAC managed deployment that would automatically clean up old resources, we saved many future hours cleaning up unused cloud resources.
Team collaboration was enhanced. With a single source of truth and a visual snapshot of resources and deployments, the number of ‘What was the ARN of that resource again?” type questions dropped considerably. Team members could simply look at the deployment in one location and see all of the resources across all three major clouds. This type of friction-reduction really enhances team productivity.

In addition to these IAC benefits, we saw a number of key advantages in implementing Pulumi. First and foremost was the ability to use full-featured programming languages to deploy IAC solutions in a multi-cloud environment. This not only allowed us to fully take advantage of programming control structures (e.g., need 100 different rules on AWS? Now you do not have to manually script each one, just put it in a loop), but also flattened the learning curve for adoption. Our team already used Python extensively in our day-to-day work. Adopting Pulumi was almost as simple as importing a package. Accessing Pulumi through Python made it much easier to embrace.

This accessibility through Python also meant that learning how to create, deploy, and share reusable templates was trivial as all team members already knew how to build, share, and import modules in Python. The ability to rapidly share these templates accelerated our development and allowed for standardization around best practices. For example, we use serverless capabilities on all three CSPs. Once we had a template for deploying our function as a service (FAAS) code on one platform, we did not have to reinvent the wheel or remember to go through a checklist; it was as simple as importing our internal FAAS module and pointing it to the correct code to deploy.

Next steps

Pulumi has become an automation and collaboration tool that is accelerating our ability to provide petabyte scale data to the public in a multi-cloud environment. As we continue to scale our program, Pulumi is allowing us to leverage a small team to manage increasingly larger, more complex deployments in a dynamic environment. In addition, it is rapidly becoming a key player in a cross-cloud fabric that we are building to facilitate working across multiple platforms. By providing a similar, intuitive interface through our language of choice, Python, it is allowing us to more quickly adapt and reproducibly deploy infrastructure across AWS, Azure, and GCP.

Please reach out to us with your thoughts, reactions, and comments at NODD@NOAA.GOV.

Teaming up to help deliver more meals to Ukraine

Wendy Smith — Mon, 16 May 2022 09:00:54 +0200

Why not do good while learning about cloud native tech? Buoyant, Civo, Cockroach Labs, Fairwinds, Pulumi, and Jetstack have teamed up to donate $2 for each virtual booth visitor to World Central Kitchen (WCK), a non-profit committed to providing meals in response to humanitarian crises.

Working with restaurants across Ukraine and in six countries welcoming refugees, WCK is providing hundreds of thousands of daily meals for people in need. WCK serves meals at shelters, hospitals, transit hubs, and more. It’s a wonderful mission and, if you agree, we encourage you to donate to WCK directly 💙 🇺🇦 💛

How it works

Participating is easy. If you registered for KubeCon, just click on the booth links below to visit our virtual booths. Each visit will count as a $2 donation, so if you visit us all, that’s $12 for the cause right there!

Of course, there are many more reasons to visit our booths. You’ll find lots of great cloud native resources including demos, tutorials, ebooks, and more. Here are some of the highlights you can expect at each booth.

Pulumi — Universal Infrastructure as Code

At the Pulumi booth, you’ll learn about our cloud engineering platform, which brings infrastructure, developer, and security teams together through a unified software engineering process that tames cloud complexity and accelerates innovation. Don’t miss our Ask an Expert live sessions to hear directly from the engineering team. Whether you want to know more about how to build with Universal Infrastructure as Code in general-purpose programming and markup languages or just learn more about what’s on the roadmap, this is your chance to ask us anything.

Buoyant — the creator of Linkerd

Buoyant is announcing something really exciting: the introduction of fully managed Linkerd. You can be one of the first to check out the demo, and you’ll also find a ton of service mesh content including a service mesh 101 in English and Spanish, their Engineer’s Guide to mTLS, and details on Buoyant’s Service Mesh Academy.

Civo — Fast Simple, Managed Kubernetes

Stop by the Civo booth for live sessions on how to get the most out of your Kubernetes environment - including demos, Civo marketplace walkthroughs and an inside look at how they live-provision a private region. You’ll find lots of resources and friendly team members to answer any questions on Civo’s easy-to-use managed Kubernetes service. It is fully CNCF certified and 100% Kubernetes upstream compatible.

Cockroach Labs — Build What you Dream

CockroachDB’s presence this year at KubeCon (both physical and virtual) is primarily a charitable one (if you attend in person, visit booth P16, where for each badge scanned they’ll donate $3 to either Black Girls Code, Women Who Code, the Cancer Research Institute, or UNICEF for Ukrainian relief). Visit their booth to see demos and architecture overviews that relate what they’ve learned from building a multi-region, managed distributed SQL database offering on Kubernetes.

Fairwinds — Kubernetes Governance and Security

Fairwinds, a leader in Kubernetes governance providing a unified view between dev, sec, and ops, offers lots of great resources at their virtual booth this KubeCon. You can get a quick tour of Fairwinds Insights, learn about Kubernetes cost savings, how to make Kubernetes service ownership possible, typical Kubernetes security mistakes and much more. Come chat with a friendly Fairwinds engineer to talk about the many great open source solutions including Polaris, Goldilocks, Nova, and Pluto. Visit the Fairwinds booth today!

Jetstack — the creator of cert-manager

Get more out of cert-manager and find out how Jetstack Secure can eliminate workload misconfigurations and provide full visibility of each certificate across every production cluster. Chat directly to one of the project maintainers and learn to use cert-manager to secure Ingress endpoints and internal workloads using mTLS.

Lots of great reasons to visit our booths

There are plenty of good reasons to check out the great content of these amazing companies while supporting a wonderful charity that is doing an incredible job at helping the people of Ukraine.

Introducing the Puluminaries

Matty Stratton — Tue, 03 May 2022 10:04:47 -0500

Pulumi is more than a way to build, deploy, and manage your infrastructure and cloud applications. Pulumi is also a strong and vibrant community. We are very excited to announce and showcase our new program of community champions, the Puluminaries!

The Puluminaries program was created to recognize members of the Pulumi community who are experts and leaders in the field of programming and infrastructure, but also who give back to the community to make it grow! They help others get started, give feedback to make Pulumi’s roadmap even better, and educate people through content or public speaking. The program is open to all members of the community, and we welcome anyone to join the program.

While the Puluminaries themselves have been active in the community for quite some time, we officially launched the program at PulumiUP in 2022, and it is our pleasure to introduce the first cohort of members to the program!

Meet The Puluminaries!

Engin Diri

Engin’s excitement for Pulumi is not possible to be measured using current science. He will always share his thoughts on new developments in the Pulumi project, as well as join in for fun and discussion in various PulumiVision livestreams!

Joshua Studt

Joshua is a longtime member of the Pulumi community, and he contributed to the implementation of Automation API for C#.

Kat Morgan

Kat has been a helpful member of the Pulumi community - not only providing feedback on real-world use of Pulumi, but also sharing her knowledge with workshops and webinars

Komal Ali

In addition to being a Puluminary, Komal is also a Pulumni, having spent time working to make Pulumi a better tool for developers - they were ever-present in making our Python SDK better and creating the Automation API!

Paul Hicks

Paul seems to have an answer to everything. He is ever-present in the Pulumi Community, sharing his knowledge to folks who are just getting started, but also bringing to bear his real-world expertise in complex Pulumi implementations. Chances are, if you see a great answer to a question in Pulumi Community Slack, it came from Paul.

Ringo De Smet

Ringo is one of the founding board members of the Pulumiverse - a place to interact and collaborate on Pulumi-based libraries, projects, and educational materials. Ringo (along with fellow Puluminary Simen A. W. Olsen) have brought their vision for expanding the Pulumi community to life with the creating of this community-operated space! Ringo also created the Pulumi Resource Type for Concourse, and is ever-present in every community he is a part of, not just Pulumi’s!

Rizel Scarlett

Rizel brings her remarkable talent for guidance and teaching to the Pulumi community. She has participated in many Pulumi events, including Cloud Engineering Summit, and appearing on the PulumiVision livestream.

Simen A. W. Olsen

Another one of the founding board members of the Pulumiverse, Simen has a great passion for technology communities, but especially for the Pulumi community. Simen was instrumental on the Pulumi GitHub Action rewrite - designing and rearchitecting it in his free time.

Do you know someone who would make a great Puluminary? Let us know! You can email us at da@pulumi.com with your nominations!

Announcing Pulumi Community Slack Archive

Lee Briggs — Mon, 02 May 2022 12:00:00 -0700

Creating a place for the Pulumi community to gather, ask questions, get help in real-time, and share successes has been an important part of the explosive growth we’ve seen in both users and customers. The Pulumi community slack has grown to over 7000 members and well over 200,000 messages.

Within those 200,000 messages are years of information kept behind a “walled garden” that is undiscoverable outside Slack’s search capabilities.

Today we’re announcing that our community slack is now publicly available via https://archive.pulumi.com thanks to Linen

For many, Slack has become the de-facto collaboration tool. Adding Slack workspaces is relatively straightforward and means you can communicate with various different communities and your colleagues in real-time.

This ability to use the tools you’re using in your everyday work to also quickly get help with Pulumi has proven to be extremely successful. Our users often tell us that having access to Pulumi staff and other community members helps them accelerate their cloud engineering journey. In addition to providing value to our users, the Pulumi team enjoys having a mechanism to engage with users to get feedback and inform them about the Pulumi ecosystem. It’s clear to us that the Pulumi community slack has proven invaluable as Pulumi has grown.

The use of Slack for our community meeting place isn’t without its problems. Often, the first port of call for users that need answers is to use search engines, and all of the information stored in Slack isn’t readily available by default with Slack. The Pulumi team regularly answers questions on StackOverflow and via GitHub Discussions which are searchable and indexed by search engines, but adding the treasure trove of information from Slack will improve the experience for eager Pulumi users.

Until now, this has been exceptionally difficult. With the recent launch of Linen we are able to bring our community chat outside of Slack’s boundaries in an easily searchable, indexable way.

We’re excited to provide this to the community and encourage you to try it out at https://archive.pulumi.com.

Using Go Generics with Pulumi

David Flanagan — Thu, 31 Mar 2022 14:00:00 +0000

This post is outdated. For the latest on Go Generics, visit this blog post.

March 15th, 2022… just two weeks ago. The Go team released Go 1.18 to the world. What seems like a trivial point release actually brings a huge new feature to the Go language: Generics.

In this article, I want to show you how you can use this new feature to build a great developer experience with your abstractions for your Pulumi programs.

Using Go 1.18

The first thing you need to do is ensure you have Go 1.18 installed. You can run go version to check. If you see anything less than 1.18, you’ll need to go upgrade first. The second thing you need to do is update the go.mod in your Pulumi program to specify 1.18 as the minimum version.

go 1.18

Once you’ve done these two steps, you’re ready to start using generics in your Pulumi programs.

Using Generics

Let’s start by asking a question. What is a good use-case for generics? In my experience, generics work really well for allowing us to provide a common interface to our consumers (developers using our APIs) that allows them to use that same interface to accomplish a collection of similar tasks that require different implementations.

For example, let’s assume we want to provide a platform to our developers and allow them to install ANY Kubernetes resource. Our goal is to provide an AddComponent method that they can call to install either pre-supported components, components created by the platform team, or their own custom components. The glue and important aspect here is that all these components conform to the same interface.

Defining The Interface

In the simplest form, we just need an Install function to call that returns either an error or an array of Pulumi resources.

type Component interface {
 Install(ctx *pulumi.Context, name string) ([]pulumi.Resource, error)
}

Creating our Components

Now we need to provide a component that satisfies this interface. So let’s assume that we want to install nginx. First, we create a struct that contains fields for any points of configuration. For today’s example, we’ll just request the version to be installed and a name; the name being used to ensure if the component is installed more than once, it can be uniquely identified.

type Nginx struct {
 Version string
}

func (c *Nginx) Install(ctx *pulumi.Context, name string) ([]pulumi.Resource, error) {
 return []
}

From there, we can begin to flesh out the Install implementation for nginx.

func (c *Nginx) Install(ctx *pulumi.Context) ([]pulumi.Resource, error) {
 deployment, err := appsv1.NewDeployment(ctx, name, &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String(name),
 },
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(3),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String(name),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String(name),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String(fmt.Sprintf("nginx:%s", c.Version)),
 Ports: corev1.ContainerPortArray{
 &corev1.ContainerPortArgs{
 ContainerPort: pulumi.Int(80),
 },
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return nil, err
 }

 return []pulumi.Resource{
 deployment,
 }, nil
}

This is rather contrived, but hopefully you can see the power of using generics as an interface for platform engineering. Our AddComponent implementation for nginx could return a deployment, a service, an ingress with horizontal pod auto-scalers, or any other resource that you want.

Using the Components

Finally, we create our generic function, AddComponent, and start having some fun.

func AddComponent[C Component](ctx *pulumi.Context, name string, component C) ([]pulumi.Resource, error) {
 return component.Install(ctx, name)
}

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 AddComponent(ctx, "my-nginx", &Nginx{Version: "1.14.2"})

 return nil
 })
}

Now you and your organization can provide a repository of these components for your developers to consume with their platforms.

The possibilities are endless.

Have fun and see you soon! 👋

Introducing the Pulumiverse

David Flanagan — Wed, 30 Mar 2022 10:00:00 +0000

Today, we’re excited to announce that we’re working with the Pulumi community to provide a place to interact and collaborate on Pulumi-based libraries, projects, and educational materials: the Pulumiverse.

This project is especially important to me, because it’s an idea that I’ve tried in the past, 2018 😯, and failed to commit to and give it the attention it deserves.

Fortunately, two other wonderful Pulumi community members stepped up and started to put together their own programme and with me now being at Pulumi, the timing was amazingly serendipitous.

I spoke with Ringo De Smet and Simen A. W. Olsen, the founding members of the Pulumiverse.

Ringo, Head of CI/CD at Napoleon Games, had this to say about why he’s excited to create the Pulumiverse:

For infrastructure automation, we had other great tools which preceded Pulumi: Chef, Puppet, Ansible. Any of these tools have a vibrant community of people preventing that all of us need to re-invent the wheel. We had Sous-Chefs for Chef & Vox Pupuli for Puppet. But I see a lot of Pulumi users implementing the same stuff over and over again. That’s why I think Pulumi needs it’s own community. So I started it: Pulumiverse

Simon, a Developer at Bjerk AS, shared:

I imagine a community filled with like-minded people who build and support Pulumi providers, opinionated components, configurations and share ideas on how we can utilize the power of Pulumi in new ways. So I created Pulumiverse to share my enthusiasm, give back to Pulumi and support developers by building a platform where we can work together!

What is the Pulumiverse?

Pulumiverse, in the simplest form, is a GitHub organization. It provides a community-operated place for anyone to communicate, contribute, and collaborate with other members of the Pulumi community and build abstractions and SDKs that can be shared and used by anyone.

There are many libraries or collections of components for Pulumi out there. Most of them are made based on requirements of individual companies and developers. Pulumiverse aims to make a community where these libraries can be created and where opinionated configurations for Pulumi resources can be stored and discussed.

Current Projects

The Pulumiverse launches with just a few projects, but we know that, with the support of such an amazing community, the Pulumiverse will grow and grow and provide invaluable resources for anyone who wants to build or use Pulumi for a library or project.

Awesome Pulumi

🐙 Awesome Pulumi

Everybody loves an awesome list, and this one delivers to you a list of awesome Pulumi resources, books, tutorials, talks, and much more.

Want your project listed? Open a PR. We can’t wait to share it with everyone.

Kubernetes SDKs

🐙 Kubernetes SDKs

The Kubernetes SDKs project aims to provide multi-language SDKs for interacting with Kubernetes Custom Resource Definitions (CRDs). Currently, it automatically builds SDKs for:

ArgoCD
CertManager
Contour
Crossplane
Istio
Kong Ingress
Knative
RedPanda
Tyk

This is just the beginning, and we want to expand this collection quickly. So if you’re working with Kubernetes and want a rich developer experience for authoring custom resources, open an issue and let us know what you need.

Pulumi Concourse & Unifi

These are community-maintained providers and are examples of how you can share your providers with the world as part of the Pulumiverse.

Your Project?

Got your own library that you think would be beneficial to the community?

Join us in the #pulumiverse channel on the Pulumi Slack, and we’ll be happy to help you get it over to the Pulumiverse.

We’ll see you soon! 👋

Pulumi Secrets Management

Sanjay Bhagia — Wed, 16 Mar 2022 09:15:57 -0800

Pulumi community member Sanjay Bhagia explores using Pulumi to manage secrets.

We will build on the Configuration Management blog post that I wrote earlier so please have a look if you haven’t done so already.

Having sensitive information such as Passwords, API Keys, Tokens, ConnectionStrings etc. is quite common for any application. However keeping them in your source code in plaintext is never a good idea, let alone checking it in your source control.

Pulumi’s configuration system lets us add secrets to our configuration in a similar way we manage any other configuration. Pulumi offers multiple approaches to manage sensitive information and exposes a number of APIs and tools we can use to work with them seamlessly, so without further a do, let’s explore how it all works.

Handling Secrets

First off, let’s see how can we set secrets in our configuration.

We use pulumi config set with --secret flag to set a secret

$ pulumi config set --secret mySecretKey mySecretValue

Check the config pulumi config

To retrieve the secrets programmatically, the use of RequireSecrets variant is recommended.

Pulumi.Log.Info($"mySecretKey: {config.RequireSecret("mySecretKey")}");

see the value is masked

If we look at the Pulumi.dev.yaml file in VSCode, this is what we see:

config:
 azure:location: australiaeast
 hello-pulumi:data:
 active: true
 nums:
 - 1
 - 2
 - 3
 hello-pulumi:location: australiasoutheast
 hello-pulumi:mySecretKey:
 secure: AAABAFybCK/w34qectPQHPWSoivUhG7uI5//y7Xqc/6Cczh3k3Yi35zvrk+U
 hello-pulumi:resourcesName: pulumi

Looking at this, what we see is that the configuration we set as a secret is not stored in plaintext, it is in fact encrypted. By default, Pulumi uses per-stack encryption key managed by the Pulumi Service to encrypt values.

However, this might not be desirable in situations where you want to manage the state by yourself by using any external storage like S3 bucket, Azure Blob storage. Or you simply don’t want Pulumi to manage the keys and would prefer keeping that control and management with you. For such scenarios, Pulumi supports multiple providers for secrets management.

As of this writing, these are the supported providers:

default
passphrase
awskms
azurekeyvault
gcpkms
hashivault

We used default provider in our examples above. passphrase allows you to keep the encryption keys to yourself but still use Pulumi’s backend service for state management OR you are self-managing the state storage (e.g., Azure Storage). The last option is where you are completely managing your own keys (bring your own key a.k.a BYOK)

Using Passphrase

Let’s create a new stack in our codebase that we used in the previous blogpost (grab the core here)

$ pulumi stack init passphrasestack --secrets-provider passphrase

or create a new project:

$ pulumi new azure-csharp --secrets-provider passphrase

It’ll ask for the passphrase. This is the passphrase that Pulumi is going to use to encrypt the configuration system.

Looking at the project in VSCode, we can see a new file has been created.

Let’s add two keys (one normal and one secret):

$ pulumi config set normalKey normalValue` `pulumi config set --secret secretKey secretValue

This is how our stack file looks like:

If you pay attention, the secure value in Pulumi.passphrasestack.yaml is different to what we had in Pulumi.dev.yaml

Notice, we need to enter the passphrase every time we are performing any operations. This can be avoided by setting PULUMI_CONFIG_PASSPHRASE in your environment variables (can be done on the local machine or in the CI system).

Using Azure Key Vault for encryption

This would be a similar approach for any other cloud-based provider but we are looking at Azure Key Vault here. In order to use the Azure Key Vault for managing the encryption keys, the approach is same i.e., we set --secret-provider to Azure Key Vault via the CLI

$ pulumi stack init <name> --secrets-provider="<provider>://<provider-settings>"

e.g.,

$ pulumi stack init <name> --secrets-provider="azurekeyvault://<keyvaultname>.vault.azure.net/keys/<key name>"

Before we could do this, we need to create the Key Vault and a Key in the vault that we need.

Create Azure Key Vault

$ az keyvault create --name pulumiazvault --resource-group rg-pulumi-dev --location australiasoutheast

Create the Key in Vault

$ az keyvault key create --name pulumiencryptionkey --vault-name pulumiazvault

Assign Access

When you create the vault and key from the CLI, a policy is created by default but it doesn’t set ’encrypt’ and ‘decrypt’ policy for us, so use this command to include that.

Extract Object Id of your account (you can pull this from Azure AD or run the command below to extract it. Use your account here.):

$ az ad user show --id <your account name>

Look up for ObjectId of your account:

$ az keyvault set-policy --name pulumiazvault --object-id <Object Id from above> --key-permissions encrypt decrypt get create delete list update import backup restore recover

I prefer using Azure AD Application instead of any individual user account for such scenarios. In that case you will need the ObjectId of the application instead.

Now we have the key vault with a key that we will use for encryption along with the right access policy set.

Set Environment Variables

You can set this if you want to use the credentials from the CLI (that you already have set up). AZURE_KEYVAULT_AUTH_VIA_CLI=true

Checkout Azure Environment Authentication if you wish to use other authentication mechanism such as Client Credentials, Username/Password etc.

I prefer using Client Credentials since I’m using Azure AD Application approach.

Create a new stack

I’m creating a new project for this scenario rather than using the previous one but you could very well create a new stack in the same project that uses Azure key Vault provider.

pulumi new azure-csharp --secrets-provider="azurekeyvault://pulumiazvault.vault.azure.net/keys/pulumiencryptionkey"

Pulumi.dev.yaml files looks like this after creating the new project. As you notice, we have the information about secrets provider and encryption key. From this point onward, Pulumi uses this information to encrypt the configuration system for us.

secretsprovider: azurekeyvault://pulumiazvault.vault.azure.net/keys/pulumiencryptionkey
encryptedkey: N1JsSnFjWGpKUl9jY3U1SlRQd2J2WXd3TEw0XzQxeWExY2hTOEhtLTRKWkd3cTViblg3OWF4VHFTcDY4amo4RDc4RUxyT3poNC0tUEVSM0ppbUxfcUdiN1dOWm9Cb0VrNU5TODF6dUlMVEg3c2VVc1ItTVZfRjFBQWpmdGNiX29xYU1BdTN2d0NlMmtWUWFUWUc0NjNJMnh4eUNybzdJMzlXZHdXMVRpODh4TnZfdDM1R2pfdS1rS3VlQlh5cDI4OXZmMkxXaFFuNzNBckJaRU9vcnl5WmJPeUVXMHpJSzlIOEZXOG5WTnl6akFXYmxWYkkyYVB6c3BsUWpDOGw5OFRJNXFyV0JBSjAybHVkTE9UMGF6MllqSW4xbm9GVW1pODRsbVJZbEh6OThPLUdBemlhcWp4V1dKaE5vN0NDeEFZdFM2cXk3emozSDI0UHlGQllESTdR
config:
 azure:location: australiasoutheast

Rest of the things works the same as default or passphrase provider for encryption.

Switching Encryption providers

You change the secrets provider of any of your stack anytime via this command.

pulumi stack change-secrets-provider "<secrets-provider>"

It’s pretty straightforward, once the provider is changed, Pulumi will use the new secrets provider to encrypt configuration and state files.

Summary

Pulumi provides a robust mechanism for configuration management of your infrastructure. It provides support for handling secrets as well. You have the possibility to use external service providers for secrets management such as Azure Key Vault by using a familiar approach. The default option is secure and configurations and secrets are encrypted by the keys that Pulumi manages for us however we have the flexibility to take ownership of this and go to that extra mile for enhanced security or compliance reasons.

You can find the source code for this blog post here.

This article was originally posted at https://www.sanjaybhagia.com/2021/01/26/pulumi-secrets-management. Reposted here with permission.

Migrating My Infrastructure From Terraform to Pulumi

Erik Näslund — Tue, 22 Feb 2022 12:36:23 -0600

Pulumi community member Erik Näslund shares his thoughts on how to migrate from Terraform to Pulumi. Read on to learn all the details of his experience!

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

I’ve been using Terraform for a couple of years and overall I’ve been quite happy with it. However there’s a few things that started to bother me more and more recently.

Terraform uses a language called Hashicorp HCL to define the infrastructure. It’s a relatively simple declarative language, but it’s something I had to learn along the way. Just like any language it has it’s little quirks, and I often found myself spending more time than I wanted to figure out how to do certain things. As I’m doing all the infrastructure myself I really wanted to be able to use a language I’m familiar with, to make things simple.

I had written my own Terraform modules to organize my code. Modules are a great help when it comes to organizing your code in a logical way, and I strongly recommend it if you haven’t tried using them already. However, after using it for a while I realized that the Terraform type system left something to be desired. There are a few basic types available, but not much more than that. This caused bad “micro interactions” between me and Terraform. It was commonplace for me to submit an MR to my Gitlab CI where I passed in a security group name instead of an arn or id.

The third and perhaps most important point is that I think a custom language like HCL is the wrong way to go long term. Don’t get me wrong - I love the people at Hashicorp and I’m forever grateful for Terraform because it’s the first product that made me enjoy writing IaC. That being said, I think it’s a better choice to write libraries for existing languages, instead of trying to create a configuration language of your own. There’s a lot of constructs that I’m used to having in Python that I sorely miss having available in Terraform. In my opinion there’s no reason to develop your own configuration language since it’s a time consuming task, and I don’t find it better than a general purpose language anyways.

Why did I choose Pulumi?

I have this thing called “Fun time Friday” where I spend the first half day on Fridays exploring new things I’ve heard about. Most of the things I look at during that time end up being scrapped, and there’s zero pressure to push things into production.

However after exploring Pulumi I felt that it might be significantly better for me than Terraform. I decided to take a day to see if I could migrate my infrastructure.

Could I have used something else, like AWS CDK? I most probably could. The nice thing about running a company of one is that there’s no need for analysis paralysis! At this point I believed that Pulumi was a better choice for me than Terraform, so I went for it.

Personally I struggle a bit with CDK being mostly Amazon-centric. I sometimes use my IaC to configure things outside of AWS as well. The other issue for me is that CDK by default uses AWS CloudFormation to provision resources. I’ve been bitten a few times by CF. Some times it’s been because of buggy behavior, e.g. a stack that refused to terminate. Other times it’s been the fact that CF often lags behind in capabilities, so it’s common to have to wait a few months for a new feature to be usable through CF.

Sure, I could use CDK for Terraform to have CDK generate Terraform HCL for me. However, now we’re quickly getting away from the simple system I wanted in the first place.

Apologies to those of you who are screaming “FFS - CDK is the best thing ever!" at your monitors right now. You might be 100% right! From what I’ve seen CDK seems pretty awesome for the most part. I didn’t end up with Pulumi because CDK seemed bad. I ended up with Pulumi because it seemed like a slightly better fit for my needs :).

Pulumi in 3 minutes

You might already know a bit of Pulumi, but here’s a very quick overview plus some advice on where to go from here.

Pulumi allows you to write your IaC using TypeScript, Javascript, Python, Go or C#.
You define resources which have inputs and outputs. As an example, a resource could be an RDS database Instance. One of it’s inputs would be the database version to use. One of the outputs would be the hostname / address of the server.
The outputs are lazily evaluated, and their exact value is often not known until after you’ve executed your code. I.e. you wouldn’t have the db_instance.address until the instance had been created. Think of outputs as promises / futures.
You can use component resources to group your resources into logical groups - similar to how Terraform modules work. If you don’t do this there’s a high risk you’ll end up with a very messy setup further down the line.
Read Pulumi’s own overview and naming docs before attempting to use Pulumi yourself. I’d say that’s the most important bit to know about.
Pulumi are using the open core model where most features are free, but some extra goodies are on a paid tier. You can use the open source version and do state management yourself, but you miss out on certain features.

So what does Pulumi code look like? Below is a minimal extract of some of the Python code I’ve written (slightly altered).

# __main__.py
...
prod_redis_server = RedisServer(
 identifier='redis-prod',
 route53_zone=example_com_route53_zone,
 route53_name='redis-prod.example.com',
 security_groups=[my_ecs_service.task_security_group]
)
...

# components/redis_server.py
class RedisServer(ComponentResource):
 def __init__(
 self,
 identifier: str,
 route53_zone: aws.route53.Zone, # Route53 Zome where the DNS record will be created.
 route53_name: str, # Full DNS name of the CNAME record, e.g. 'redis.example.com'.
 security_groups: List[aws.ec2.SecurityGroup], # Security groups which should be granted access to the server.
 opts=None
 ):
 super().__init__('vt:RedisServer', identifier, None, opts)

 # Prefix with `self.` if you want to access the security group from outside this component.
 security_group = aws.ec2.SecurityGroup(
 identifier,
 ingress=[
 aws.ec2.SecurityGroupIngressArgs(
 from_port=6379,
 to_port=6379,
 protocol='tcp',
 security_groups=[sg.id for sg in security_groups],
 )
 ],
 opts=ResourceOptions(parent=self),
 )

 # Create all the other sub resources needed.
 ...

As you can see it’s really simple and clean for me to set up a Redis server now.

There’s also the fact that I can use the Pulumi Console (web interface) to check the state of my infrastructure really quickly. I found myself doing this instead of logging in to the AWS console or using the AWS cli.

How I went about the migration

After reading the migration guide I immediately tried out what seemed to be the obvious option - tf2pulumi. This is supposed to be able to convert Terraform code directly to Pulumi. In my case it simply errored out and was not able to convert my Terraform code. To be fair my TF code was using slightly more “advanced” features like custom written modules, so it wasn’t the easiest thing to handle for tf2pulumi. I recommend you try it out, because it sounds great in theory. However it wasn’t an option for me, so I had to go for a more manual approach. In retrospect I’m actually quite happy I did, because it gave me the opportunity to organise my code in a way that I preferred.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

Don’t worry though - “manual” imports using Pulumi is still rather easy to perform, albeit a bit time consuming. Here’s how I did it.

I started by testing out the AWS getting started guide to get a feel for how Pulumi worked. This was a great move and I recommend everyone else to do the same.

I was then ready to start importing my existing resources. The general workflow was to look at my existing Terraform code, rewrite it using Pulumi, and then import the existing resources. It was a bit time consuming in the beginning, but I got a lot faster at it after a few hours.

Since Pulumi starts out “empty” you can rewrite and import a piece at a time. I’d recommend doing this because you will most probably do some mistakes initially, and it’s good to get early feedback.

You can check the API reference for details on how to import each specific resource. The instructions on how to do it is usually at the bottom of the reference page for each resource type.

Once I had everything migrated to Pulumi I set up a Gitlab CI/CD flow which allowed me to preview and apply the infrastructure changes. Up until then I’d been doing it locally from my own computer. If possible, I recommend doing it that way. The extra time taken by involving a separate CI/CD pipeline would make your migration take a lot longer. It’s perfectly fine when everything is done and dusted, however when learning Pulumi you’ll often need to use the pulumi command to look at the resources.

Issues, workarounds and general tips

Use component resources to group resources

When your infrastructure starts growing it becomes hard to manage unless you group your resources. Pulumi’s solution to this are the so called component resources. They are “logical components” which you define yourself.

$ pulumi stack
...
─ vt:RedisServer vt-redis-prod
 │ ├─ aws:elasticache/parameterGroup:ParameterGroup vt-redis-prod
 │ ├─ aws:ec2/securityGroup:SecurityGroup vt-redis-prod
 │ └─ aws:elasticache/cluster:Cluster vt-redis-prod
...

As you can see I’ve defined a component called “vt:RedisServer” which I use to group all the resources associated with my Redis server. You’ll get a graph representation of your whole infrastructure in the Pulumi Web Console .

It might seem like a small thing, but it really helps to have a logical grouping of your resources as your infrastructure grows. It also help because you can suddenly decide which resources your component exposes and which ones are “internal” to the component. Check out the code sample from earlier in the article to see how to do this using self..

Importing a resource that has a parent was a bit confusing at first

Each resource in pulumi has a globally unique URN. You can show the URNs for the infrastructure you’ve codeified in pulumi by issuing pulumi stack --show-urns

$ pulumi stack --show-urns
...
├─ vt:RedisServer vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod
 │ ├─ aws:elasticache/parameterGroup:ParameterGroup vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/parameterGroup:ParameterGroup::vt-redis-prod
 │ ├─ aws:ec2/securityGroup:SecurityGroup vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:ec2/securityGroup:SecurityGroup::vt-redis-prod
 │ └─ aws:elasticache/cluster:Cluster vt-redis-prod
 │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/cluster:Cluster::vt-redis-prod
...

Let’s have a look at one of the URNs in more detail, shall we?

urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/parameterGroup:ParameterGroup::vt-redis-prod

stack name: prod
project name: VT
parent resource type: vt:RedisServer
resource type: aws:elasticache/parameterGroup:ParameterGroup
resource name: vt-redis-prod

All URNs need to be unique, which means every resource of the same type, with the same parent (in the same stack and project) needs to have a unique resource name.

Normally I would have written and imported my parameter group definition something like this:

# components/redis_server.py
class RedisServer(ComponentResource):
 """
 Redis server running on ElastiCache (single node).
 """
 def __init__(self, identifier: str, opts=None):
 super().__init__('vt:RedisServer', identifier, None, opts)

 ...

 parameter_group = aws.elasticache.ParameterGroup(
 identifier,
 name=identifier,
 family='redis6.x',
 opts=ResourceOptions(parent=self),
 )

# __main__.py
vt_prod_redis_server = RedisServer(
 identifier='vt-redis-prod'
)

And I would then try to import it like this:

# Executing the import (THIS WON'T WORK, DON'T TRY IT)
pulumi import\
 aws:elasticache/parameterGroup:ParameterGroup\
 vt-redis-prod\
 vt-redis-prod

The first vt-redis-prod in the command above is the identifier used by the aws.elasticache.ParameterGroup. This is actually called “resource name” using Pulumi-speak. I prefer to call it identifier in my own code though, because “resource name” is too similar to the actual name it has to AWS for my liking.

The second vt-redis-prod is the actual name of my parameter group on AWS (remember we’re importing existing stuff).

However, the above won’t work for one simple reason - the URN won’t match because we haven’t told Pulumi that this parameter group resource we’re importing is a child of our RedisServer component. What’s the solution? Pretty simple, but somewhat obscure at the same time:

# Executing the import (the right way)
pulumi import\
 --parent vt_prod_redis_server=urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod\
 aws:elasticache/parameterGroup:ParameterGroup\
 vt-redis-prod\
 vt-redis-prod

You tell Pulumi about the parent by using the --parent argument (this is the simple part). The parent argument is a key=value pair, and the key is actually the python variable name you’ve used in your code. If you scroll back up you can see that I stored my RedisServer instance in a variable called vt_prod_redis_server. This part took longer than I’d like to admit for me to figure out.

The value is the URN of the parent component, which you can get with pulumi stack --show-urns.

You can’t import a child without having the parent in place first

That last one was a bit heavy, let’s take a breather and go for something a bit simpler!

When importing a child component you need to import the parent first. If you simply run pulumi up it will try to import all your resources all at once, and it won’t work because the parent haven’t been created yet.

Instead simply do this first to update just a single resource:

$ pulumi up --target urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod

This will cause my vt-redis-prod resource to be created in the Pulumi state. Nothing actually happens on AWS since it’s just a logical component. Once this is done you’ll be able to import all of it’s child resources as you saw above.

Writing IAM (JSON) policies with Pulumi outputs

Imagine having an IAM policy that referenced an ECR repository, just as an example. You might want the policy to grant access only to that particular ECR repository. You might be temped to try something like this:

aws.iam.Policy(
 ...
 policy=json.dumps({
 'Version': '2012-10-17',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Action': [
 'ecr:GetAuthorizationToken',
 ...
 ],
 'Resource': ecr_repository.arn
 }
 ]
 }))

While this might look sensible, it won’t quite work. Remember early on I sad that Pulumi outputs are futures, and not variables whose values you can access just like that. If you did like the example above your policy would just end up with something similar to 'Resource': <Pulumi.Output(...)> in it. Not quite what we wanted.

Instead we have to tell Pulumi to resolve these futures, so we get the actual value for them, like this:

aws.iam.Policy(
 ...
 policy=ecr_repository.arn.apply(lambda arn: json.dumps({
 'Version': '2012-10-17',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Action': [
 'ecr:GetAuthorizationToken',
 ...
 ],
 'Resource': arn
 }
 ]
 }))

If you have more than one Output you need to access you can use Output.all, like this:

policy=(
 Output.all(origin_access_identity.iam_arn, self.s3_bucket.arn)
 .apply(
 lambda args: json.dumps({
 'Version': '2008-10-17',
 'Id': 'PolicyForCloudFrontPrivateContent',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Principal': {
 'AWS': args[0]
 },
 'Action': 's3:GetObject',
 'Resource': f'{args[1]}/*'
 }
 ]
 })
 )
)

The .apply method can also be useful for troubleshooting when you need to access the particular value of an output.

This might seem tricky at first, but digest this for a bit, then have a look at the docs for Inputs and Outputs and I’m sure it’ll all make sense.

Don’t name resources unless you really have to

Pulumi is pretty smart when it comes to naming the actual resources on AWS. If you don’t specify a name when creating a resource Pulumi will use the identifier and append a hex character suffix .

This gives two great advantages. The first is that you can have multiple stacks next to each other, e.g. a “review” and a “production” environment. If you hard coded the names this wouldn’t be possible.

The second advantage is that it makes replacement of resources much easier. If you hard code the name you leave Pulumi no choice but to tear down the existing resource before creating the replacement one. This makes a big difference if you are to perform a version update of one of your dependencies. In case you allow Pulumi to handle the naming it can create the new resource before terminating the previous one. This coupled with a DNS entry update can make for updates with virtually zero downtime.

Since I imported existing resources I had to keep hard coded names in a few places. However I’ve taken the time to get rid of the hard coding as much as possible since then.

Integrating with CI/CD (Gitlab)

I use Gitlab for my Ci/CD, and I’m really happy with it. Let me show you what my very simple Pulumi pipeline looks like right now:

stages:
 - build

preview:
 stage: build
 image:
 name: pulumi/pulumi-python:3.2.1  # Choose the right image for your language
 entrypoint: [""] # This is important for things to work
 script:
 - pulumi stack select prod
 - pulumi preview --color always --diff  # I added '--diff' to always see more details
 only:
 - merge_requests  # Only run the preview on merge requests

update:
 stage: build
 image:
 name: pulumi/pulumi-python:3.2.1
 entrypoint: [""]
 script:
 - pulumi stack select prod
 - pulumi update --color always --diff --yes  # Answer "yes" when asking to apply the changes
 only:
 - master  # Only run the update on the master branch (post merge)

As you can see my pipeline is very simple for this little project which is still in an Alpha phase. There’s just the “production” environment, and changes are automatically applied once the MR is merged.

You’ll have to make use of your gitlab-fu (or other-ci-cd-fu) to adapt this to your own needs. Read the docs from Pulumi to get started. It’s usually very simple and straight forward!

I did things quite differently from what’s described in the Pulumi docs. I used the ready made image from Docker Hub, while they build their own. I did this to make the whole pipeline quicker. It seems to work really well so far, and I haven’t found any downsides to my approach (yet).

I couldn’t get the Gitlab integration webhooks to work

You are normally able to have Pulumi post a report of the planned changes attached to your merge request .

However I couldn’t make this work because of authentication issues. With the help of their support (which was excellent btw) we figured out that it was related to the fact that I was using Gitlab groups to organise my repositories.

As things stand right now you have to create a Pulumi Organization for each Gitlab Group. There’s a bit of an issue with Pulumi’s pricing right now, because the only way to be able to get more than 1 organisation is to go for their enterprise plan.

I think Pulumi will be fixing this in one way or another in the future. For now I’m still pretty happy just checking my changes from the Gitlab job output.

What’s the verdict?

So I’ve been using Pulumi for a few weeks now, and here are my impressions.

It’s really fast compared to Terraform. I didn’t even expect this one, but for some reason Pulumi manages to at least “feel” a lot faster. I haven’t done any measurements to verify this, but it definitely feels much faster.

Their documentation is really good, although it took me a bit to learn to navigate it.

Having type annotations is great! This makes it a lot more friendly to work with as a developer. No more passing around just plain strings in your code. However the type annotations doesn’t quite work in PyCharm because of how the Pulumi code is written. I consider this a PyCharm bug though, and I expect it to be fixed in the future. For now I can simply ctrl+b the class in question, and read the type annotations from the source code.

The Pulumi (Web) Console is a neat way to look at your infrastructure. I know there are similar solutions for Terraform, but I hadn’t used them personally.

The developer ergonomics are great. Using your language of choice helps a lot. When running a company of one, like myself, it’s really important to limit the number of things you need to learn as much as possible. While HCL is simple it’s yet another language to remember. If you’re working at a larger company it’s a great way to encourage developers to do more infrastructure work.

Pulumi is an open core type product, and you will have to pay to access the features on the higher tiers. Their current plans are very fairly priced. I myself is using the free community plan, which works great for me. There’s obviously the risk that they will remove the free plan in the future. However even if that happens I could either pay the fee, or transfer to state storage to AWS S3, and retain most of the product functionality. This feels like an acceptable level of risk for me.

My goal was to do the whole migration in one day. In reality it ended up being closer to two days. However, given that I learned Pulumi and migrated ~130 resources in that amount of time I’d say that Pulumi is quite easy to get started with.

All in all, I’m very happy with making the transition to Pulumi! I became way more productive managing my infrastructure. At the same time I ended up enjoying the work of doing so a lot more - which is really important too.

This article was originally posted at https://blog.ekik.org/my-experience-migrating-my-infrastructure-from-terraform-to-pulumi. Reposted here with permission.

Observable Infrastructure as Code

Kay Plößer — Thu, 20 Jan 2022 08:36:11 -0600

Pulumi community member Kay Plößer spent some time digging into setting up observability of a Pulumi deployment using Honeycomb. Read more to find out all the details on configuring Honeycomb and Pulumi together, with a side dish of Automation API!

Last few weeks, I procrastinated doing real product-related things with setting up monitoring for my infrastructure deployment.

I’m using @pulumidev, a cloud-provider independent IaC framework that allows me to define my infrastructure in TypeScript and use whatever provider I need.

I read something about Honeycomb’s observability approach to monitoring, and it got me thinking about my deployment code. Later, it will probably run with something like GitHub Actions, so it’s not on my machine when it executes. This means I need some insights in case things go wrong. But it would also be good to see how the deployment behaves over time. How do new resources affect deployment durations etc.?

This is what a trace could look like in Honeycomb:

Why not setting things up before I move deployment from my development machine to the cloud?

This article will show you how to make your Pulumi deployment observable with @honeycomb_io

Prerequisites

The Pulumi Automation-API

This API is a programmatical interface to the Pulumi CLI. Think Puppeteer for Pulumi instead of Chrome.

Since it controls the CLI, it can do everything the CLI can and more.

For our use case, we need information about which resource is deployed to tell Honeycomb about it, and it will create nice trace graphs for us.

To run a pulumi up command with the automation API, we need the following code:

const { LocalWorkspace } = require("@pulumi/pulumi/automation")

const main = async () => {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up()
}

main()

The @pulumi/pulumi package already includes the automation API, so it’s the only one we need.

We use the LocalWorkspace object to load our stack. This leaves the wholes stack intact; we can still deploy/destroy it with the CLI.

Then we call stack.up() and wait. The outputs generated from the API aren’t as pretty as the ones from the CLI, but they contain the same information.

Also, the up command of the API allows us to catch all events that are happening during the deployment. This is the data we’re after.

The Deployment Events

The up method of the stack object takes a config object. We can use the onEvent attribute to listen to all the changes.

stack.up({
 onEvent: (event) => {
 console.log(event)
 },
})

Here we log out every event. If we look at the output, we notice there are different types of events.

-The preludeEvent fires at the start with information about the stack config. -The resourcePreEvent fires before a resource is deployed, and the resOutputsEvent fires after it has been deployed. -The diagnosticEvent contains eventual errors that could happen. -The summaryEvent includes info about what has been done and if e everything was successful.

The event structure is a bit strange. The events don’t have a type attribute we could switch over, but an attribute called like the type of the event.

The following code shows how to get the right event, and where the data hides in that event:

onEvent: (event) => {
 if (event["preludeEvent"] !== undefined)
 return console.log(event.preludeEvent.config)

 if (event["resourcePreEvent"] !== undefined)
 return console.log(event.resourcePreEvent.metadata)

 if (event["resOutputsEvent"] !== undefined)
 return console.log(event.resOutputsEvent.metadata)

 if (event["diagnosticEvent"] !== undefined)
 return console.log(event.diagnosticEvent)

 if (event["summaryEvent"] !== undefined)
 return console.log(event.summaryEvent)
},

If we run things like this, we get all the information we need to send to Honeycomb. The next step is to tell Honeycomb what’s up.

Sending Data to Honeycomb

Honeycomb provides us with two libraries. The Node.js Beeline and Libhoney.

Libhoney is a low-level Honeycomb client that allows sending raw events to their service API. The Beeline is an abstraction on top of Libhoney to help with instrumentation and traces.

Usually, it would be best if you used the Beeline because it’s easy to set up and works with Node.js’s internal HTTP module and the Express framework out of the box.

In this case, we aren’t building an HTTP server, so the Beelines instrumentation doesn’t help much.

So for our deployment example, I will stick to Libhoney.

Honeycomb Events & Traces

Honeycomb uses events to figure out what’s happening in your system. Suppose the events have special attributes, like trace.trace_id and trace.span_id, Honeycomb can correlate them. This way, it can tell you things like, “The trace for your request took 300ms and consisted of 100ms authentication and 200ms database access.”

So you can pump all your information to Honeycomb via events. Sometimes it doesn’t need to be correlated; you only want to tell Honeycomb that your server uses 20% of memory. But in our case, we want to tie up all events related to one deployment into a trace, so we can figure out what we deploy, how long it takes, and if something goes wrong, what resource was responsible.

Linking up Pulumi & Honeycomb

We need to initialize Libhoney and send the correct “Honeycomb event” for every “Pulumi event.”

But let’s start with getting one event into Honeycomb: the pulumi up event.

const Libhoney = require("libhoney")
const { LocalWorkspace } = require("@pulumi/pulumi/automation")

const hny = new Libhoney({
 writeKey: "<HONEYCOMB_API_KEY>",
 dataset: "example-iac",
})

const id = (name) => `${name}-${Date.now()}`

const traceId = id`trace`
const rootSpanId = id`trace-root-span`
const startTimestamp = Date.now()

const main = async () => {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up({
 onEvent: (event) => {},
 })
}

main().then(() => {
 hny.sendNow({
 name: "up",
 service_name: "Pulumi",
 "trace.trace_id": traceId,
 "trace.span_id": rootSpanId,
 duration_ms: Date.now() - startTimestamp,
 })
})

We define a traceId, a rootSpanId, and a startTimestamp at the beginning, and then we send it off when everything is done.

An event with trace.trace_id will be grouped up with all other events with the same trace ID. In this case, we only have one event.

The name is usually the action you perform and want to measure, and the service_name is, well, the service that performs this action. In this case, we run pulumi up.

In the end, we also send a duration_ms attribute to track how long things take.

The result will look like this:

Not very interesting, but at least we know it run without crashing and how long it took.

Getting More Details

The next step is to get the details. What were the config parameters? How long did different parts of the deployment take?

For this, we need to look into two events, the preludeEvent and the summaryEvent.

Until now, we only send one Honeycomb event for the Pulumi up command.

Now, we will send three events for the stack.

The init event will contain the time it took to get to the actual deployment of resources and the config parameters of the stack
The run event will contain the duration all resources took to do their thing.
The finish event will contain the duration of shutting down the automation API and the summary data.

Let’s look at the updated example code:

...

const traceId = id`trace`
const rootSpanId = id`trace`
const startTimestamp = Date.now()
let stackFinishStartTimestamp

const main = async () => {
 const initStartTimestamp = Date.now()
 const runStackSpanId = id`stack`
 let stackRunStartTimestamp

 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up({
 onEvent: (event) => {
 if (event["preludeEvent"] !== undefined) {
 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(initStartTimestamp)
 hnyEvent.add({
 name: "init",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": id`stack`,
 duration_ms: Date.now() - initStartTimestamp,
 ...event.preludeEvent.config,
 })
 hnyEvent.send()
 stackRunStartTimestamp = Date.now()
 return
 }

 if (event["summaryEvent"] !== undefined) {
 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(stackRunStartTimestamp)
 hnyEvent.add({
 name: "run",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": runStackSpanId,
 duration_ms: Date.now() - stackRunStartTimestamp,
 })
 hnyEvent.send()
 stackFinishStartTimestamp = Date.now()
 return
 }
 },
 })
}

main().then(() => {
 let hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(stackFinishStartTimestamp)
 hnyEvent.add({
 name: "finish",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": id`stack`,
 duration_ms: Date.now() - stackFinishStartTimestamp,
 })
 hnyEvent.send()

 hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(startTimestamp)
 hnyEvent.add({
 name: "up",
 service_name: "Pulumi",
 "trace.trace_id": traceId,
 "trace.span_id": rootSpanId,
 duration_ms: Date.now() - startTimestamp,
 })
 hnyEvent.send()
})

First, we need to sprinkle variables on the code that capture the start times of our three events.

The init event starts before the call to the up command and takes until the preludeEvent is fired by the automation API.

The run event starts right after the init event finishes and takes until the summaryEvent fires.

The finish event starts right after run finishes and stops right before the parent event for Pulumi up is finished.

These events all get the trace.parent_id of the Pulumi up event, so they get nested in the Honeycomb view later.

Also, the init event gets the event.preludeEvent.config object, so we can see how the stack was configured inside Honeycomb.

If we run that example, we should see something like this:

Getting Resource Deployment Durations

The resources have additional details for us. As a name for the event, we will use the operation (create, update, delete) and as service_name we use the name of the resource in the Pulumi definition.

To get the parent of a resource, we have to save the URNs of every resource for later. But since we have to keep the start times anyway, we can do this in one step when the resourcePreEvent is fired.

Let’s look at the example code. This time I removed some boilerplate code and the other events to make things a bit more readable.

const main = async () => {
 // ...

 const resourceSpans = new Map()

 // ...

 await stack.up({
 onEvent: (event) => {
 // ...

 if (event["resourcePreEvent"] !== undefined) {
 const { metadata } = event.resourcePreEvent
 resourceSpans.set(metadata.urn, {
 spanId: id`span`,
 startTimestamp: Date.now(),
 })
 return
 }

 if (event["resOutputsEvent"] !== undefined) {
 const { metadata } = event.resOutputsEvent

 const serviceName = metadata.urn.split("::")[3]
 const { spanId, startTimestamp } = resourceSpans.get(metadata.urn)
 const parentUrn = metadata.new?.parent ?? metadata.old?.parent
 const parentId = resourceSpans.get(parentUrn)?.spanId ?? runStackSpanId

 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(startTimestamp)
 hnyEvent.add({
 name: metadata.op,
 service_name: serviceName,
 "trace.trace_id": traceId,
 "trace.parent_id": parentId,
 "trace.span_id": spanId,
 duration_ms: Date.now() - startTimestamp,
 })

 if (metadata.op === "update")
 hnyEvent.addField("diffs", metadata.diffs.join(", "))

 hnyEvent.send()
 return
 }

 // ...
 },
 })
}

So, what’s going on here?

First, the resources are deployed by their hierarchy, but siblings can be deployed in a different order. That’s why we need to save their spanId and startTimestamp when we get a resourcePreEvent.

I used a Map for this and keyed it with the URN of the resource because URNs are used for parent-child relationships and are part of all events we care about here.

Later, when the resOutputsEvent fires, it’s time to wrap it all up and send it to Honeycomb.

The event has the URN to find the related spanId and startTimestamp. We can also use the URN in the parent attribute of our resource to fetch the spanId of our resource’s parent. This way, our spans are correctly nested in Honeycomb later.

If the operation for the resource is an update, we also add the diffs to that event. This way, we can see in Honeycomb what inputs have changed since the last deployment.

The final trace should look a bit like this:

Getting Errors

The last step, getting deployment errors, I leave as an exercise for the reader.

But I’ll give you a few hints!

First, the interesting event is the diagnosticEvent, which comes with a severity and message attribute. If you only need errors, you have to filter out debug and info severity.

Also, the event includes an urn that can be used to correlate the error to a Honeycomb span later (resourceSpans etc.). If the errors isn’t related to an URN, you can simply use the run stack event as target.

If you add an error field to a Honeycomb event, it will automatically counted to your errors in Honeycomb.

Summary

IaC observability becomes rather simple when using Pulumi and Honeycomb.

Pulumi’s automation API allows us to control the CLI with JavaScript, and Honeycomb’s Libhoney is low-level enough to instrument every part of our deployment process.

This way we can keep track which resources are slowing our deployment down, or which was responsible for a failed deployment.

This article was originally posted at https://dev.to/fllstck/observable-infrastructure-as-code-52ha. Reposted here with permission.

Pulumi and LocalStack — beyond the basics

Josh Graham — Mon, 10 Jan 2022 11:10:06 -0600

Recently, Pulumi community member Josh Graham decided to bootstrap a simple application using a serverless approach, with a focus on using good engineering practices and being able to run the application locally. Given that Josh is the (OG) SaaS architect of Atlassian and an AWS user, LocalStack was a natural choice.

Introduction

I first encountered Pulumi a few years ago when I was getting a little frustrated at the lack of some expressibility in Terraform. I’ve been using Terraform from almost day 1 and it really does the job well — providing you really grok its declarative nature, are very good at modularisation, and never need an escape hatch to non-declarative instructions. At the time, though, Pulumi wasn’t ready for me to use, so I watched from a distance.

Recently, I decided to bootstrap a simple application using a serverless approach, making it a good opportunity to try Pulumi again. I still wanted to use good engineering practices and be able to run the application locally. As I was the (OG) SaaS architect of Atlassian and am an AWS user, LocalStack was a natural choice.

Pulumi has come a long way since I first saw it. I am super happy with it so far and I’m using it for real infrastructure work at the startups I’m working with. It seems the software engineers have less cognitive load using a more familiar language to express infrastructure definitions. It is worth noting that Pulumi uses Terraform in various ways under the covers and I’m still a big fan of Terraform and Hashicorp in general.

The Application and its Infrastructure

The application I’ve based the following sample on is a website with a simple form to capture a visitor’s email address, responding with a cookie to set so the site knows not to bother showing them a form the next time they visit.

In keeping with a serverless goal, I use AWS DynamoDB to store the information. I also want to be able to query for the latest new contacts. In order to try out Pulumi more thoroughly (and although I’m no fan of NodeJS), I decided to do the backend on NodeJS in an AWS Lambda so as to try out the Pulumi Cloud API. Both the backend and the infrastructure-as-code with Pulumi are coded using TypeScript.

There’s lots more to the application, but the extract in the following sample is sufficient to showcase #Pulumi and #LocalStack using a #TDD approach to #IaC for #serverless apps.

The Sample

Unlike every sample or blog I found for either Pulumi or LocalStack, I wanted to organize the code properly. Inevitably, most samples did nothing other than return a simple string as a response (a distributed “hello world”). Otherwise, everything I found was one big file with no cohesion and no separation of concerns. Even worse, the samples mix the IaC “deploy/provision-time” code with the application “runtime” code in the same file. Exactly zero had any tests, let alone ones that were written to specify the expected behaviour.

I’ve divided this article into various sections depending on what you’re wanting to focus on.

Modularized Pulumi
Running Pulumi locally with LocalStack
TDD infrastructure code and an AWS Lambda calling DynamoDB

If you’d like to follow along or use parts as a template, you can fork the repository.

Acknowledgements

Grazie mille to my former ThoughtWorks colleague Toni Terreno for reviewing the article and the code to help smooth out some wrinkles.

Modularized Pulumi

As with any Pulumi project, it is defined by a Pulumi.yaml file. I’ve split the “infrastructure code” — the code that drives Pulumi — apart from the application code. This infrastructure code is in the infra directory. So, the package.json also specifies a main (even though this app isn’t a module for others to include) for pulumi CLI to find the Pulumi entry point.

Outputs / Inputs

There’s probably a good reason most samples that show how to use Pulumi are one big file. That’s because it can be initially difficult to grok the concept of provision-time “output” and “input” values. This isn’t isolated to Pulumi — I’ve seen similar problems with examples (and real code) with other IaC tools like Terraform, Puppet, and Chef. It’s not a big surprise, either. When thinking imperatively or defining infrastructure manually we tend to have a different mindset to the one required to think declaratively.

Despite the documentation, it can be quite confusing and becomes even more so when you deviate from the “one big file”. It can become positively baffling when, as this application does, the infrastructure code is aware of the application code and performs the deployment of the application as part of the infrastructure update. I wanted to test the application logic separately from having to run Pulumi, so that separation of code was essential.

The big thing to remember is that a Pulumi “output” value is not known until after the infrastructure has been fully provisioned. They are a promise of a future value. If you want to refer to the value in other infrastructure code, Pulumi creates a dependency graph to be able to resolve the value at the right time. If, however, you want to refer to the value at runtime and you want to define that runtime code in a modular way, you have to be very clear about how that value is presented by Pulumi to that runtime code.

Our infrastructure code provisions a DynamoDB table and an AWS Lambda. That Lambda needs to know the name of that table at runtime in order to query it. Because the actual name of the table isn’t known until provisioning is underway (Pulumi creates many infrastructure resources based on the logical name but with suffixes to avoid naming collisions), we need a way to get the “output” table name from Pulumi’s aws.dynamodb.Table API to the Lambda we’ll create with the “simpler” way to create a Lambda using Pulumi’s cloud.API API.

You’ll see in the ContactApi infrastructure resource an Inputs class with static members. This is the key to transferring, in the simplest way I’ve found, values from provision-time infrastructure code to runtime application code. This class is only available in that file, so for each resource that needs to pass provision-time values to runtime code, you can create one specifically for the values you want to pass.

In the case of ContactApi we receive a Table as an “input”, convert it to an “output” and store it in a static variable. This variable is then used in the constructor for our Lambda application class when wiring up the integration between the API Gateway route / method and the Lambda. The way the Lambda constructor is called in the infrastructure code is the same as we’d do from a test or any other application code that wanted to call it.

I played with various ways to transfer Pulumi inputs to runtime code but this was the most intentional, most decoupled, and least convoluted.

Resources

As mentioned above, we provision a DynamoDB table, an API Gateway, and a Lambda.

We could (and indeed I originally did) provision the table with the “simpler” cloud.Table API but I wanted specific control over the attributes and secondary indexes for DynamoDB (so I could, for example, query for new records since a particular date). The cloud.Table abstraction works across cloud platform providers, and there may be mechanisms to offer finer-grained / platform-specific options (but I haven’t found them yet).

We do provision a Lambda, but with the actual simplicity of this application’s needs, we could dispense with the Lambda and have the API Gateway do a non-proxy integration straight to DynamoDB’s HTTP API. Given many applications would include a Lambda somewhere along the way, I thought it was worth showing Pulumi provisioning a Lambda and integrating it to an API Gateway using it’s cloud.API abstraction.

AWS tags

Because we’re using AWS, we can set default tags on taggable resources. These can be put into the configuration file (Pulumi.<stack>.yaml) which is great for static / non-computed, optional tags.

If we want to ensure certain tags are applied, even if they were omitted from the stack-specific configuration, or the tag keys or values are computed, then we can use the registerStackTransformation method of Pulumi runtime. Our [registerAutoTags](https://bitbucket.org/delitescere/contact/src/main/infra/registerAutoTags.ts) (based on some code by Joe Duffy) function does this nicely for us, while including resource-specific tags and overrides.

Pulumi + LocalStack

Using LocalStack to emulate AWS services on your local machine is a great way to get faster feedback, isolate errors, and reduce costs. It’s obviously not a perfect emulation but for most purposes, it is more than sufficient.

Stack templates

Because Pulumi doesn’t (yet!) support project-wide configuration, I have a project-wide template stack.template.yaml (for any stack) and a local stack template local-stack.template.yaml (just for stacks targeting LocalStack).

These templates are merged and interpolated with shell environment variables in the init-local-stack script. Keys in the local template override keys in the non-local template.

You may wish to use a similar technique to the concatAndInterpolateTemplates function for non-local stacks. Perhaps each engineer has their own stack, but you have configuration keys that everyone should set, that either have common values or machine-specific values. In combination with various .env files (including .env files per-stack), this could be quite useful for those of you, like Tony, with many engineers (who have a stack each) and also many shared stacks.

In a subsequent article, I’ll write about how to set things up if you’re using multiple AWS accounts, across multiple regions, each with multiple VPCs.

Dockerized Lambdas

One issue I encountered was because the Lambda I wrote uses the NodeJS runtime, LocalStack will create a Docker container to run it in. While much of the necessary environment was copied to that container, a subtle bug in LocalStack’s Lambda executor failed to include the AWS credentials in the environment. This meant that any other resources the Lambda needs to access (like a DynamoDB table) would fail to authorize as the requests could not be authenticated. I fixed the bug, added some tests, and the behavior is now correct from LocalStack version 0.12.16 (available since 1 Aug, 2021.

Using the included docker-compose file, when you docker compose up --detach you will have a LocalStack container running with the requisite faked AWS services for this application.

If you ever want to get a shell into that container, you can either use standard docker compose exec commands if you didn’t install localstack, or use MAIN_CONTAINER_NAME=$(basename $PWD)_localstack_1 localstack ssh if you did install it (you don’t have to install it because Docker Compose is handling it for you).

The docker-compose file also mentions what to change if you’re going to be doing lots of tests / Lambda invocations. In this case, you pay the price of a slower pulumi up—because the Lambda creation is writing lots of files, particularly node_modules— to the file system. If that is being synchronized to the host file system, in my single Lambda case, the infrastructure creation time goes from about 20 seconds to about 50 seconds. However, you get back time in Lambda invocations when LAMBDA_REMOTE_DOCKER is falsey which takes off roughly 1/3 of the time to execute a Lambda.

TDD Infra + Lambda + DB

This test set up might not look like many JavaScript projects you’ve encountered. We have “unit” tests segregated from “functional” tests. If we had more things to test, I might have “integration” tests, too.

If you’re curious about the different approaches to unit testing I’ve employed here, please read the excellent (now almost two decades old!) “Mocks aren’t Stubs” article by my friend and Thoughtworks Chief Scientist, Martin Fowler.

Unit tests

These are “pure” logic tests requiring just the JavaScript class under test and test-doubles for infrastructure dependencies.

In the case of our contact Table class, we mock the DynamoDB SDK and stub responses from its getItem and putItem methods to match the case we want to test. In the case of our contact Lambda class, we mock itsTable class and stub responses from its findOne and insert methods, and we also need to mock the Pulumi Cloud API’s Request and Response classes.

Although the infrasructure-as-code in our project driving Pulumi is fairly declarative and therefore doesn’t really need testing, we do have a little logic that should be tested. In those cases, Pulumi provides the capability set mocks for testing purposes so you don’t have to be running the Pulumi engine for unit tests.

Unit tests can be run with npm test unit.

Stack class

We’ve abstracted the notion of a “stack” into something that is useful in the context of having using LocalStack or a real AWS region, and having “production” and non-production environments. As mentioned earlier, things can get a lot more complicated than that for more sophisticated AWS users, which I’ll write about in a subsequent article.

In the jest config we include tests/env.ts as a setup file. This sets the environment variables before imports and static initialization, allowing us to test things like aws.config.region having been set from the environment. You can also set this value in the Pulumi.<stack>.yaml file.

We use the pulumi.runtime.setMocks function so that we can mock the parts of Pulumi called by our Stack class (getProject and getStack). We now specify the logic for when a resource is qualified by the stack name (e.g. contact-sandbox for the sandbox stack versus contact for the production stack), and also for when a DNS record is qualified by the stack name (e.g. www.example.com for the production environment record but www.test.example.com for the test environment record).

Result type

I don’t particularly like the JavaScript approach to Promises and its many flaws. However, as this exercise is for a fairly simple function-as-a-service, I didn’t want to go down the path of using a replacement approach like Fluture. I did, however, want to work with the results of promises using a something approximating a sum type and be able to transform exceptions and rejected promises into the “not right” side of that type.

The Result type uses lightweight type checking and convenience methods. All the Promises we are working with end up resolving to a Result which is either an Error or a non-error value. We’re careful to catch rejected Promises or exceptions and turn them into a resolved Promise<Error>or Error.

Table class

The Table class in contact only has two methods so far, findOne and insert, because that’s all the application needs right now. You can imagine other applications might need other typical methods like find, findOneByEmail, delete, and so on.

To use this class in a unit test, we need to mock the DynamoDB client and provide various stubs for its getItem and putItem methods. Each of these stubs tests a different code path in how Table handles responses from the database.

If we had more than one table to interact with, there’d probably be an interface and perhaps an abstract class or other mechanism of providing behavior common to all tables.

Lambda class

Given the combination of inputs to the lambda and processing issues that may arise, we want to be able to test the lambda behaves in the expected way and handles those issues as specified.

We provide stub responses on the Table class methods to stimulate successful and unsuccessful operations. We also stub the input to the lambda (because we’re using the Pulumi Cloud API this is a pulumi.cloud.Request) to initiate different logic in the lambda.

The result of the lambda is a pulumi.cloud.Response which we mock to capture the HTTP status code generated, which is sufficient to verify almost all of the behavior of this lambda. The other thing we need to verify is that a HTTP cookie is set when either a new item is created or a matching previously created item is found.

AbstractLambda class

Here are some convenience methods that lambdas might generally use. We have two for dealing with when the lambda is running on LocalStack and two for working with HTTP request headers and JSON entities.

Normally, the “endpoint” for AWS is left undefined, which is interpreted bu the AWS SDK as “use the default for the region”. When running in LocalStack, however, we set it to the URL that LocalStack is listening to on our local machine.

When running locally, we also include error information in the HTTP Response to help make diagnosing issues during development easier. (I’m still shocked that the NodeJSErrorhas non-enumerable properties that mean*JSON.stringify*is useless on instances of it).

To handle the situation where, depending on the HTTP client’s approach and because the NodeJS libraries don’t normalize for us, the same HTTP request header may either be repeated in the request or have been concatenated with multiple values (with some delimiter, usually a comma), we have a convenience method to deal with that.

To avoid lots of conditionals and error checking when parsing the request entity, we have a convenience method to parse it as JSON. I hope you like the Easter Egg in the test Tommy Hall and Dave Coombes 😆

Functional tests

These test the application as it would be used in production. The infrastructure must be set up, the application must be running, and the tests interact with the application like a user or integrating system would.

The [bin/test](https://bitbucket.org/delitescere/contact/src/main/bin/test) script does the heavy-lifting of starting LocalStack, initializing a Pulumi stack for testing, running the Jest tests, and pulling everything down again, all handily invoked directly or more likely just with npm test.

If something goes wrong with the tests, or you use KEEP_TEST_STACK=1 npm test, the Pulumi test stack and LocalStack container stays around for diagnostic purposes.

I have found that this test can be “brittle” due to timeouts when the LocalStack container is low on resources, or your computer is running slow and the Pulumi automation API results are slow (it seems exec’ing a pulumi stack outputs command from the test is more efficient, albeit clumsier).

Use docker compose down --volumes to totally clear out the LocalStack environment.

Contact resource

Given everything else has been tested in unit tests, this functional test doesn’t need to do much. However, it gives us confidence of the following:

that Pulumi has hooked up the API Gateway and Lambda properly
that LocalStack is emulating the AWS services we need sufficiently well
and most importantly, that our Lambda actually does talk to DynamoDB correctly at runtime (as the unit tests have mocked this out)

That’s it, I hope this has given you some hope that your Pulumi code doesn’t have to be a big ball of mud, and that using LocalStack to try it out before hitting AWS is straightforward.

This article was originally posted at https://delitescere.medium.com/pulumi-and-localstack-beyond-the-basics-d993f3b94d17. Reposted here with permission.