Pulumi Blog: Community

KubeCon EU 2026 Recap: The Year AI Moved Into Production on Kubernetes

Engin Diri — Wed, 01 Apr 2026 00:00:00 -0700

Amsterdam in late March still has that sharp North Sea wind, but inside the RAI Convention Centre, 13,350 people generated enough energy to heat the building twice over. KubeCon + CloudNativeCon EU 2026 was the biggest European edition yet, and the shift from previous years was impossible to miss. AI dominated the conference.

I spent most of the conference at the Pulumi booth, and that turned out to be the best vantage point. Hundreds of visitors stopped by over four days, and I kept asking the same question: what are you actually running in production with AI on Kubernetes? The answers shaped this post more than any keynote did. Almost everyone had a proof of concept. Almost nobody had a production story they were happy with.

Here is the stat that framed the entire conference for me: 66% of organizations use Kubernetes to host generative AI workloads, but only 7% deploy to production daily. That gap between experimentation and actual production use matched what I was hearing at the booth. The CNCF’s own survey now counts 19.9 million cloud native developers worldwide, 7.3 million of them building AI workloads. The tooling and the infrastructure need to catch up.

My takeaway after four days on the ground: lots of working demos, very few production setups people trust. Teams are trying to scale inference, put guardrails around agents, and make GPU infrastructure behave like anything else they run.

Here is what I saw.

From training to inference: the big pivot

About 67% of AI compute now goes to inference, not training. The inference market is projected to hit $255 billion by 2030. It’s also where most of the operational complexity lives.

NVIDIA leaned into this hard. Their open-source stack around NeMo and Dynamo got significant stage time, but the bigger move was donating three projects to the CNCF: the DRA driver for fractional GPU allocation, the KAI Scheduler for GPU-aware scheduling, and Grove. Moving these to community governance signals that GPU infra is becoming part of the standard Kubernetes toolkit.

The CNCF donations that will reshape AI on Kubernetes

Every KubeCon has its crop of new CNCF projects, but this year’s batch felt different. We are starting to see the building blocks of an AI runtime for Kubernetes.

llm-d was the headline donation. Created by IBM Research, Red Hat, and Google Cloud, it splits inference workloads by separating prefill and decode phases across different pods. The collaborator list reads like an industry consortium: NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley, and UChicago. When that many organizations agree on a single approach to distributed inference, pay attention.

NVIDIA’s DRA driver enables fractional GPU allocation and multi-node NVLink support. GPU multi-tenancy is one of the hardest unsolved problems in Kubernetes right now. Scheduling, isolation, cost attribution — all of it breaks down when multiple workloads share a GPU. The DRA driver does not solve everything, but it gives the community a real starting point.

KAI Scheduler entered the CNCF Sandbox for GPU-aware scheduling. If llm-d handles the inference runtime and the DRA driver handles allocation, KAI Scheduler handles placement. Together, these three projects form the skeleton of a GPU-native Kubernetes stack.

Velero, donated by Broadcom, moved into CNCF Sandbox for backup and restore. AI workloads are stateful now (model weights, checkpoints, fine-tuning data), and backup is no longer optional. Good timing.

Microsoft AI Runway is an open-source Kubernetes API for inference that plugs in Hugging Face model discovery, GPU memory fit calculations, and cost estimates. Think of it as a model-aware control plane. HolmesGPT and Dalec, also from Microsoft, entered CNCF Sandbox for AI-powered troubleshooting and dependency analysis.

The Kubernetes AI Conformance Program is growing fast, with certifications nearly doubled and three new requirements proposed for Kubernetes 1.36. Conformance programs are boring until they are not. This one will determine which distributions can credibly claim AI readiness.

Agentic AI gets an identity layer

If inference was this year’s production story, agentic AI was the architecture story. Agents are proliferating, and nobody has quite figured out how to manage and secure them inside Kubernetes yet.

kagent, donated to CNCF Sandbox by Solo.io, defines agents as Kubernetes CRDs. It ships with pre-built MCP (Model Context Protocol) servers for Kubernetes, Istio, Helm, Argo, Prometheus, Grafana, and Cilium. An agent becomes a first-class Kubernetes resource, schedulable and observable and subject to RBAC, instead of a rogue process running in someone’s notebook.

kagenti from IBM goes after the identity problem directly. Using SPIFFE/SPIRE, it gives agents cryptographic identities. When an agent calls an API, you can verify exactly which agent made the call, what trust domain it belongs to, and whether it is authorized. This kind of security work needs to happen before agents proliferate across production clusters. Retrofitting identity later is ugly.

Dapr Agents took a different angle with the actor model and durable execution. Each agent gets reliable state management and exactly-once messaging semantics. If your workflows cannot tolerate lost messages or duplicate actions, this matters.

agentregistry showed up as a centralized discovery service for MCP servers and agents. As agents and tool servers multiply, you need a registry to find and manage them, the same way container registries became necessary for images.

David Soria Parra from Anthropic gave a talk on MCP evolving beyond simple tool-calling into richer interaction patterns (sched). Google announced the Kubernetes Agent Sandbox for running agentic AI workloads in secure, isolated environments.

AI gateways and inference routing

Gateway infrastructure had its own mini-conference within KubeCon. The Gateway API Inference Extension from the Kubernetes SIG introduces model-aware routing and load balancing at the gateway level. Instead of routing by URL path, your gateway routes by model name, version, and capacity. That changes how inference traffic flows through a cluster in a fundamental way.

Envoy AI Gateway builds on Envoy’s existing proxy capabilities with token-aware rate limiting and provider failover. If your primary inference provider is saturated, traffic shifts to a secondary automatically. Rate limiting by token count rather than request count makes much more sense for LLM workloads, where a single request can consume vastly different amounts of compute.

I want to call out Agentgateway specifically. Written in Rust, it proxies LLM traffic, MCP connections, and agent-to-agent communication, with Cedar and CEL policy engines for fine-grained access control. Rust’s performance characteristics matter here because inference gateway latency adds directly to user-perceived response time.

Kuadrant, now in CNCF Sandbox, layers policy on top of gateway infrastructure and includes MCP server aggregation. Gateways are evolving from dumb traffic proxies into intelligent control planes for AI workloads, and these four projects are driving that shift.

Platform engineering absorbs LLMOps

The observability and platform engineering vendors showed up in force. The message was consistent: LLMOps is just platform engineering with new requirements.

Chronosphere demonstrated parallel AI investigation, with multiple agents analyzing different aspects of an incident simultaneously and combining their findings. SUSE Liz takes a domain-specialized approach, deploying different AI agents for different operational domains rather than one general-purpose assistant. groundcover combines eBPF with OpenTelemetry to give coding agents rich runtime context about the systems they are modifying. That last one is subtle but important: if an AI agent is writing code that touches a service, it should understand that service’s actual runtime behavior, not just its source code.

Dynatrace and DevCycle partnered to make feature flags observable primitives via OpenFeature. Rolling out AI features behind feature flags is table stakes, but having those flags show up in your observability pipeline as first-class signals closes a real gap.

Shadow AI governance emerged as its own theme. CAST AI’s Kimchi can route requests across 50+ models while providing centralized visibility into what models are being used, by whom, and at what cost. Every large organization I talked to had some version of the same problem: teams spinning up model endpoints without central oversight, burning through GPU budgets, creating compliance blind spots they did not even know about.

GPU multi-tenancy remains genuinely unsolved. Scheduling, workload isolation, cost attribution across shared GPUs — all of it breaks down at scale. Multiple talks addressed pieces of this, but nobody had a complete answer.

Sovereignty shapes infrastructure architecture

Regulation came up in almost every conversation. The EU Cyber Resilience Act is driving compliance requirements deep into software supply chains, and every European organization I spoke with is feeling the pressure. Teams are already changing how they build and deploy software.

Sovereign Kubernetes is a platform architecture requirement now, not something you can defer to next quarter. Organizations need Kubernetes distributions and cloud regions that guarantee data residency, and they need the tooling to enforce those guarantees programmatically. Self-hosted models are proliferating partly because of capability and cost, but data sovereignty is the accelerant. If your data cannot leave a jurisdiction, neither can your model.

Runtime isolation is expanding beyond containers. Several talks covered KVM-based isolation for AI workloads, which is heavier than containers but necessary when the threat model includes side-channel attacks on shared GPU memory. The sandboxing conversation has gotten more sophisticated since last year.

These constraints are not uniquely European. Any organization operating across jurisdictions faces similar pressures, and the regulatory direction globally is toward more data sovereignty requirements, not fewer.

What this means for your team

Four days in Amsterdam distilled into five things I would act on now:

Treat inference workloads like production services. If you are still deploying models with scripts and hope, stop. Inference infrastructure needs the same IaC discipline as any other production system: version-controlled, tested, policy-enforced.
Evaluate the Gateway API Inference Extension and llm-d. These are not speculative projects. They have broad industry backing and solve real problems around inference routing and distributed serving. Get them into your test environments.
Plan agent identity before agents proliferate. SPIFFE/SPIRE for agent identity is not optional if you are running agents in production. Retrofitting identity onto an existing agent fleet is painful. Start with kagenti now.
Platform teams should own AI infrastructure. Shadow AI is already happening in your organization. The platform engineering team needs to provide self-service AI infrastructure with guardrails before ungoverned model endpoints become a security and cost problem.
Sovereignty and GPU multi-tenancy are universal. Even if you are not subject to the EU Cyber Resilience Act today, data residency requirements are spreading globally. GPU multi-tenancy will affect every organization running inference at scale.

Kubernetes spent the past decade proving it could orchestrate containers. The next decade will test whether it can orchestrate intelligence. Based on what I saw in Amsterdam, the community is building the right pieces, but the gap between what exists and what production demands is still wide. That gap is where the interesting work happens.

Adding PostgreSQL State Backend Support to Pulumi: A Community Contribution Journey

Matan Baruch — Thu, 19 Jun 2025 14:46:07 +0300

When managing infrastructure as code at scale, reliable state storage is essential. Pulumi Cloud provides a fully managed, secure, and scalable solution out of the box. For teams that choose to build and maintain their own backend, Pulumi now offers support for PostgreSQL as a DIY state storage option—though this requires additional operational overhead and careful consideration around performance, security, and maintenance.

The Need for Database-Backed State Storage

Traditional DIY backends in Pulumi have relied on object storage systems like AWS S3, Google Cloud Storage, or Azure Blob Storage. While these work well for many use cases, they have limitations when it comes to handling very large state files, complex locking mechanisms, and transactional guarantees that some enterprise environments require.

PostgreSQL stood out as an excellent candidate for state storage due to its:

Large object support: Ability to handle substantial state files without size constraints
ACID compliance: Robust transactional guarantees for state consistency
Mature ecosystem: Well-established tooling and operational practices
Scalability options: From single instances to complex replication setups
Security features: Comprehensive authentication and authorization capabilities

The Community Contribution Process

This feature came to life through PR #19581, which addressed a long-standing community request tracked in issue #5632. The development process showcased the collaborative nature of open-source development, with multiple rounds of feedback, testing, and refinement.

Key Technical Challenges

The implementation required several technical considerations:

1. Integration with Go Cloud Development Kit (CDK)

The PostgreSQL backend needed to integrate seamlessly with Pulumi’s existing blob storage abstraction layer. This was achieved by implementing the blob.BucketURLOpener interface:

// URLHandler is a URL opener for PostgreSQL URLs.
type URLHandler struct{}

// OpenBucketURL implements blob.BucketURLOpener.
func (p URLHandler) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error) {
 pg, err := NewPostgresBucket(ctx, u)
 if err != nil {
 return nil, err
 }
 return pg.Bucket(), nil
}

2. Database Schema Design

The implementation uses a simple but effective schema for storing state data:

CREATE TABLE IF NOT EXISTS pulumi_state (
 key TEXT PRIMARY KEY,
 data JSONB NOT NULL,
 created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
 updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

This design allows for efficient key-based lookups while leveraging PostgreSQL’s native JSON support for flexible state data storage.

3. Cross-Platform Testing

One of the significant challenges was ensuring the implementation worked across different operating systems and CI environments. The solution involved using testcontainers-go to spin up PostgreSQL instances during testing, with appropriate skip conditions for environments where Docker isn’t available:

func skipIfDockerNotAvailable(t *testing.T) {
 if runtime.GOOS == "windows" || runtime.GOOS == "darwin" {
 t.Skip("Skipping test: Docker not available on this platform in CI")
 }

 // Additional Docker availability checks...
}

Performance and Limitations

PostgreSQL wins over a plain S3 bucket

ACID transactions with row-level locking – A postgres backend means strong consistency guarantees.
Faster for smaller state files – Community benchmarks show improvements on S3 backend speed for smaller state files.
Point-in-time restore – WAL-based backups allow for easier rollback.
Deep observability – plug pg-metrics into Grafana/APM for real-time insight.

Gaps you’ll still close only with Pulumi Cloud

Server-side policy guardrails (CrossGuard) – automatically enforced policies on every preview or deploy.
Always-on drift detection – automatic scans catch out-of-band changes.
Managed Deployments – fully managed runners, PR previews, and dependent stack updates.
Org-wide RBAC, SSO/SAML, immutable audit logs – turnkey for teams and auditors.
Search & analytics across all cloud resources (Insights) – not just those managed by Pulumi.
Enterprise Secrets & Configuration (ESC) - encrypted secrets and config with RBAC, versioning and audit trail.
SOC 2, high availability, and painless upgrades – Pulumi operates the control plane so you don’t have to.

Getting Started

Using the PostgreSQL backend is straightforward:

Login to your PostgreSQL backend:

pulumi login postgres://pulumi_user:secure_password@localhost:5432/pulumi_state?sslmode=require

Use Pulumi normally:

pulumi up
# Your state is now stored in PostgreSQL!

Community Impact

This contribution demonstrates the power of community-driven development in open source projects. It addresses a real need expressed by Pulumi users while maintaining the high quality standards expected from the project.

The implementation follows Pulumi’s architectural patterns and coding standards, making it a seamless addition to the existing codebase. The comprehensive documentation and testing ensure that future maintainers can easily understand and modify the code.

What’s Next?

The PostgreSQL backend opens up several possibilities for future enhancements:

High Availability: Support for PostgreSQL clustering and replication
Performance Optimizations: Caching strategies and connection pooling
Advanced Features: Custom backup strategies and state analytics
Multi-tenant Support: Isolation patterns for multiple teams or environments

Conclusion

The addition of PostgreSQL state backend support to Pulumi represents more than just a new feature—it’s a testament to the collaborative nature of open source development and the power of community contributions. By providing enterprise-grade state storage options while maintaining the simplicity that makes Pulumi great, this feature enables organizations to adopt infrastructure as code with confidence.

Whether you’re running a small startup or managing infrastructure for a large enterprise, having robust, reliable state storage options is crucial. The PostgreSQL backend provides an improved approach to Do-It-Yourself state management.

If you’re interested in contributing to Pulumi or have ideas for new features, the community welcomes your contributions. This PR serves as an excellent example of how community members can make meaningful improvements to the project while learning from experienced maintainers and following established best practices.

Introducing the Pulumi Puluminaries 2.0 Program

Engin Diri — Mon, 10 Feb 2025 12:15:00 -0800

We are excited to announce the Pulumi Puluminaries 2.0 Program. This is a fresh and revitalized way to celebrate and support Pulumi’s most passionate community members. Pulumi Puluminaries are individuals who demonstrate leadership in the Pulumi ecosystem by sharing best practices, creating valuable content, and helping fellow practitioners succeed.

Before we dive into what is new, we want to recognize and applaud the incredible achievements of our existing Pulumi Puluminaries. You can check out the great folks currently making a difference in our community on the Pulumi Puluminaries page. Their hard work and dedication have laid a strong foundation for what is next.

What is the Pulumi Puluminaries 2.0 Program?

The Pulumi Puluminaries 2.0 Program is designed to recognize and empower developers, architects, and technologists who are committed to driving innovation with Pulumi. This program offers exclusive opportunities to collaborate with Pulumi engineers, gain early access to new features, and share insights with the community.

Similar to how other ambassador programs bring together dedicated advocates around a specific focus, the Pulumi Puluminaries 2.0 Program unites a community of individuals who champion Pulumi’s approach to cloud engineering. Pulumi Puluminaries lead conversations, help new users adopt Pulumi effectively, and showcase best practices and use cases that highlight the power of Infrastructure as Code. They are also key participants in shaping Pulumi’s future by providing feedback to the product team and taking part in special initiatives.

What does it mean to be a Pulumi Puluminary?

Pulumi Puluminaries serve as champions of cloud engineering. They help users navigate the complexities of building, deploying, and managing applications with Pulumi. They also promote the benefits of Infrastructure as Code in modern organizations by creating thought leadership content, hosting or co-hosting events, and offering guidance for those just getting started.

Key Responsibilities for Pulumi Puluminaries:

As a Pulumi Puluminary, you will be expected to engage with the community at least once a quarter. Here are some ways you can contribute:

Community Engagement
- Actively share Pulumi-related articles, tutorials, and project stories.
- Present at conferences, meetups, or webinars on topics that highlight Pulumi’s value.
- Mentor new users and help them get started with Pulumi.
- Participate in Pulumi’s community forums, Slack channels, and social media discussions.
Thought Leadership
- Write blog posts (like Simen A. W. Olsen did in his latest post Your Perfect Infrastructure May Not Be So Perfect or Tyler Mulligan) or create videos showcasing innovative ways to use Pulumi.
- Contribute to discussions around best practices and patterns for cloud engineering with Pulumi.
Product Feedback
- Offer beta-testing support and provide product feedback directly to the Pulumi team.
- Participate in early-access programs and help guide product roadmap decisions.

Benefits of Joining the Program

Early Access: Get sneak peeks at new Pulumi features and releases before they are publicly announced.
Direct Collaboration: Collaborate one-on-one with Pulumi engineers and product managers, influencing future capabilities.
Recognition and Swag: Receive official Pulumi Puluminaries 2.0 digital badges, exclusive swag, and spotlight opportunities on Pulumi’s blog and at events.
Networking and Growth: Build relationships with peers in the Pulumi ecosystem, deepen your technical expertise, and grow your professional network.

How to Get Involved

We are looking for Pulumi enthusiasts who have shown dedication and passion for empowering others in the Pulumi community. Whether you have built cutting-edge projects, delivered insightful conference talks, or helped users overcome technical hurdles, we would love to hear your story.

Whether you want to apply for yourself or nominate someone else, we’d love to hear from you!

Apply or Nominate Now

Integrating DevOps and Security in Platform Engineering

Sara Huddleston — Wed, 11 Dec 2024 07:41:06 +0000

Platform engineering has become essential for mid-to-large organizations, moving beyond a DevOps trend. Gartner predicts that by 2026, 80% of software companies will have internal platform services to streamline development. The goal is to empower developers with self-service tools while maintaining security, compliance, and reliability through DevSecOps practices.

At PulumiUP Europe 2024, experts shared insights on aligning DevOps with security to build scalable, secure platforms:

Jess Mink, Sr. Director of Platform Engineering at Honeycomb
Kief Morris, Global Head of Infrastructure Engineering at ThoughtWorks
Lindsay Jack, VP of Engineering & Security at Snyk
Nariman Aga-Tagiyev, Application Security Architect at WiseFrog Security
Komal Ali, Engineering Manager at Pulumi

The panel discussed key strategies, challenges, and pillars of successful platform engineering.

The Core Pillars of Platform Engineering

Platform engineering teams comprise multiple professionals with many responsibilities and focus areas. According to our panel of experts, the core pillars of platform engineering include:

Developer Experience (DevEx): Provide developers with the tools, frameworks, and abstractions they need to be productive and proactive without getting stuck in infrastructure or operational concerns.
Reliability and Scalability: Ensure that the platform and infrastructure can support the organization’s needs, with the ability to scale up or down as required.
Security and Compliance: Embed robust, accessible security and compliance frameworks into the development lifecycle while making it easy for developers to adhere to these policies.
Automation and Tooling: Leverage Infrastructure as Code (IaC) and automation to enforce standardized processes and consistency and reduce cognitive load and manual effort.
Observability and Monitoring: Provide visibility into the platform’s health and performance, delivering actionable insights that allow teams to identify and resolve issues quickly.

These pillars work together to create a platform that empowers developers to innovate and deliver value to the organization and customers while maintaining the necessary controls and safeguards.

As Jess Mink, Director of Platform Engineering at Honeycomb, explains:

“The goal of platform engineering is to help the company run smoother and faster and unlock things people didn’t know were possible […] We tend to focus on tools and software, but it’s really about people, processes, and tools. If you consider this, platforms are responsible for social and technical support across the organization. A common pitfall is building tools no one uses because you didn’t meet people where they are.”

Aligning DevOps and Security for Secure Platform Engineering

Integrating security into platform engineering ensures it becomes a proactive part of the development lifecycle and ensures that “DevSecOps” is not an afterthought but a core consideration.

Key best practices shared by the panel include:

Shift Left Security

One of the fundamental principles of DevSecOps is to “shift left” - that is, to integrate security earlier in the development process rather than waiting until the end. This can involve activities such as:

Incorporating security requirements and threat modeling into the initial design phase
Automating security scans and tests as part of the continuous integration (CI) pipeline
Providing developers with secure coding guidelines and tools to identify and remediate vulnerabilities

By addressing security concerns upfront, organizations can reduce the time and cost of remediating issues later in the development lifecycle.

Embrace Automation and Standardization

Consistency is key. Platform engineering teams should leverage automation and built-in safeguards and security processes. This may include:

Automating the provisioning of secure infrastructure and application environments
Implementing Infrastructure as Code (IaC) to define and manage infrastructure in a declarative, version-controlled manner
Standardizing security controls, policies, and configurations across the platform

By automating these tasks, platform engineering teams can reduce the risk of human error, improve visibility and auditability, and free up developers to focus on building features rather than managing infrastructure.

Prioritize Observability and Monitoring

Effective security requires visibility into the health and performance of the platform. Platform engineering teams should invest in robust observability and monitoring solutions to:

Detect and respond to security incidents and anomalies in real-time
Gain insights into the behavior and usage patterns of the platform
Identify and address vulnerabilities or misconfigurations proactively

By integrating security-focused monitoring and alerting into the platform, organizations can quickly identify and mitigate threats while also providing developers with the necessary context to understand and address security-related issues.

Foster a Culture of Collaboration

Platform engineering is often referred to as being the practical application of DevOps practices. Integrating security practices often requires a cultural shift towards collaboration and shared responsibility between all teams in development, operations, and security, thus the name DevSecOps. Platform engineering teams can facilitate this by:

Treating Platforms as Products
Involving security stakeholders in the design and planning of platform initiatives
Providing security training and education to developers to empower them to make informed decisions
Establishing communication channels and feedback loops between teams

As Kief Morris, Global Head of Infrastructure Engineering at ThoughtWorks, explains:

“There is a new way of thinking that is trying to avoid that “build it, and they will come mentality,” which leads to building it and nobody using it. One of the trends we are seeing is product thinking […]—using techniques like creating user personas of different types of users in the organization, conducting research to understand their needs, and talking with them.”

Breaking silos and fostering communication helps organizations build secure, scalable platforms that support the needs of developers, platform engineers, architects, and security teams.

Challenges of Integrating Security in Platform Engineering

While the benefits of integrating DevOps and security in platform engineering are clear, the journey has expected challenges. Our panel of experts highlighted several key obstacles that organizations may face:

Balancing Autonomy and Control

The primary goal is to empower developers to be more productive and proactive. However, this autonomy needs to be balanced with necessary security controls and governance measures.

Jess Mink points out the importance of achieving harmony between developer autonomy and operational control, stating:

“It’s a delicate balance - you want to make things easy for developers, but you also need to maintain the right level of control and security. It’s about finding the right abstractions and interfaces that give developers the freedom they need while still ensuring the platform remains secure and compliant.”

Driving Adoption and Changing Mindsets

Integrating security into the platform engineering workflow can often be met with resistance from developers accustomed to moving quickly and may view security as an obstacle to their productivity.

As Nariman, a Software Security Architect, notes:

“The challenge is not just about the tools or the technology - it’s about changing the mindset and getting people to understand the importance of security. You need to find ways to motivate developers and make them feel like they’re part of the solution rather than just imposing more rules and processes.”

Effective communication, education, and a focus on the business value of security are key to driving adoption and fostering a culture of shared responsibility.

Adapting to Evolving Needs and Technologies

As organizations grow and their technology stacks evolve, the demands on the platform engineering team can shift rapidly. Keeping up with these changes, while maintaining a secure and reliable platform, can be a significant challenge.

Lindsay Jack, VP of Engineering for the Platform Division at Snyk, explains:

“You might have a platform team that’s really good at a certain set of technologies, but then the organization starts moving in a new direction, and suddenly those skills don’t match up anymore. It’s about being agile and adaptable and making sure you have the right mix of skills and expertise to support the organization’s evolving needs.”

Fostering internal mobility, continuous learning, and a flexible, modular platform architecture can help platform engineering teams navigate these changes more effectively.

Measuring Success in Secure Platform Engineering

Measuring the success of a platform engineering initiative can be complex as it involves balancing a range of technical, operational, and business-oriented metrics but also considers human factors. According to our panel, some key metrics to include:

Developer Experience: Metrics such as developer satisfaction surveys, time-to-onboard new developers, and the number of self-service platform capabilities can provide insights into the effectiveness of the platform in supporting developer productivity and autonomy.
Reliability and Scalability: Monitoring service-level objectives (SLOs), incident response times, and the ability to handle increased traffic or user demands can help assess the platform’s reliability and scalability.
Security and Compliance: Tracking the number of security incidents, the ratio of security issues found during threat modeling versus post-deployment, and the adoption of security best practices can indicate the platform’s security posture.
Automation and Tooling: Metrics like the percentage of infrastructure provisioned through code, the frequency of platform updates, and the time saved through automation can demonstrate the platform’s operational efficiency.
Observability and Monitoring: Measuring the effectiveness of observability tools, the time to detect and resolve issues, and the quality of incident reports can provide insights into the platform’s overall health and performance.

As Jess Mink emphasizes, it’s important not just to collect these metrics but to use them to drive meaningful action and improvement:

“We look at all of those [metric categories] every quarter and write summaries that go up to the executive level. This creates visibility and a shared understanding of problems so that there’s room for movement and change in the right ways.”

The Future of Secure Platform Engineering

Software development and infrastructure management are evolving, and the role of platform engineering will only become more critical to support it. By integrating DevOps and security practices, platform engineering teams can create scalable, secure platforms that empower developers to be more productive and innovate, delivering business value.

Learn how Pulumi customers build secure, scalable platforms and empower their development teams:

Tivity Health: DevSecOps Game-Changer: Security Automation That Delivers Business Results
BMW Group: Unified and Programmatic Approach to Infrastructure Management at BMW Using Pulumi
Lemonade: How the DevOps team supports a much larger group of developers using by Pulumi to standardize infrastructure components and enforce best practices.

Discover platform engineering best practices in The Guide to Platform Engineering: 7 Steps to Get It Right.

Build secure, scalable platforms with confidence—get started with the Platform Engineering Workshop Series & Course

PulumiUP 2024: Dive Into the Future of Cloud, Platform Engineering, and AI/ML

Sara Huddleston — Wed, 11 Sep 2024 07:04:40 +0000

PulumiUP 2024 is just around the corner! It will be held on September 18th, starting at 8 AM PT | 15:00 UTC +0, and with over 5,500 engineers from all over the world already registered, this is shaping up to be the must-attend event for cloud professionals, platform engineers, and AI/ML enthusiasts alike. From entry-level engineers to tech executives, this event brings together professionals from companies of all sizes to explore the latest innovations and best practices in Cloud and IaC, Platform Engineering & DevOps, and AI/ML.

If you haven’t registered yet, now’s the time! Start building your schedule today, select the talks you want to watch live and on-demand and add them to your schedule.

PulumiUP at Glance

Main Track: Keynote, Cloud Culture Talks, and More

In the Main Track, you’ll hear directly from the Pulumi leadership team about exciting product innovations. Don’t miss the Cloud Culture talks focused on the human side of technology and collaboration.

Keynote and New Product Announcements - Joe Duffy (Co-founder and CEO), Luke Hoban (CTO), and other key Pulumi leaders will unveil new products and innovations shaping the next wave of cloud development. Expect to see live demos showcasing how these tools can revolutionize your workflows, from cloud infrastructure to security automation.
"Why Care About Building a Cloud Center of Excellence?" - Alex Radu (VP Product & Marketing Manager, Public Cloud, J.P. Morgan) discusses why building a Cloud Center of Excellence is essential for long-term cloud strategy success.
"What Is ‘HugOps’ and Why Is It Important?" - Rees Pozzi (Senior Platform Engineer, Kainos) explores the meaning and importance of “HugOps,” a practice promoting empathy and collaboration in the tech world but also by all those impacted by it.

AI/ML Track Highlights

Panel: AI for Cloud Development

AI is transforming how we approach development and infrastructure, and PulumiUP offers an incredible lineup of talks and panels on the topic.

"Data Privacy Challenges with Large Language Models" - Aditi Godbole (Data Science, AI & ML Leader, SAP) will address the critical question of managing data privacy, addressing LLM basics, AI data privacy fundamentals, and specific privacy issues in LLMs.
"Pulumi Powered AI/ML on Kubernetes" - Jason Smith (App Modernization Specialist, Google Cloud) will show you how to serve an open-source LLM with a RAG on Kubernetes and set up a Kubernetes environment for AI/ML workloads with Pulumi.
"AI Tools for Developers" - Jim Clark (Principal Software Engineer, Docker) will explore how Tools can enhance the capabilities of AI assistants and show how the combination of AI, Tools, and containerized runtimes, can enhance interactions between developers and their tools.
"The AI Governance Challenge" - Patty O’Callaghan (Technical Director, Charles River) will explore the complexities of AI governance and how to navigate them.
Panel Discussion: “AI for Cloud Development” - Luke Hoban (CTO, Pulumi), Phillip Carter (Principal Product Manager, Honeycomb), Giri Sreenivas (CPO, Docker), Clare Liguori (Senior Principal Software Engineer, AWS) and Meagan Cojocar (Principal Product Manager, Pulumi) will provide diverse perspectives on how AI is shaping cloud infrastructure.

Platform Engineering and DevOps Track Highlights

Panel: Secrets and Policies—Automating Cybersecurity

The Platform Engineering & DevOps track at PulumiUP is perfect for anyone looking to automate infrastructure, secure their systems, and scale their environments with ease. This track features talks and panels on cutting-edge technologies and practices.

"Moving Mountains: How IaC Unlocks Massive Refactoring Possibilities" - Shaun Verch (Infrastructure Engineer, Oso) will show you how infrastructure as code can transform large-scale refactoring projects.
"Using Pulumi to Empower Kubernetes Fleet Management" - Blake Romano (Senior Software Engineer, Imagine Learning) will show you how Imagine Learning leverages Pulumi to manage a fleet of Kubernetes clusters and other platform resources quickly and reliably, and how it revolutionized their workflow.
"Streamline Your Infrastructure Deployment with GitOps and Pulumi Operator" - Sam Cogan (Solutions Architect & Azure MVP, WTW) will highlight how you can define your infrastructure in Git, and have Pulumi automatically reconcile it into deployed infrastructure.
"Is This a Platform? Platform Engineering Before PMF" - Jk Jensen (Team Lead, Staff Software Engineer, MystenLabs) will share how they built their service platform around Pulumi to help engineers and researchers move from prototypes to deployed environments, enabling their teams to iterate more quickly and develop an increasing number of systems without additional complexity.
"Security Automation—Faster. Cheaper. Better." - David Giambruno (VP of Engineering and DevOps, Tivity Health) and Joe Duffy (Founder/CEO, Pulumi) will discuss security automation and how these efforts impacted speed and agility, cost optimization and improved security and compliance.
"Meet Devs Where They Are: Why IaC Must Be Real Code" - Jeremy Adams (Head of Ecosystems, Dagger) will present why real code (that teams already use for applications) is essential for both IaC and CI/CD pipelines, and to meet modern DevOps best practices.
"Security of IaC Pipelines and Infrastructure Governance With Policies-As-Code" - Marina Novikova (Senior Partner Solutions Architect, AWS) and Andy Taylor (Senior Networking Specialist/ Solutions Architect, AWS) will explore security best practices for IaC pipelines and infrastructure governance using policies-as-code, cover best practices for handling sensitive information like access keys, tokens and encryption keys, and scaling and automating your DevSecOps.
Panel Discussion: “Secrets and Policies—Automating Cybersecurity” - Arun Loganathan (Senior Product Manager, Pulumi), Maya Kaczorowski (PM of Security), Jason Meller (VP, Product, 1Password), and Ofir Cohen (CTO, Container Security, Wiz) will explore the intersection of innovation and security, offering insights into how AI is reshaping the threat landscape and how organizations can stay ahead.

Cloud and IaC Track Highlights

Panel: Infrastructure as Code - Can we do better?

The Cloud and IaC track is ideal for those wanting to deepen their understanding of infrastructure as code and its impact on modern cloud computing. Learn from industry pioneers and see where the future of IaC is headed.

"From CDK to Pulumi: Evolution of SST" - Dax Raad (Founding Engineer, SST) and Jay V (Founder, SST) will share insights into the early development of their product, initially built on AWS Cloud Development Kit (CDK), the challenges they encountered and the groundbreaking decision to transition to Pulumi.
"Unified Software Development at BMW Software Factory with Pulumi" - Jan-Peter Alten (Expert Software Engineer, DevOps, BMW Group) will showcase how BMW uses Pulumi to unify their software development processes, resulting in an increase in developer productivity, streamlined infrastructure management, and enhanced scalability, security, and compliance across their complex software ecosystem.
"Stacking Accounts: Multi-Account Deployments in a Single Stack for Monitoring, Management, and More" - Denis Willett (Software Engineer, North Carolina Institute for Climate Studies) will explain how they use Pulumi for managing 60+PB of environmental data across AWS, GCP, and Azure and leverage multi-account deployments in a single stack via the Pulumi Automation API.
"Industrialize the Configuration of Your GitHub Repositories With IaC" - Alexandre Nédélec (Software Engineer, Avanade) will live code using the GitHub, Azure native, and Azure AD Pulumi providers to create a GitHub repository and configure its pipeline to deploy to Azure using OpenID Connect.
Panel Discussion: “Infrastructure as Code — Can We Do Better?” - Joe Duffy (Co-Founder and CEO, Pulumi), Brian Grant (CTO, Stealth), Elad Ben-Israel (Co-Founder and CEO, Winglang), Adam Jacob (Co-Founder and CEO, System Initiative), and Luke Hoban (CTO, Pulumi) will discuss the past, the present, and the future of Infrastructure as Code and the overall infrastructure technologies.

Workshops and Hands-On Learning

Want to get hands-on experience? Our live workshops are designed to deepen your understanding of trending topics, Pulumi, and its applications.

"Managing Team Secrets with 1Password & Pulumi ESC" on September 25, led by Phil Johnston (1Password) and Diana Esteves (Pulumi).
"Securing IaC Pipelines in Regulated Industries" on September 26, led by Josh Kodroff (Pulumi) and Marina Novikova (AWS).
"Deploy Tailscale infrastructure in AWS with Pulumi" on October 15, led by Diana Esteves (Pulumi) and Lee Briggs (Tailscale).
"Observability as Code for AI Apps with New Relic and Pulumi" on October 30, led by Diana Esteves (Pulumi) and Harry Kimpel (New Relic).

Amazing Partners and Sponsors

Thank you to our amazing customers, sponsors, and partners for supporting PulumiUP!

AWS The world’s most comprehensive and broadly adopted cloud, offering over 200 fully featured services from data centers globally.
Google Cloud Help developers build quickly, securely, and cost effectively with the next generation of modern infrastructure designed to meet specific workload and industry needs.
New Relic Data for engineers to monitor, debug, and improve their entire stack.
Docker Accelerate how you build, share, and run applications.
1Password Streamline how you manage SSH keys, API tokens, and other infrastructure secrets throughout the entire software development life cycle.
Tailscale Make creating software-defined networks easy: securely connecting users, services, and devices.
Pinecone Build remarkable GenAI applications fast, with lower cost, better performance, and greater ease of use at any scale.
Wiz Secure everything you build and run in the cloud.
Honeycomb Observability that helps solve problems you couldn’t before.
Mysten Labs Building critical infrastructure to enable a more decentralized internet.
Imagine Learning Empowering educators to inspire breakthrough moments in every student’s unique learning journey with digital-first, K–12 education solutions.
Dagger Powerful, programmable open source CI/CD engine that runs your pipelines in containers.
Oso Authorization as a service.

Build Your Schedule Now

With so many compelling topics across AI/ML, Platform Engineering, DevOps, and Cloud and IaC, now is the time to build your PulumiUP schedule. Select the sessions that match your goals and interests to maximize your experience. Plus, you’ll receive a link to watch your chosen panel discussions and tech talks on-demand!

Haven’t registered yet? It’s not too late! Join 5,500 engineers worldwide in this groundbreaking event and gain the insights you need to drive innovation at your company. Register now->

We can’t wait to see you there!

Platform Engineering: Cloud-Native, Maturity Models, and Platforms as Products

Sara Huddleston — Fri, 26 Jul 2024 17:40:17 +0000

The Platform Engineering & DevOps in-person series launched in Berlin with two great speakers. This blog article is an overview of Dominik Kress’s talk, “What the Heck is the CNCF Platform Working Group? Answers from a Member!” in which he discussed Cloud-Native Platforms, The Platform Maturity Model, and approaching Platforms as Products.

In this article, you’ll learn more about platform engineering and how to get involved with the CNCF Platform Working Group. Make sure to check our Pulumi User Groups (PUGs) to find a meetup near you.

On this platform engineering article:

What is a Platform?

According to CNCF, a platform in cloud-native computing is an integrated collection of capabilities designed to meet users’ needs. It serves as a cross-cutting layer, ensuring a consistent experience integrating and managing services and capabilities for various applications and use cases. A good platform provides consistent user experiences through portals, project templates, and self-service APIs.

Atlassian describes platform teams as creators of capabilities used by multiple product teams, reducing their overhead and cognitive load. Martin Fowler and Evan Bottcher define a digital platform as a foundation of self-service APIs, tools, services, and support arranged as an internal product. These platforms enable autonomous teams to deliver features faster with less coordination.

The specific capabilities of a platform are tailored to the needs of its stakeholders and users. While platform teams provide these capabilities, they do not always implement them directly. Managed service providers or dedicated internal teams can maintain the underlying implementations, ensuring consistency across the organization. Platforms are tailored to an enterprise’s internal users and are particularly relevant for cloud-native architectures, separating supporting capabilities from application-specific logic.

Defining Cloud Native Platforms

The core of the discussion at the CNCF Platform Working Group is: “What exactly is a cloud-native platform, and why do organizations need one?” A cloud-native platform is a strategic business initiative that enables organizations to deliver value more efficiently and effectively. It is characterized by:

Vendor-neutrality: A cloud-native platform should be open and agnostic, allowing organizations to leverage various technologies and services without being locked into a single vendor. Scalability and flexibility: The platform should be designed to accommodate the organization’s growing and changing needs, with the ability to add or remove components as needed easily.
Developer-centricity: The platform should empower developers to be more productive by providing self-service capabilities, streamlined workflows, and a consistent developer experience.
Operational efficiency: The platform should simplify and automate the management and maintenance of the underlying infrastructure, allowing the organization to focus on delivering value to customers.
Continuous improvement: The platform should be designed with a mindset of continuous evolution, with regular updates and enhancements to address emerging needs and technologies.

By aligning with these principles, organizations can achieve faster time-to-market, improved developer productivity, and reduced operational overhead.

The Platform Maturity Model: Charting the Path to Success

Dominik Kress at the Berlin Pulumi Group meetup explaining the platform maturity model

The Platform Working Group has developed a Platform Maturity Model. This framework can help assess the current state of a cloud native platform and identify steps to enhance its maturity. It defines five levels:

Ad-hoc: At the lowest level, the platform is ad-hoc, with no dedicated team or budget and a reliance on individual efforts and workarounds.
Operational: The platform has a dedicated team and budget, which focus primarily on operational tasks and maintenance rather than strategic initiatives.
Scalable: The platform is now viewed as a product with a clear roadmap and a focus on delivering value to internal customers. The platform team has more autonomy and the ability to drive innovation.
Enabled: The platform has become a strategic asset, with a thriving ecosystem of contributors and consumers. The platform team is empowered to make decisions and drive the platform’s evolution.
Optimized: The platform is continuously optimized, focusing on measuring and improving key performance indicators. The platform is deeply integrated into the organization’s technology strategy and decision-making processes.

By using the CNCF’s Platform Maturity Model as a guide, organizations can assess their current level of platform maturity, identify areas for improvement, and develop a roadmap to reach a higher level of platform success. This model has proven to be a valuable tool for many organizations, helping them to align their platform initiatives with their broader business objectives.

Platforms as Products: Driving Success

One of the key initiatives currently underway within the Platform Working Group is the “Platforms as Products” research project. This effort aims to explore the parallels between traditional product management and the management of cloud-native platforms to identify best practices and patterns that can help organizations unlock their platforms’ full potential.

The core premise of the “Platforms as Products” initiative is that successful cloud-native platforms should be treated and managed like products rather than just technology projects. This means applying principles of product management, such as:

Defining a clear vision and roadmap: Establishing a long-term vision for the platform and a roadmap to get there, focusing on delivering value to internal customers.
Measuring and optimizing for key metrics: Identifying and tracking the right performance indicators to ensure the platform delivers the desired outcomes.
Fostering a thriving ecosystem: Encouraging and enabling a community of contributors and consumers to drive the platform’s evolution and adoption.
Continuously iterating and improving: Embracing a continuous improvement mindset, with regular updates and enhancements to the platform based on user feedback and emerging needs.

By adopting a product management approach to their cloud native platforms, organizations can better align their platform initiatives with their broader business goals and ensure that the platform delivers tangible value to the organization.

Conclusion

This article and Dominik Kress’s talk highlighted the importance of platform engineering in cloud-native computing. Key points included defining cloud-native platforms, assessing platform maturity, and managing platforms with product management principles.

If you’re interested in learning more about platform engineering and getting involved in CNCF’s initiatives, there are several ways to participate:

Register for the Platform Engineering workshop series course.
Join the bi-weekly CNCF Platform Working Group meetings every other Tuesday at 5 PM Berlin time (11 AM ET).
Learn how to build Developer Portals with Pulumi. You can get off the ground faster with organization templates, a new project wizard, or leveraging the backstage plugin.

Frequently Asked Questions

What is the CNCF Platform Working Group?

Within the CNCF, the Platform Working Group is a collaborative effort focused on the critical topic of cloud native platforms. As a working group member, you can share insights and experiences from our discussions.

The Platform Working Group is part of the CNCF’s App Delivery Technical Advisory Group, which includes sustainability, runtime, observability, and security initiatives. The working group is open to anyone interested in contributing to the conversation around cloud native platforms, and its members come from diverse backgrounds, including product managers, engineers, and industry experts.

Who is Dominik Kress?

Dominik Kress is a product manager at Giant Swarm and a CNCF Platforms Working Group member.

He is on a mission to simplify developers’ lives by delivering intuitive developer platforms. He has been in the IT industry for over seven years, starting his journey as a Full Stack Software Engineer, falling in love with DevOps and Product Organisations to become a Product Manager for Cloud Technology later.

Dominik is passionate about Cloud Transformation, Product Management, and helping people. He loves contributing to the community with talks, articles, or publications.

How can I participate in the Platform As A Product research?

The “Platforms as Products” research project is currently underway, with the Platform Working Group conducting interviews and gathering insights from organizations that have successfully implemented cloud native platforms. The goal is to distill these learnings into a set of best practices and patterns that can be shared with the broader community.

To participate in this research, sign up here.

Platform Engineering & DevOps Series Kickoff Announcement

Sara Huddleston — Mon, 15 Jul 2024 14:38:46 +0000

We are excited to announce the kickoff of the Platform Engineering & DevOps Series, which will run from July 16 to October 31. This series will feature in-person events across various cities, including Berlin, London, Paris, Sydney, Boston, San Francisco, Seattle, Austin, Denver, and NYC. For those unable to attend an in-person event, we encourage you to participate in a virtual DevOps & Platform Engineering workshop.

Upcoming In-Person Events

Pulumi is hosting in-person meetups and virtual workshops for engineers interested in Platform Engineering and DevOps, covering topics from best practices to setup and implementation to advanced topics. Join us for tech talks, hands-on workshops, and opportunities to network with fellow engineers.

First Platform Engineering & DevOps in-person event: Berlin - July 16

Join Dominik Kress (Product Manager at Giant Swarm and member of the CNCF Platforms Working Group) and Guy Menahem (Solutions Architect at Komodor and community leader at The Platformers) at LOBE Block for an engaging discussion on platform engineering.

Dominik Kress will present “What the Heck is the CNCF Platform Working Group? Answers from a Member!”
Guy Menahem will present “For This Backstage, You Don’t Need a Special Pass: Platform Engineering in Practice.”

For more details and to RSVP, visit the Berlin Meetup page.

Second In-Person Event: Boston, July 24

Join Pulumi’s founder & CEO, Joe Duffy, and Jamie Baldanza, Director of DevOps at Relay Therapeutics, for the launch of our Boston meetup at The Foundry. This is a great opportunity for DevOps engineers, platform engineers, software engineers, and Cloud enthusiasts to connect and share insights.

RSVP is required due to limited capacity. Secure your spot, and RSVP at the Boston Meetup page.

More coming soon!

We expected Seattle, Paris, and London’s meetups to happen in early September. Details for the other in-person events will be coming soon, so stay tuned for more updates at Platform Engineering & DevOps Series.

No upcoming in-person events in your area? Attend one of our virtual workshops and keep up with the latest in DevOps & Platform Engineering!

Virtual Workshops

Getting Started with CI/CD for AWS using GitHub Actions: July 17. Learn more and register here.
From Zero to Production in Kubernetes: July 25. Learn more and register here.
Advanced CI/CD for AWS using Pulumi and GitHub Actions: August 14. Learn more and register here.
Mastering Platform Engineering: From Setup to Scaling Success: TBD.

Join us for tech talks, hands-on workshops, and opportunities to network with industry peers. We are excited to meet you in person or virtually. See you there!

Announcing PulumiUP Conference 2024

Sara Huddleston — Wed, 01 May 2024 17:27:25 +0000

Today we announce our annual PulumiUP virtual conference taking place on September 18, 2024. PulumiUP is our global cloud and IaC conference for anyone interested in infrastructure as code, cloud, AI, platform engineering, DevOps, and cloud culture. Every year, we strive to build a program packed with technical talks, demos, and best practices to empower you with new learnings and inspiration that you can apply in your professional life. Attend to hear from industry leaders, experts, and community peers.

What is PulumiUP?

PulumiUP isn’t just another conference, it’s a global gathering of minds passionate about cloud technologies, AI, DevOps, and everything in between. Whether you’re an industry veteran or just dipping your toes into the vast ocean of cloud computing, PulumiUP has something for everyone.

You’ll hear keynotes from Pulumi Founder/CEO Joe Duffy and CTO Luke Hoban, stories from industry leaders on their experiences with cloud adoption and transformation (and how they used Pulumi to make it happen), and talks from other members of the Pulumi community (like you!) aimed at helping you navigate the ever-changing world of the cloud.

Call for Speakers: Your Story Matters

One exciting addition to this year’s event is the call for speakers. We invite you, the community, to share your stories, experiences, and expertise with fellow attendees. Whether it’s a deep dive into a groundbreaking project, lessons learned from failures, or innovative best practices, your voice matters at PulumiUP.

Key Dates to Remember

CFP opens: Thursday, April 25, 2024.
CFP closes: Monday, June 17, 2024.
CFP Notifications: Monday, July 22, 2024.

Selection Criteria

At PulumiUP, diversity and originality are celebrated. We welcome proposals from all corners of the tech community and prioritize fresh perspectives and solutions. Regardless of where you are in the world, we want to hear from you. Whether you’re a seasoned speaker or a first-timer, we encourage you to submit your ideas. Just remember, no vendor pitches allowed!

Talk Formats and Themes

You can submit up to two proposals in either the Lightning Talk (15-minute) or traditional session (30-minute) formats. The conference will cover a wide array of themes, including:

Infrastructure as Code: From cloud providers to Kubernetes, explore the latest trends and best practices in cloud computing and IaC.
Platform Engineering & DevOps: Learn how to streamline CI/CD practices, automate infrastructure operations, build internal developer portals (IDP), and improve developer experiences.
AI / ML: Dive into the world of AI and ML with topics like deploying AI application, LLM models, MLOps, and AIOps.
Cloud Culture: Discover the human side of tech, focusing on diversity, inclusivity, and collaboration within the industry.

Do you have a story to tell? Submit your proposal.

Join Us on September 18!

Register for PulumiUP. Stay tuned for more updates and announcements as we gear up for PulumiUP 2024. We can’t wait to see you there!

How AI is Transforming DevOps: AI Talks for DevOps Insights

Sara Huddleston — Mon, 13 Nov 2023 22:05:34 +0000

The integration of artificial intelligence (AI) with DevOps signals a new era in software development. DevOps possesses unique characteristics and needs that make it exceptionally compatible with AI augmentation. Given that code fundamentally relies on language, and large language models (LLMs) serve as the core of GPT functionality, these models are particularly well-suited for tasks such as code generation. This article unwraps the topics addressed during our “AI: Friends or Foe | AI Talks for DevOps” event in San Francisco.

How will AI Impact the Future of DevOps

In the Fireside chat, Luke Hoban expressed his enthusiasm for incorporating AI into DevOps, emphasizing its potential utilization by both DevOps professionals and the development team. In contrast, Nana Janashia, TechWorld with Nana, adopted a more cautious stance, drawing attention to potential risks, particularly regarding reliability.

What is DevOps in AI

When considering the adoption of AI and AI engineering, such as prompt engineering and other specific disciplines, AI engineers rely heavily on the cloud. Many AI-based systems are built around cloud-native tools, practices, and managed services.

The extensive use of cloud-native primitives means that AI is fundamentally a cloud adoption story. Consequently, established DevOps and infrastructure as code tooling and best practices play an important role in AI engineering, in what is being done, and in leveraging its use.

What is AI in DevOps

Integrating AI in DevOps entails employing machine learning (ML) and other artificial intelligence technologies to automate and optimize various software development and delivery aspects, including tasks such as automating testing and deployment processes and refining resource management.

The incorporation of AI in DevOps can result in:

Enhanced speed
Improved resource management
Increased reliability throughout the software development lifecycle

Consequently, organizations may experience swifter deployments, reduced errors, and heightened overall productivity. By substituting manual processes with automation and AI-powered solutions, DevOps teams can elevate product quality and more efficiently oversee their systems.

How AI is Currently Used in DevOps

Luke Hoban at the San Francisco Pulumi Group meetup discussing the usage of AI, during the AI Talks for DevOps

DevOps significantly impacts all stages of the development and operations lifecycle. It has helped break silos and optimize to increase developer velocity and productivity, creating a positive impact on the ops side.

AI assists DevOps teams to refine their operations by detecting inefficiencies and triggering warnings as soon as issues appear. This technology also enables real-time observation of systems and applications, permitting prompt actions from operations personnel and minimizing downtime.

AI-driven automation can also help the software development process with code generation. The operations teams from both departments quickly pinpoint any issues in their workflows while speeding up decision-making processes.

In the long term, the AI may use cloud credentials to create and autonomously automate deployment and management and perform DevOps tasks on behalf of the DevOps or platform engineering team. In the near term, it is used to assist humans, from DevOps professionals to developers.

“In the near term, AI usage in DevOps is about how do you keep the human in the loop, how do you use it as a learning tool, as a code generation tool, and for auditing to provide actionable notifications, all of which are still very human-dependent. There is so much more it can be used for. Currently, it’s a tool serving humans. But it is fascinating to imagine it in the future when it can be trusted to be an autonomous agent.” - Luke Hoban, CTO of Pulumi and co-creator of TypeScript

So, how does this technology bring about these advancements? Two main ways are by streamlining development tasks and improving monitoring and security protocols.

Streamlining Development Workflows

AI has the potential to significantly change the software development life cycle by automating routine processes, using resources efficiently, and offering solutions for improved code. DevOps teams can leverage AI-driven tools to increase effectiveness when managing their assets while also improving code quality, leading to faster project deliveries.

For instance, AI can identify any slowdowns during progress and recommend improvements, analyze root causes, and help make judicious decisions regarding resource allocation, ultimately resulting in efficient delivery of software products.

Enhancing Monitoring and Security

The usage of AI significantly boosts DevOps monitoring and security. With the assistance of AI tools, operations teams can use machine learning algorithms to examine a massive amount of data to protect their applications from potential risks and enhance reliability. These algorithmic solutions provide valuable advice for development personnel, which helps strengthen protection measures while optimizing performance simultaneously.

How Can a DevOps Team Take Advantage of Artificial Intelligence

A DevOps team can leverage AI in several ways to enhance efficiency, automation, and overall productivity. Here are some key strategies for a DevOps team to leverage AI:

Code Generation and Learning

AI can be a powerful tool to make the development process much smoother. It can help engineers focus on what matters instead of searching for documentation on websites, developer forums, or search engines for answers to help solve specific problems. The perfect code snippet is often hard to find or may not even exist.

Generative AI, powered by LLMs, can help developers and DevOps engineers quickly discover, learn, and, for example, use new cloud infrastructure APIs.

Pulumi AI response to Run metabase in Azure, using TypeScript

You can see it in practice using Pulumi AI. You can use natural language to ask for a specific cloud infrastructure goal and obtain a corresponding Pulumi program in your preferred programming language. Moreover, Pulumi AI facilitates the iterative refinement of your cloud infrastructure, incorporating new features, enhancements to security and performance, resolution of correctness issues, and clarification of requirements. Try Pulumi Neo.

Intelligent Resource Management

Platform and developer teams are spinning up resources across many cloud accounts within multiple cloud providers. It can be almost impossible to aggregate and search your infrastructure across these accounts since every cloud and every account is (by design!) a walled garden.

Often, to answer key questions about their business, such as “Which team has the largest cloud footprint?”, teams need to combine data across various clouds, cloud accounts, and SaaS tools. This prevents organizations from getting quick answers to operational questions and barriers to building aggregations to derive insights. AI can provide actionable insights over infrastructure with natural language processing (NLP) queries.

“I can get developers using IaC immediately with Pulumi Deployments and its GitHub integration, while Pulumi Insights [Resource Search] makes it really easy to find idle developer environments that need to be shut down, which reduces our cloud costs” - Santosh Dornal, Head of Software Test & DevOps, Alkira.

When using Pulumi Cloud, you can leverage AI Assist. You can provide a plain English (Spanish, French, or Japanese) request to express queries where you might not know the exact syntax, type tokens, or package names. AI Assist makes it easier by providing answers to queries like:

show me all s3 buckets not tagged in production
show azure and azure native security groups
show all AWS VPCs

Remediation Tool: Continuous and Automatic Compliance

Defining and managing security rules, criteria, and conditions are key to enabling security at scale. Policy as Code, or Compliance as Code, has the ability to verify and spot problems before deploying the infrastructure. When policies are written with code, you can apply software development practices such as testing, automated deployment, and version control.

AI can be valuable not just in anomaly detection but also in remediation, which doesn’t just warn or error but can change the infrastructure directly to remediate compliance gaps. With Pulumi, Remediation policies can be configured and applied across any subset of stacks within an organization.

Consider the unfortunate incident at GitLab, marked by a disastrous sequence of events resulting in the loss of segments of production databases and substantial user data.

In this scenario, AI could have swiftly pinpointed the root cause and proposed a rapid solution, potentially mitigating the extent of damage before it escalated.

While the benefits of writing policies as code are evident for DevOps engineers, the organizational benefits are even more significant.

Enhance Collaboration and Communication

Nowadays, many professionals with DevOps in their titles are held accountable for the multiple aspects of Dev + Ops. At its core, DevOps is a cultural movement that removes barriers among teams and promotes collaboration and shared responsibility between dev and ops.

DevOps teams have seen great benefits from AI-driven tools, particularly in improving transparency, encouraging feedback, providing quick answers to common queries, and automating routine tasks. Many major companies, like Microsoft, use AI to build efficient workflows that make better decisions, leading to increased business outcomes. This includes smoother application delivery, a prime function of DevOps initiatives.

Improved Product Delivery

Companies incorporating AI into their DevOps strategy reap the rewards of accelerated product delivery. This has been made possible through improved workflows, automated testing, and strengthened security measures, leading to greater efficiency in software development by allowing development and operations teams more scope for dealing with complex tasks using machine learning models. They can make better decisions, resulting in faster product delivery and enhanced business outcomes.

By streamlining existing systems within this context, customers can benefit from superior products delivered faster than before while companies provide and reinforce their business value.

Planning and Cost Optimization

Implementing AI algorithms or AI tools that analyze usage patterns can assist in predicting future resource needs and optimize cloud infrastructure costs. This also ties in with intelligent resource management mentioned before, which can enhance cost efficiency, optimize resource allocation, and prevent unnecessary expenses.

Top 5 Skills for DevOps Engineers in the AI Era

Adora Nwodo with Nana discussing skills that DevOps engineers will need to learn to make AI work

DevOps engineers must open new skills, mainly in cloud services, data, and AI/ML.

Programming Languages: Although DevOps don’t usually need to know general programming languages in-depth, they should know what is most used by application teams. It’s a nice-to-have to find errors in CI/CD pipelines and fix them themselves as needed, but it is a must-have if the goal is to work MLOps.
Infrastructure as Code: Automation is important for DevOps and critical for MLOps. IaC, using familiar languages, has been used since the early days of MLOps and is a key component to solving the cloud orchestration challenge present in AI/ML. Dive into more detail in The Real AI Challenge is Cloud, not Code!
Continuous Integration and Delivery: CI/CD is mandatory knowledge, and it must be understood in detail to predict data preparation code, AI/ML code, CI/CD code, and infrastructure-related code.
Custom containers: As a DevOps professional, you are expected to understand Docker and/or Kubernetes concepts and best practices.
Orchestration: This is the holy grail of Ops and your key to unlocking AI.

How to Overcome AI for DevOps Challenges

The integration of AI in DevOps brings forth several challenges that organizations must navigate to leverage the full potential of these technologies. Here are some key challenges and potential strategies to overcome them:

1. Skills and Expertise

Challenge: AI & ML expertise is often scarce within DevOps teams, leading to a skill gap in implementing and managing AI-driven tools.

Overcoming Strategy: Organizations will need to hire professionals with AI and ML expertise, or existing team members will need training in machine learning, becoming proficient in MLOps combined to DevOps principles. Collaboration between DevOps and data science teams can also bridge the knowledge gap.

2. Integration Complexity

Challenge: Integrating AI seamlessly into existing DevOps workflows can be complex, given the diverse technologies and processes involved.

Overcoming Strategy: Prioritize a phased integration approach, gradually introducing AI capabilities into specific stages of the DevOps pipeline. This allows teams to adapt without disrupting established processes.

3. Scalability

Challenge: As AI applications within DevOps expand, ensuring scalability becomes crucial to accommodate growing workloads.

Overcoming Strategy: Design AI solutions with scalability in mind, utilizing cloud resources and containerization. Regularly assess performance and adjust infrastructure to meet evolving demands.

4. Data Quality and Availability

Challenge: AI models rely heavily on high-quality and readily available data. Inconsistent or inadequate data can lead to inaccurate predictions or suboptimal performance.

Overcoming Strategy: Prioritize data quality and establish robust data pipelines. Conduct regular audits to ensure data accuracy and implement strategies for handling missing or incomplete data.

5. Cultural Resistance

Challenge: Resistance to change or skepticism about the benefits of AI within DevOps and in software development can impede successful integration.

Overcoming Strategy: Foster a culture of innovation and continuous improvement. Provide clear communication about the benefits of AI, involve team members in the decision-making process, and showcase success stories to build confidence.

Navigating these challenges requires a thoughtful and strategic approach, focusing on collaboration, skill development, and the gradual incorporation of AI into existing DevOps practices. Regular evaluation and adaptation are essential to ensure a successful and sustainable integration of AI in the DevOps environment.

Conclusion

AI in DevOps can transform software development by making workflows more efficient, monitoring and security stronger, and overall effectiveness increased. Organizations that embrace AI/ML aim for better collaboration between teams, automating and intelligently managing infrastructure, reducing time-to-market, and increasing developer productivity and cost savings due to streamlining processes within their development lifecycle. AI and ML knowledge will become necessary skills expected to be applied to enhance and benefit the development, security, and operation teams as a whole. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Can AI replace DevOps?

DevOps has not been taken over by AI yet. Rather, its use has made a difference in code quality and accelerated certain processes. Improvements to the quality of coding have been seen as one of the main benefits brought on through integration with artificial intelligence.

What is an example of AI in DevOps?

AI can be utilized to make DevOps processes more efficient by creating test cases and pipelines, assessing infrastructure configurations for security concerns, and deploying applications. This technology includes machine learning algorithms, natural language processing systems, computer vision capabilities, chatbot programs, and virtual assistants. All are designed with one goal in mind – optimization of operations within development teams.

What is the future of DevOps?

Operations teams collaborating with developers to create quality products quickly and effectively is the goal for DevOps moving forward. The vision of this process involves both sides working together, resulting in an improved system and increased efficiency.

LangChain for DevOps: Learn LLM & GenAI for Dev, Sec & Ops

Sara Huddleston — Thu, 26 Oct 2023 00:00:00 +0000

The emergence of DevOps revolutionized software development. Now, with AI powered tools like LangChain, these transformations are being accelerated. Unsurprisingly, our distinguished speaker at the launch of Pulumi’s in-person AI Talks, Patrick Debois, who coined the term “DevOps,” has recently tuned into LLM and GenAI Ops using the Langchain framework.

This article unwraps the topics addressed during our “Generative AI, Apps, and DevOps | AI/ML Talks” event in Seattle, focusing on Langchain and GenAI for DevOps. You’ll be able to watch Patrick Debois teach LangChain to learn LLMs and GenAI, with code examples and lessons that are easily understood by traditional software engineering.

What is LangChain used for?

LangChain is an open-source Python framework designed to empower developers to construct robust generative AI applications. It facilitates the integration of advanced AI models such as OpenAI’s GPT, Google’s Gemini, and Meta’s LLaMA. This versatile tool finds application in developing retrieval-augmented chatbots and other personalized assistants, utilizing technologies like ChatGPT. Furthermore, it enables tasks such as question-answering (GQA), summarization, code comprehension, API interactions, data cleansing, and much more.

Why is LangChain becoming popular?

The landscape of AI development is rapidly evolving, and LangChain is packed with incredible features for building LLM applications and tools. Developers with a wide range of expertise use LangChain for tasks such as managing interactions with language models, establishing seamless connections between diverse components, and integrating external resources like APIs and databases effortlessly.

The LangChain platform hosts a diverse collection of APIs seamlessly integrated into applications. This enables developers to incorporate advanced language processing functionalities without the need for laborious construction from scratch.

LangChain Accelerating DevOps

Patrick Debois presenting the workshop Dev, Sec & Ops meet LangChain

LangChain can streamline the integration of large language models and data sources. By helping bring in AI capabilities, DevOps can revolutionize software development life cycles by making it easier for teams to handle challenges while optimizing resource allocation and usage through their cutting-edge, AI-powered solutions. This makes the automation of manual processes easier and the development of new resources far more accessible with faster utilization rates by leveraging artificial intelligence. Significantly important as “Developer Velocity and Productivity” become business performance metrics across various industries.

The features included in LangChain include chain interface memory functions for scalability within apps, control over dataflow, as storage – making code quality more reliable, plus seamless machine learning models integration, which allows them to build custom automated AI programs quickly. Thus providing users with an advanced but easy way of creating sophisticated solutions by combining their knowledge of coding and machine training models together.

How to Use LangChain to Learn LLMs and GenAI for Dev(Sec)Ops

Patrick teaches in the way he would like to be taught. The workshop mainly uses the Langchain framework and basic Python knowledge. OpenAI will also need to be installed. You’ll learn essential skills in multiple use cases, applying Langchain for LLM application Development.

You can see the breakdown of the lessons per role below, and find the code examples in the Learning LLMs and GenAI for DevSecOps repo.

Developer

Calling a simple LLM using OpenAI
Looking at debugging in Langchain
Chatting with OpenAI as a model
Using prompt templates
Use Docloader to read your local files and prepare them for the LLM
Explain the calculation and use of embeddings
Understand how splitting and chunking are important
Loading embeddings and documents in a vector database
Use a chain for Questions and Answers to implement the RAG pattern (Retrieval Augmented Generation)
Show the use of OpenAI documentation to have the llm generate calls to find real-time information
Implement an Agent and provide it with tools to get more real-time information

Operations

Find out how many tokens you are using and the cost
How to cache your calls to an LLM using exact matching or embeddings
How to cache the calculation of embeddings and run the calculation locally
Run your own local LLM (using Ollama)
Track your calls and log them to a file (using a callback handler)
Impose output structure (as JSON) and have the LLM retry if it’s not correct

Security

Explain the OWASP top 10 for LLMS
Show how simple prompt injection works and some mitigation strategies
How to detect prompt injection using a 3rd party model from Hugging Face
Detect project injection by using a prompt
Check the answer LLMs provide and reflect if it is ok
Use a Hugging Face model to detect if an LLM output was toxic
Show a simple prompt for asking the LLM’s opinion on Kubernetes and Trivy vulnerabilities

Real-World Applications of LangChain AI in DevOps

Vector similarity search in a RAG application. Credit to LangChain and Neo4J team

LangChain AI has practical applications in DevOps, particularly related to question-answering using documents as context, extraction, evaluation, and natural language processing tasks, all powered by AI tools. An interesting scenario is implementing a knowledge graph based RAG application with LangChain to support the DevOps team.

ChatGPT based on LLMs enabled with these AI tools is one of the foremost implementations at present. Tracking capabilities through LangChain enables developers to determine which prompts are more effective, thus supporting what artificial intelligence offers when integrated into development operations.

Monitoring and Anomaly Detection with LangChain AI

Monitoring LLM performance with LangChain and LangKit. Credit to WhyLabs

LangChain AI provides an efficient approach to DevOps through continuous monitoring, and anomaly detection. LangChain’s integration in DevOps allows for intelligent monitoring of various system parameters, application performance metrics, and log files. It can process large volumes of data generated by applications and infrastructure components. Through natural language processing (NLP) techniques, LangChain can interpret logs, error messages, and system alerts in real-time.

LangChain’s AI-driven anomaly detection capabilities are pivotal in identifying deviations from expected behavior in the monitored data. By analyzing historical data patterns, LangChain can learn the normal behavior of the system and its components. When the system encounters an unusual event or behavior, LangChain can promptly detect it as an anomaly. These anomalies could be security threats, performance bottlenecks, code quality, or any irregularities within the system.

Keep in mind that monitoring LLM performance is important to ensure you can rely on the model’s accuracy and relevance in real-world applications. Our customer WhyLabs has a great article showing the significance of monitoring LLMs performance with LangChain and how to get started with monitoring.

Future: AI, Apps, and DevOps

Platform engineering, DevOps, or MLOps may be the next steps in the AI-driven software development future. AI tools are the gateway to giving developers access to big language models that allow them to create highly sophisticated solutions.

Accelerating Development with AI-Powered Tools

AI-powered tools are revolutionizing the development process, enabling DevOps teams to access intelligent recommendations and improve code quality or obtain suggestions on new ways to accomplish goals. An example is the usage of Pulumi Neo.

Tools like LangChain, Amazon CodeGuru, and Pulumi AI provide actionable insights for efficient resource utilization while enforcing policy and compliance best practices. An IaC that leverages AI empowers teams to automate and more effectively manage infrastructure.

Developers can reap multiple benefits from higher quality assessment of code lines, instantaneous detection of policy and security issues, useful cues on how to resolve problems, and improving code quality, productivity, and time management.

AI has greatly impacted optimization during the review stage, providing developers with better insights into performance issues, code effectiveness, and resource utilization without any extra effort.

Security and Vulnerability Management with AI

The marriage of AI, machine learning, and DevOps/DevSecOps practices revolutionizes security and vulnerability management in software engineering. AI-enabled tools technologies not only strengthen the defenses of software systems but also enable organizations to stay one step ahead in ensuring successful deployments by removing manual processes through automation, using single sources of truth, enforcing least-privileged access through role-based access controls, and automated remediation to correct configuration violations like auto-tagging, removing Internet access, and enabling storage encryption.

DevSecOps can foster collaboration between developers, security, and operations and still enhance security and compliance best practices using Artificial Intelligence (AI) and Machine Learning techniques.

Leveraging Machine Learning in DevOps

Applying Machine Learning in DevOps is a game-changer. Not only does it automate and enhance various processes but also brings an intelligent, predictive, and adaptive dimension to development workflows. By embracing these technologies, development teams can achieve higher efficiency, improve code quality, optimize performance, and seamlessly integrate with artificial intelligence, thereby shaping the future of software engineering.

ML models can evaluate past outcomes from compilation/builds plus performance metrics within an operation so that developers may leverage these insights when writing code accordingly with machine learning algorithms, allowing them to identify vulnerabilities or issues faster through root cause analysis while ameliorating system security against outside attacks like hackers or DDOS activity thus boosting overall system productivity thanks to AI implementation into existing processes.

Implementing AI models into DevOps: Key Considerations

When considering AI, it’s essential to assess data quality, integration complexity, cost savings, and ethical/legal considerations:

Ensuring the data used for training these AI tools are precise, exhaustive, and up-to-date will help gain more reliable results.
An important aspect of the implementation process is integrating existing DevOps instruments and guaranteeing scalability, depending on the company’s necessities.
Ensuring that all laws applicable when working with such technology.

Conclusion

AI and ML are rapidly becoming a necessity, which can greatly enhance and benefit Development, Security, and Operation teams. At first glance, it may appear daunting or out of reach, but tools like Langchain and Pulumi can make it easier to get started. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Who is Patrick Debois?

Patrick Debois is a highly accomplished technologist whose expertise spans the domains of Development (Dev), Security (Sec), and Operations (Ops). Acknowledged as a respected confidant within the developer, security, and operations communities, Patrick is presently deeply engrossed in the realm of Artificial Intelligence and Machine Learning, continually pushing the boundaries of his technical acumen.

He was the organizer of the inaugural DevOpsDays in 2009. He is credited with coining “DevOps” and co-authoring the renowned DevOps Handbook. In the past, Patrick has collaborated with esteemed technology organizations such as Atlassian and Snyk. Currently, he wears dual hats as the Vice President of Engineering and a Distinguished Engineer at Showpad.

Is LangChain Free?

Yes, LangChain is an open-source framework, free to use and modify.

How Does LangChain Work?

LangChain is a meticulously designed open-source framework tailored to simplify the creation of applications powered by large language models (LLMs). First, you will need to have a language model or you can use a public language model or train your own.

Once you have a language model ready, you can start building applications. The framework offers an array of tools and APIs, simplifying the process of connecting language models to diverse data sources, interacting with their environment, and constructing intricate applications.

LangChain operates by linking together various components, referred to as links, to establish a workflow. Each link in the chain performs a task and is connected in a sequence, enabling the output of one link to serve as the input for the subsequent one. This sequential linking allows the chain to execute intricate tasks by combining simple tasks seamlessly.

See LangChain’s get started docs for more information.

Does ChatGPT use LangChain?

ChatGPT utilizes LangChain to create custom artificial intelligence (AI). This allows for an individualized experience.

Pulumi 💜's Open Source

Joe Duffy — Fri, 11 Aug 2023 00:00:00 +0000

Pulumi stands with the open source community.

We are unaffected by HashiCorp relicensing their software yesterday, and express profound support for many of our cloud friends who have been affected.

Pulumi is true open source, uses the Apache 2.0 license, and does not and never will depend on BSL-licensed software in any way, HashiCorp owned or otherwise.

We look forward to continuing to serve our new and loyal customers, always with open source and our amazing, fast-growing community at our core.

Announcing the Speaker Lineup for PulumiUP 2023

George Huang — Wed, 24 May 2023 00:00:00 +0000

PulumiUP is our virtual user conference for the Pulumi community and anyone interested in Infrastructure as Code. Every year, we strive to build a program packed with technical talks, demos, and best practices with the goal of leaving you with new learnings and inspiration that will help you become a better cloud engineer. You’ll hear from industry leaders and experts about IaC, software engineering, DevOps, Platform engineering, and AI.

We hope you’ll join us virtually and take this opportunity to grow your knowledge and become infrastructure as code stars.

Get Ready for Amazing Launches and Demos

Learn about Pulumi’s newest features across the core IaC SDKs and Pulumi Cloud. And watch hands-on demos by the Pulumi team:

Joe Duffy, Co-founder and CEO
Luke Hoban, Co-creator of TypeScript and CTO
Meagan Cojocar, Principal Product Manager
Evan Boyle, Engineering Manager
Zaid Ajaj, Software Engineer

Final Speakers Announced

You’ll hear from innovators, leaders, and experts from multiple industries and learn about infrastructure as code, software engineering, and cloud engineering.

Sam Cogan - Solution Architect, WTW

Talk: Keeping your Infrastructure Code DRY (Don’t Repeat Yourself)
Sam is a Solution Architect with WTW and a Microsoft Azure MVP. Sam is focused on delivering applications into the cloud using IaaS, PaaS, Serverless, Containers, and Kubernetes. Sam is particularly focused on automation and DevOps, including Infrastructure as Code, Cloud automation tooling, PowerShell, and CI/CD tooling.

Dennis Sauvé - DevOps Engineer, Washington Trust Bank

Talk: How a Bank Modernized its Software Engineering with Infrastructure as Code Automation
Dennis is a development operations engineer at Washington Trust Bank in Spokane, WA. With years of experience in software development and cloud engineering, he works to streamline deployment pipelines and create solutions for developers.

Ala Shiban - Co-Founder & CEO, Klotho

Talk: Creating a ‘Cloud-Aware’ Code Solution
Ala is the co-founder of Klotho, a startup that transforms plain code to cloud-native code. He started his journey on Commodore64 with cassette tapes and (much) more recently worked on distributed systems, developer tools, and large-scale web, cloud, and microservice backends.

He’s led smaller teams and larger organizations at Microsoft’s Developer Division and Riot Games’ Infrastructure Platform organization.

Matt Stephenson - Sr. Principal Software Engineer, Starburst Data

Talk: Creating Infrastructure Automation Magic with Code
Matt is a Sr. Principal Software Engineer at Starburst Data. He’s responsible for Starburst’s infrastructure architecture. He’s worked at many tech companies, including Square, Oracle, Google, and Amazon. Aside from technology, Matt is a pilot and helps build The Man for Burning Man each year.

Tim Holm - Co-Founder & CTO, Nitric

Talk: Engineering a Multi-cloud Platform
Tim Holm is the Co-founder and CTO at Nitric, with 12 years of experience in software engineering and cloud development. In disciplines ranging from Server and Storage Virtualisation and Automation, Mobile Application development, Full-stack application development, and Cloud infrastructure automation. Before founding Nitric, Tim worked to deliver critical infrastructure projects in Government and Fintech.

In his spare time, Tim enjoys video games, building Lego, and mountain biking.

Tyler Scheuble - Head of Platform, People Data Labs

Talk: Using Monorepos to Level-up your Infrastructure as Code and Software Delivery
Tyler possesses a genuine enthusiasm for technology. At People Data Labs, he has designed petabyte-scale data pipelines and high-performance APIs with demanding availability requirements, which informed his approach to leading the development of the next generation of the startup’s development pipelines and environments.

Tyler is passionate about finding simple solutions to cross-cutting concerns such as security, observability, and development velocity.

Sefi Genis - Co-Founder & CTO, Firefly

Talk: Lessons Learned from Writing Thousands of Lines of IaC
Sefi is the CTO and Co-Founder at Firefly. He started his career in the 8200 Unit of the Israeli Intelligence Corps and then continued on to security and developer tooling companies.

Sefi was a founding engineer at Dome9 Security (later acquired by Checkpoint), where he led the Backend Infrastructure of the product post-acquisition. Before founding Firefly, he became Head of Engineering in an Israeli cybersecurity startup company. Today he is building Firefly to help engineers manage the new reality of cloud-native operations.

Panel Discussion - AI and the Future of Cloud Development

Prepare yourself for the future of cloud development! Gain valuable insights from industry experts in our panel discussion, where we explore the transformative impact of AI on the cloud infrastructure and software engineering industries. The panel discussion will be moderated by CEO Joe Duffy. These are the panelists:

Amanda Silver - CVP Developer Tools, Microsoft

Amanda is the CVP of PM for Microsoft’s Developer Division, including the Visual Studio family of products, .NET, TypeScript, and our developer platforms. She has been key to Microsoft’s transformation to contribute to open source with the introduction of TypeScript, Visual Studio Code, and the acquisition of both Xamarin and GitHub. She believes that a tight digital feedback loop with zero distance between end-users and engineering teams is a critical element of great product development.

Beyang Liu - Co-Founder & CTO, Sourcegraph

Beyang is the CTO and co-founder of Sourcegraph. Prior to Sourcegraph, Beyang was a software engineer at Palantir Technologies, where he developed new data analysis software on a small, customer-facing team working with Fortune 500 companies. Beyang studied Computer Science at Stanford, where he published research in probabilistic graphical models and computer vision at the Stanford AI Lab and thoroughly enjoyed his compilers course.

Luke Hoban - TypeScript Co-Creator and CTO, Pulumi

Luke is CTO at Pulumi. He has worked on developer tools his entire career. He started out on Visual Studio, C#, and .NET in the early 2000s, later joined the ECMAScript standards body as a representative of Microsoft and then became one of the co-founders of the TypeScript programming language.

Featured Partners

Thank you to our amazing partners for supporting PulumiUP! Visit these booths for tech talks, demos and Q&A about your favorite cloud platforms and developer tools:

AWS - Solutions Architects will be on hand to demo the latest AWS and Pulumi capabilities and help you get the most from the AWS platform.
Google Cloud - Customer Engineers will help you accelerate your cloud journey with helpful tips and Q&A.
Microsoft - Cloud Advocates will be available to show you how to kickstart your projects on Azure.
Equinix - Stop by the Equinix booth to learn all about the new Equinix Pulumi provider.
CircleCI - Stop by the CircleCI booth to learn best practices for adopting CI/CD in your organization.
New Relic - The New Relic team will demonstrate how to get started with Observability as Code with Pulumi and New Relic.
Firefly - If you have existing cloud resources you can convert them to Pulumi code easily with Firefly. Stop by the Firefly booth for a demo.
Nitric - The Nitric team will be on site to show you how to create cloud-agnostic infrastructure code from your application code.
Klotho - The Klotho team will show you how devops teams an application teams can easily collaborate to moderize and scale infrastructure without disruptions.
Port - Join the Port team to learn how to setup a self-service portal for your developers.

Want to attend PulumiUP 2023? Register now!

Join a Pulumi User Group (PUG) Meetup!

Sara Huddleston — Mon, 01 May 2023 00:00:00 +0000

We have a global community made up of people from many different countries, but we also have many local communities. We want to enable our users to meet, share knowledge and ideas, answer questions, and learn about Pulumi and best practices. For that reason, we created the Pulumi User Groups, aka PUGs on Meetup.

Members of PUGs will have access to a global network of experts who can offer help, advice, and support on using Pulumi effectively. The PUGs will also feature regular in-person meetups, webinars, in-person and virtual workshops, and other events to help members stay up-to-date with the latest cloud infrastructure and Pulumi developments.

23 Pulumi User Groups (PUGs) Meetups in 10 countries

PUGs are led by Pulumi employees and Puluminaries, who are passionate community members that are experts in topics related to cloud infrastructure, Pulumi Cloud, cloud-native software development, and much more, and are willing to share their knowledge with the community.

United States Pulumi User Groups

South America Pulumi User Groups

São Paulo Pulumi User Group

European Pulumi User Groups

Middle East Pulumi User Groups

Tel Aviv Pulumi User Group

Asia Pulumi User Groups

Singapore Pulumi User Group

Oceania Pulumi User Groups

The First PUG in person

Our very first in-person PUG meetup took place in London, UK. The event was well-attended by developers, cloud architects, and other infrastructure enthusiasts. The attendees had the opportunity to network with other like-minded professionals and learn more about Pulumi’s tools and features from expert speakers.

During the meetup, attendees heard from several speakers, including Joe Duffy, Co-Founder & CEO at Pulumi, who was there to speak about Infrastructure as Code that writes itself after the release of Pulumi Insights, which happened that same day.

Other speakers included Liam Hampton, Sr. Regional Cloud Advocate at Microsoft, who spoke about Microsoft Azure and IaC, and Sam Cogan, Director of Solution Architecture at Willis Towers Watson, who spoke about building a library of reusable modules you can repeatedly reuse in your IaC projects.

Overall, the first PUG meetup was a huge success, and attendees left feeling inspired and energized about Pulumi’s tools and the community’s possibilities. With the launch of PUGs, Pulumi is demonstrating its commitment to building a strong and supportive community of professionals passionate about cloud infrastructure.

More PUGs and events to come

We are planning more in-person PUG events for the months to come. With PUGs, you’ll have the opportunity to connect with like-minded professionals, share knowledge and best practices, and stay up-to-date with the latest developments in cloud infrastructure and Pulumi.

Join the Pulumi User Groups in your local area, or if there isn’t one yet, you are welcome to join the Pulumi Slack community and meet people from all over the world.

International Women's Day: Celebrating our Women in Tech

Sara Huddleston — Wed, 08 Mar 2023 13:00:00 -0800

Today is International Women’s Day, and this year the theme is #EmbraceEquity - which means creating an equitable environment. An equitable work environment means understanding that everyone, regardless of gender, religion, ethnicity, background, or resources, brings strength to the workforce and that opportunities should be given to them based on their individual needs.

For Pulumi, it means a work environment where everyone can share ideas and respect them even when disagreeing. Women’s experiences - as well as men’s and nonbinary’s experiences - inform the direction of digital technology and innovation.

Being International Women’s Day, today we focus on women in tech.

Women in Tech Stats Awareness

Women earn an average of 17% (US) to 13% (EU) less than men, according to U.S. Census Bureau and European Commission.
60% of women in tech said a parent or teacher encouraged them. Logitech and Girls Who Code’s report on understanding the barriers women face in tech noted the importance of having role models and people encouraging them.
In small companies, 30.2% of employees are women, according to the report Top Companies for Women Technologists.
75% of women in tech are asked to handle more administrative tasks compared to their male colleagues, according to Navisite’s survey.

Below, you can meet some of Pulumi’s tech women - their experience, why they chose tech, and advice to thrive.

Pulumi’s Women in Tech

Sarah Hughes, Software Engineer

“I was lucky enough to go to high school with a ton of other girls who were into tech, so I took AP Computer Science in a class that was close to a 50/50 split. It really helped to start from scratch on equal footing. (Almost) everyone knew absolutely nothing, and it was obvious that our strengths and weaknesses were completely independent of gender.”

Lina Desloge, Account Executive

“I have always enjoyed problem-solving and working with people…which is what ultimately led me to a technical product. The advice I would give my younger self is don’t worry, just start!”

Kelsey Dirks, Customer Success Manager

“When I entered university, I chose Electrical Engineering as my major. I soon learned coding was not my specialty and switched to Communications, but I loved the people in the classes. It was clear both the professors and students were passionate about their craft. I knew that when I left university, I wanted to work with these types of people, but I had no idea in what capacity. Over the years, my experience led me to be a Customer Success Manager.

What I would tell my younger self is to not stress about trying to figure everything out right away. Every job will have its ups and downs, but when it comes to a career, trust that it will all work out. The biggest piece of advice I can give to women entering the tech field is don’t be afraid to lean on your female support system. If you aren’t able to advocate for yourself, your sisters in STEM will.”

Jonelle Boyd, UX Designer

“If I could give advice to women entering the tech field, I would tell them don’t be afraid of change. Whether that be a role change or a job change, because every new opportunity is a chance to learn and grow. I’ve changed roles as well as worked at a variety of different places during my career and I’ve learned something new in every position. There’s tremendous value in having a wide range of experience, so never be afraid of making that change.”

Casey Huang, Software Engineer

“To my younger self - At some point, you’ve heard people sell working in the tech industry by saying the hours are more flexible - you’re not chained to your desk 9 to 5, you can leave early as long as you get your work done. This is somewhat true, but you should know: there is always more work to do. Some of it is important enough to move the business needle. Some of it isn’t as important, but you just care a lot about it. It’ll be up to you to define your boundaries, and it will be up to you to defend them. But it’s okay; if your work environment is truly a good fit for you, they’ll follow your lead and support you. You can’t min/max life like you like to do in video games. What you’re doing is good enough. Don’t be so afraid of making mistakes. If anything, you’ll learn faster that way.”

Guinevere Saenger, Software Engineer

“Working in tech provides me with significant financial independence, professional respect, and a flexible work schedule. These are worthwhile for anyone to have, but especially so for women.”

We’re always looking for more amazing talent to join us in our community and as part of the Pulumi team.

Announcing KubeCrash Fall 2022 — the KubeCon Detroit Warm-up

Kat Cosgrove — Fri, 23 Sep 2022 10:19:23 -0500

100% Virtual. 100% Free. 100% Open Source. Arrive in style and up to date on the biggest trends for KubeCon + CloudNativeCon North America this October in Detroit, Michigan! KubeCrash is a fully-virtual two-day event curated by the coolest companies in cloud native, providing you with a practical, hands-on learning opportunity in the run-up to the conference. All for free! Held October 5 and 6 in both Americas and European time zones, you’ll emerge with new development skills and conversation starters for KubeCon. Take a peek at the program and register online!

The Importance of Open Source

Your DevOps team drives the technology and tooling choices that build production. The benefits of open source mean that even in enterprise-grade environments, they increasingly choose an open-source tool over a closed-source option. Large communities of maintainers and other users, great documentation, and personalized support make open source an extremely appealing choice.

This means that open source software has become the backbone of any modern tech stack. That rapidly-increasing reliance on open source means that any development team needs the skills to understand and effectively utilize these tools to remain on top. The KubeCrash event is tailor-made for this, with sessions and hands-on workshops for SREs, developers, platform engineers, and security engineers hosted by the maintainers of some of the most popular open source tools in the cloud native landscape.

About KubeCrash

Our inaugural KubeCrash event, held alongside KubeCon + CloudNativeCon Europe, was a resounding success. This year, we’re making it even better with feedback from all of you! KubeCrash Fall will be held before KubeCon, with two half-day events held in different timezones to accommodate everyone. You’ll be able to interact with speakers and other participants live, no matter where you are! One event will be held from 3pm to 6pm Central European Time, and the next will be held from 9am to 12pm US Pacific Time. KubeCrash is a collaborative effort by seven different open source companies in the cloud native space, all of us coming together to bring you high-quality crash courses on cloud native technologies before KubeCon. We aren’t here to sell you anything or make you sit through pitches; you’ll get content on Pulumi, and also other cool open source projects like Goldilocks, cert-manager, Linkerd, CockroachDB, and Polaris. Check out our last event for an idea of what you’ll see!

The KubeCrash Program

Show up ready to learn straight from the people who build and maintain the most popular and critical tooling in the ecosystem. From keynotes by Cheryl Hung to overviews of CNCF projects by their maintainers, to hands-on workshops and an end-user case study, there’s something for you no matter where you are in your cloud native journey.

Day 1: Wednesday, October 5 (European and East Americas-friendly timezone)

3 to 6 pm CET | 9 am to 12 pm EST | 8 to 11 am CST

Keynote: to be announced

Lighting talk: Intro to the CNCF Cloud Native Maturity Model by Danielle Cook, Co-chair CNCF Cartografos Group

Whose cert is it anyway? How to build TLS trust using cert-manager by Ashley Davis, cert-manager team

Going Straight to the Edge by Dinesh Majrekar and Mark Boost, founders of Civo Emissary-ingress session

Hands-on workshop: Service mesh observability with Linkerd by Flynn, Linkerd team and Emissary-ingress maintainer

Day 2: Thursday, October 6 (Americas-friendly timezone)

12 to 3 pm EST | 11 am to 2 pm CST | 9 am to 12 pm PST

Keynote: TBS by Cheryl Hung, Engineering Manager, Apple

Lighting talk: The CNCF Cloud Native Glossary: Trusted. Simple. Community-driven by Catherine Paganini, Cloud Native Glossary Maintainer

Getting Started with Kubernetes Guardrails with Polaris by Andy Suderman Creator and Maintainer of Goldilocks and Stevie Caldwell, Creator of GoNoGo

Hands-on Workshop with Aaron Friel of Pulumi: Program the Cloud with Six Pearls in Six Languages using Pulumi

Check out the program page for more details.

Join us on October 5 and 6

Join us on October 5 and 6 for a specifically curated set of sessions, each led by a project maintainer, from projects covering modern cloud native security to improving the developer experience. It will be live, interactive, and fun. Register today!

Teaming up to help deliver more meals to Ukraine

Wendy Smith — Mon, 16 May 2022 09:00:54 +0200

Why not do good while learning about cloud native tech? Buoyant, Civo, Cockroach Labs, Fairwinds, Pulumi, and Jetstack have teamed up to donate $2 for each virtual booth visitor to World Central Kitchen (WCK), a non-profit committed to providing meals in response to humanitarian crises.

Working with restaurants across Ukraine and in six countries welcoming refugees, WCK is providing hundreds of thousands of daily meals for people in need. WCK serves meals at shelters, hospitals, transit hubs, and more. It’s a wonderful mission and, if you agree, we encourage you to donate to WCK directly 💙 🇺🇦 💛

How it works

Participating is easy. If you registered for KubeCon, just click on the booth links below to visit our virtual booths. Each visit will count as a $2 donation, so if you visit us all, that’s $12 for the cause right there!

Of course, there are many more reasons to visit our booths. You’ll find lots of great cloud native resources including demos, tutorials, ebooks, and more. Here are some of the highlights you can expect at each booth.

Pulumi — Universal Infrastructure as Code

At the Pulumi booth, you’ll learn about our cloud engineering platform, which brings infrastructure, developer, and security teams together through a unified software engineering process that tames cloud complexity and accelerates innovation. Don’t miss our Ask an Expert live sessions to hear directly from the engineering team. Whether you want to know more about how to build with Universal Infrastructure as Code in general-purpose programming and markup languages or just learn more about what’s on the roadmap, this is your chance to ask us anything.

Buoyant — the creator of Linkerd

Buoyant is announcing something really exciting: the introduction of fully managed Linkerd. You can be one of the first to check out the demo, and you’ll also find a ton of service mesh content including a service mesh 101 in English and Spanish, their Engineer’s Guide to mTLS, and details on Buoyant’s Service Mesh Academy.

Civo — Fast Simple, Managed Kubernetes

Stop by the Civo booth for live sessions on how to get the most out of your Kubernetes environment - including demos, Civo marketplace walkthroughs and an inside look at how they live-provision a private region. You’ll find lots of resources and friendly team members to answer any questions on Civo’s easy-to-use managed Kubernetes service. It is fully CNCF certified and 100% Kubernetes upstream compatible.

Cockroach Labs — Build What you Dream

CockroachDB’s presence this year at KubeCon (both physical and virtual) is primarily a charitable one (if you attend in person, visit booth P16, where for each badge scanned they’ll donate $3 to either Black Girls Code, Women Who Code, the Cancer Research Institute, or UNICEF for Ukrainian relief). Visit their booth to see demos and architecture overviews that relate what they’ve learned from building a multi-region, managed distributed SQL database offering on Kubernetes.

Fairwinds — Kubernetes Governance and Security

Fairwinds, a leader in Kubernetes governance providing a unified view between dev, sec, and ops, offers lots of great resources at their virtual booth this KubeCon. You can get a quick tour of Fairwinds Insights, learn about Kubernetes cost savings, how to make Kubernetes service ownership possible, typical Kubernetes security mistakes and much more. Come chat with a friendly Fairwinds engineer to talk about the many great open source solutions including Polaris, Goldilocks, Nova, and Pluto. Visit the Fairwinds booth today!

Jetstack — the creator of cert-manager

Get more out of cert-manager and find out how Jetstack Secure can eliminate workload misconfigurations and provide full visibility of each certificate across every production cluster. Chat directly to one of the project maintainers and learn to use cert-manager to secure Ingress endpoints and internal workloads using mTLS.

Lots of great reasons to visit our booths

There are plenty of good reasons to check out the great content of these amazing companies while supporting a wonderful charity that is doing an incredible job at helping the people of Ukraine.

Introducing the Puluminaries

Matty Stratton — Tue, 03 May 2022 10:04:47 -0500

Pulumi is more than a way to build, deploy, and manage your infrastructure and cloud applications. Pulumi is also a strong and vibrant community. We are very excited to announce and showcase our new program of community champions, the Puluminaries!

The Puluminaries program was created to recognize members of the Pulumi community who are experts and leaders in the field of programming and infrastructure, but also who give back to the community to make it grow! They help others get started, give feedback to make Pulumi’s roadmap even better, and educate people through content or public speaking. The program is open to all members of the community, and we welcome anyone to join the program.

While the Puluminaries themselves have been active in the community for quite some time, we officially launched the program at PulumiUP in 2022, and it is our pleasure to introduce the first cohort of members to the program!

Meet The Puluminaries!

Engin Diri

Engin’s excitement for Pulumi is not possible to be measured using current science. He will always share his thoughts on new developments in the Pulumi project, as well as join in for fun and discussion in various PulumiVision livestreams!

Joshua Studt

Joshua is a longtime member of the Pulumi community, and he contributed to the implementation of Automation API for C#.

Kat Morgan

Kat has been a helpful member of the Pulumi community - not only providing feedback on real-world use of Pulumi, but also sharing her knowledge with workshops and webinars

Komal Ali

In addition to being a Puluminary, Komal is also a Pulumni, having spent time working to make Pulumi a better tool for developers - they were ever-present in making our Python SDK better and creating the Automation API!

Paul Hicks

Paul seems to have an answer to everything. He is ever-present in the Pulumi Community, sharing his knowledge to folks who are just getting started, but also bringing to bear his real-world expertise in complex Pulumi implementations. Chances are, if you see a great answer to a question in Pulumi Community Slack, it came from Paul.

Ringo De Smet

Ringo is one of the founding board members of the Pulumiverse - a place to interact and collaborate on Pulumi-based libraries, projects, and educational materials. Ringo (along with fellow Puluminary Simen A. W. Olsen) have brought their vision for expanding the Pulumi community to life with the creating of this community-operated space! Ringo also created the Pulumi Resource Type for Concourse, and is ever-present in every community he is a part of, not just Pulumi’s!

Rizel Scarlett

Rizel brings her remarkable talent for guidance and teaching to the Pulumi community. She has participated in many Pulumi events, including Cloud Engineering Summit, and appearing on the PulumiVision livestream.

Simen A. W. Olsen

Another one of the founding board members of the Pulumiverse, Simen has a great passion for technology communities, but especially for the Pulumi community. Simen was instrumental on the Pulumi GitHub Action rewrite - designing and rearchitecting it in his free time.

Do you know someone who would make a great Puluminary? Let us know! You can email us at da@pulumi.com with your nominations!

PulumiUP Workshop Series

Matty Stratton — Tue, 03 May 2022 08:46:08 -0500

At Pulumi, we’re incredibly fortunate to have over 70 integration partners in our ecosystem – helping shared end-users to build, deploy and manage practically any cloud service they can imagine. Our most popular content often includes workshops that show end-users how to use these powerful integrations. This year for PulumiUP, we’re excited to announce that we’ve teamed up with a number of partners to deliver a workshop track that provides hands-on labs and demonstrations for a variety of platforms and scenarios.

Introduction to Infrastructure as Code

Thursday, May 5th at 2:30 PM PDT

We’ll explain the basics of IaC by exploring how to use Pulumi to build, configure and deploy a real-life, modern application using Docker.

From Zero to Production in Kubernetes

Tuesday, May 10th at 9:00 AM PDT

Our friends from NGINX will show you how to leverage the power of Python with Pulumi to define and manage your Kubernetes deployments and build powerful abstractions that make getting to production easier than ever before. You’ll learn about the new open-source Modern Application Reference Architecture (MARA) that helps you quickly deploy an app platform that includes Amazon EKS, NGINX Kubernetes ingress controller, logstore, logagent, cert-manager, Prometheus, Grafana, and more.

Kubecrash

Tuesday, May 17th at 9:00 AM PDT

Join Pulumi and the maintainers of Linkerd, cert-manager, and other projects for a series of crash courses about Kubernetes and cloud-native, open source technologies.

Getting Started with Azure and Infrastructure as Code

Wednesday, June 1st at 9:00 AM PDT

In this workshop, you will use Pulumi’s Azure Native Provider, built directly from the Azure API, to provision infrastructure with Pulumi’s Typescript SDK. Join Microsoft Sr. Cloud Advocate, April Edwards, and Pulumi Developer Advocate Matty Stratton for this introduction to IaC.

End-To-End Infrastructure, Apps, and Auth with Auth0

Tuesday, June 7th at 4:00 PM PDT

In this live-coding session, Pulumi and Auth0 will walk through the process of building a three-tier web app: a single-page app built with React, a back end consisting of a REST API managed with Express and MongoDB. Finally, we’ll show you how to enable authentication to restrict access to your app.

AWS Immersion Day: Hands-On Infrastructure as Code

Tuesday, June 14th at 9:00 AM PDT

Pulumi and friends from VirtusLab and AWS have planned an interactive session to introduce IaC concepts using familiar programming languages to provision modern cloud infrastructure. This session will focus on new Pulumi capabilities.

GitOps with GitHub Actions and the Pulumi Operator

Tuesday, June 21st at 9:00 AM PDT

Combining GitHub Actions with the Pulumi Kubernetes Operator helps you to implement powerful GitOps workflows and automation for both your infrastructure and workloads. We’ve partnered with the GitHub team to deliver step-by-step labs to help you to get started with GitOps.

CI/CD Pipelines for Kubernetes Apps with Pulumi & Codefresh

Wednesday, June 22nd at 9:30 AM PDT

The Codefresh team will help you to learn the basics of CI/CD and how to declare cloud resources and set up a simple pipeline for Kubernetes deployments using your favorite programming languages with Pulumi and Codefresh.

Deploying Microservices with AWS Lambda and Pulumi

Tuesday, June 28, 12:00 PM PDT

In this workshop, we’ll examine how Pulumi can rapidly accelerate provisioning of cloud infrastructure. We’ll focus on AWS Lambda and build an example set of microservices utilizing AWS’s newest Lambda features. Matty Stratton (Pulumi) and Marina Novikova (AWS) will guide you through the process of provisioning a set of example Lambda resources in AWS.

Getting Started with Infrastructure as Code on Oracle Cloud

Thursday, July 14th at 9:00 AM PDT Register

Now that we have a brand new integration with Oracle Cloud Infrastructure (OCI) the team is preparing a new workshop to help you to get the most out of the new provider. Laura Santamaria (Pulumi) and Gaurav Jain (Oracle) will show you how to define, deploy and manage OCI resources using your favorite programming languages.

Getting started with Infrastructure as Code on DigitalOcean

Thursday, July 28th at 8:00 AM PDT Register

In this hands-on workshop, Matty Stratton (Pulumi) and Chris Sevilleaja (DigitalOcean) will show you how to stand up basic services using Infrastructure as Code with hands-on labs using JavaScript/TypeScript.

Announcing Pulumi Community Slack Archive

Lee Briggs — Mon, 02 May 2022 12:00:00 -0700

Creating a place for the Pulumi community to gather, ask questions, get help in real-time, and share successes has been an important part of the explosive growth we’ve seen in both users and customers. The Pulumi community slack has grown to over 7000 members and well over 200,000 messages.

Within those 200,000 messages are years of information kept behind a “walled garden” that is undiscoverable outside Slack’s search capabilities.

Today we’re announcing that our community slack is now publicly available via https://archive.pulumi.com thanks to Linen

For many, Slack has become the de-facto collaboration tool. Adding Slack workspaces is relatively straightforward and means you can communicate with various different communities and your colleagues in real-time.

This ability to use the tools you’re using in your everyday work to also quickly get help with Pulumi has proven to be extremely successful. Our users often tell us that having access to Pulumi staff and other community members helps them accelerate their cloud engineering journey. In addition to providing value to our users, the Pulumi team enjoys having a mechanism to engage with users to get feedback and inform them about the Pulumi ecosystem. It’s clear to us that the Pulumi community slack has proven invaluable as Pulumi has grown.

The use of Slack for our community meeting place isn’t without its problems. Often, the first port of call for users that need answers is to use search engines, and all of the information stored in Slack isn’t readily available by default with Slack. The Pulumi team regularly answers questions on StackOverflow and via GitHub Discussions which are searchable and indexed by search engines, but adding the treasure trove of information from Slack will improve the experience for eager Pulumi users.

Until now, this has been exceptionally difficult. With the recent launch of Linen we are able to bring our community chat outside of Slack’s boundaries in an easily searchable, indexable way.

We’re excited to provide this to the community and encourage you to try it out at https://archive.pulumi.com.

Introducing KubeCrash: Cloud Native Crash Courses

Kat Cosgrove — Fri, 08 Apr 2022 00:00:00 +0000

Can’t make it to Valencia for KubeCon this year? Timezone doesn’t work for the virtual conference either? We can’t fix time, but if you’re feeling left out and still want some of that sweet cloud native content, you can still join us for KubeCrash, a new event hosting live crash courses and sessions on cloud native tech. Come hang out and learn directly from the maintainers of cloud native open source projects!

About KubeCrash

Five open source companies have teamed up to bring you top-notch, KubeCon-grade crash courses on cloud native tech. No vendor pitches, just awesome open source content on projects such as Linkerd, cert-manager, CockroachDB, Pulumi, Polaris, and Goldilocks.

With Kubernetes becoming the new standard for cloud-hosted application development, DevOps teams are driving the technology choices for enterprise-grade cloud native tooling. Freely available open source solutions are often the primary source for these tooling decisions.

KubeCrash provides a half-day knowledge sharing and virtual learning environment for developers, reliability engineers, cloud security specialists, and platform engineers. Learn directly from the maintainers of some of the most popular open source projects in this series of focused talks and workshops.

The KubeCrash program

Come prepared for a schedule packed with great content and actionable insights directly from the teams that maintain some of the ecosystem’s most popular open source projects. The program will cover the latest learnings on implementing scalable zero-trust, scanning workloads for improved cloud native security, using service mesh to ensure high availability across multi-cluster infrastructure, and delivering “serverless” for multi-cloud deployments.

Program overview

Using cert-manager to enable zero-trust identities for intra-pod communication — Jake Sanders, cert-manager maintainer

Modern cloud native architectures require the network to be considered untrustworthy and this is why internal workloads are rapidly driving the use of mTLS and private PKI. This workshop from Jetstack will demonstrate how to use cert-manager to issue, manage and rotate mTLS certs, allowing users to have strongly attested and verified Machine Identities between Kubernetes pods. All without the workload private keys leaving node memory! Think of this session as a precursor to implementing a service mesh solution, using cert-manager to establish zero trust environments, perhaps defined by trust domains, and enforce security for pod to pod traffic.

Multi-cluster failover using Linkerd — Eliza Weisman, Linkerd maintainer

Failover across clusters is a great way to improve the overall uptime and reliability of Kubernetes applications. While whole-cluster failover can be accomplished at the global ingress layer, failing over individual services is a little more difficult. During this session, Linkerd maintainer Eliza Weisman will walk you through how to use Linkerd, the CNCF graduated service mesh, to enable traffic failover for individual services across clusters. Attendees will learn how to combine service mesh metrics, traffic shifting, and cross-cluster communication in a cohesive and automated way using pure open source, while preserving fundamental security guarantees such as mutual TLS.

Optimizing and Securing Kubernetes Workloads with Polaris and Goldilocks — Rachel Sweeney, Fairwinds and Andy Suderman, Polaris and Goldilocks maintainer

Learn how to scan your Kubernetes workloads to improve your resource utilization and security using open source tools Polaris and Goldilocks. You will watch Andy Suderman, Director of R&D and Technology, and Rachel Sweeney, SRE at Fairwinds, as they show how to correctly configure your clusters based on Kubernetes’ best practices for security and efficiency.

Using Kubernetes to deliver a “serverless” service — Lisa-Marie Namphy and Jim Walker, Cockroach Labs

In this talk, Cockroach Labs team members will share how they leverage Kubernetes to deliver a “serverless” experience. Serverless promises to change the way we consume software. It allows us to potentially pay for what we use only and help drive down operational costs by minimizing resource consumption. Architecting for serverless requires a unique look at app logic and how it is deployed—a combination of the logical and physical worlds. An architectural pattern has emerged where we can scale ephemeral compute separate from services that need to persist.

Multi-cloud, single deploy: cloud engineering with Kubernetes and Pulumi — Aaron Friel and Guinevere Saenger, Pulumi

Business constraints and customer requests often require to stand up new Kubernetes environments across multiple cloud providers. This growing complexity in computing infrastructure will incur greater operational costs for organizations when coordinating across multiple teams. Pulumi engineers Aaron Friel and Guinevere Saenger will demonstrate standing up Kubernetes clusters, deploying applications, and automating ops tasks by building a CLI using the Pulumi Automation API. These tools empower every engineer—from application developers to site reliability engineers—to be a cloud engineer.

Join us on May 17

If you’re staying in the Americas (or are up for a late-night session) this KubeCon, join us on Tuesday, May 17th starting at 9 am PST/10 am CST/12 pm EST. Enjoy a specifically curated set of sessions, each led by a project maintainer from projects covering modern cloud native security to improving the developer experience. Register today!

Announcing PulumiUP 2022: The Cloud Awakens on May 4th

George Huang — Thu, 31 Mar 2022 00:00:00 +0000

Calling all cloud engineers! Today we announce the second annual PulumiUP virtual conference taking place on May 4, 2022. We will also host the inaugural Pulumi Community Summit on May 5 to virtually assemble the world’s cloud, infrastructure, and Pulumi practitioners of all experience levels (hint: that’s you—all are welcome)! Register for both events here.

We’ve worked hard to curate a half-day of learning and expert insights from the people that are forming the best practices of the modern cloud and cloud native infrastructure era. On May 4, join us for four hours of exciting Pulumi product news and demos, technical talks, case studies of modern cloud innovators, and conversations on cloud topics that you won’t hear anywhere else. Hear from Pulumi’s CEO and CTO, pioneering engineers and engineering leaders from different industries, and Pulumi’s engineers. We are also introducing two tracks designed for Practitioners and Engineering Leaders.

Who’s speaking at PulumiUP 2022?

Aaron Friel, Staff Software Engineer, Pulumi
Elad Ben-Israel, Principal Software Development Engineer, Amazon Web Services
Joe Duffy, CEO & Founder, Pulumi
Justin Fitzhugh, VP of Cloud Engineering, Snowflake
Kat Cosgrove, Staff Developer Advocate, Pulumi
Luke Hoban, CTO, Pulumi
Meagan Cojocar, Senior Product Manager, Pulumi
Tomas Jansson, Software Architect, Elkjøp Nordic AS

In the coming weeks, we will be announcing even more speakers. Stay tuned!

What’s in store at PulumiUP 2022?

Stay tuned as we will announce the full agenda and speakers soon on our website and in an email to registrants. Here’s what you can expect at this year’s event:

Keynotes: Hear from Pulumi leaders Joe Duffy (Founder & CEO) and Luke Hoban (CTO) as they share new product announcements, demos, and thoughts on trends in the modern cloud.

Technical Talks: Dive deep with Pulumi’s engineers and guest speakers from the industry to learn best practices for infrastructure as code with popular programming languages and see demos.

Case Studies: Learn from engineers and leaders about how they have accelerated development velocity and increased reliability for their modern cloud applications with Pulumi.

Expert Conversations: Listen in on lively debates and discussions about the intersection of software development and cloud infrastructure from panels made up of industry leaders and practitioners.

Two Tracks: Choose from two session tracks that are designed for Practitioners and Engineering Leaders, in addition to the keynote and main stage presentations.

Inaugural Community Summit: Connect and engage with Pulumi practitioners from around the world—from the comfort of your home. We are hosting a Community Summit on May 5 featuring lightning talks and discussions.

Workshops: Get hands-on with Pulumi and cloud engineering in 1-hour workshops that will cover the fundamentals of Infrastructure as Code as well as more advanced subjects for experienced Pulumi users.

Join us on May 4 and 5

Register for PulumiUP and the Community Summit now! We will send you updates on the final agenda and speakers and post-event updates such as session recordings and upcoming events.

Introducing the Pulumiverse

David Flanagan — Wed, 30 Mar 2022 10:00:00 +0000

Today, we’re excited to announce that we’re working with the Pulumi community to provide a place to interact and collaborate on Pulumi-based libraries, projects, and educational materials: the Pulumiverse.

This project is especially important to me, because it’s an idea that I’ve tried in the past, 2018 😯, and failed to commit to and give it the attention it deserves.

Fortunately, two other wonderful Pulumi community members stepped up and started to put together their own programme and with me now being at Pulumi, the timing was amazingly serendipitous.

I spoke with Ringo De Smet and Simen A. W. Olsen, the founding members of the Pulumiverse.

Ringo, Head of CI/CD at Napoleon Games, had this to say about why he’s excited to create the Pulumiverse:

For infrastructure automation, we had other great tools which preceded Pulumi: Chef, Puppet, Ansible. Any of these tools have a vibrant community of people preventing that all of us need to re-invent the wheel. We had Sous-Chefs for Chef & Vox Pupuli for Puppet. But I see a lot of Pulumi users implementing the same stuff over and over again. That’s why I think Pulumi needs it’s own community. So I started it: Pulumiverse

Simon, a Developer at Bjerk AS, shared:

I imagine a community filled with like-minded people who build and support Pulumi providers, opinionated components, configurations and share ideas on how we can utilize the power of Pulumi in new ways. So I created Pulumiverse to share my enthusiasm, give back to Pulumi and support developers by building a platform where we can work together!

What is the Pulumiverse?

Pulumiverse, in the simplest form, is a GitHub organization. It provides a community-operated place for anyone to communicate, contribute, and collaborate with other members of the Pulumi community and build abstractions and SDKs that can be shared and used by anyone.

There are many libraries or collections of components for Pulumi out there. Most of them are made based on requirements of individual companies and developers. Pulumiverse aims to make a community where these libraries can be created and where opinionated configurations for Pulumi resources can be stored and discussed.

Current Projects

The Pulumiverse launches with just a few projects, but we know that, with the support of such an amazing community, the Pulumiverse will grow and grow and provide invaluable resources for anyone who wants to build or use Pulumi for a library or project.

Awesome Pulumi

🐙 Awesome Pulumi

Everybody loves an awesome list, and this one delivers to you a list of awesome Pulumi resources, books, tutorials, talks, and much more.

Want your project listed? Open a PR. We can’t wait to share it with everyone.

Kubernetes SDKs

🐙 Kubernetes SDKs

The Kubernetes SDKs project aims to provide multi-language SDKs for interacting with Kubernetes Custom Resource Definitions (CRDs). Currently, it automatically builds SDKs for:

ArgoCD
CertManager
Contour
Crossplane
Istio
Kong Ingress
Knative
RedPanda
Tyk

This is just the beginning, and we want to expand this collection quickly. So if you’re working with Kubernetes and want a rich developer experience for authoring custom resources, open an issue and let us know what you need.

Pulumi Concourse & Unifi

These are community-maintained providers and are examples of how you can share your providers with the world as part of the Pulumiverse.

Your Project?

Got your own library that you think would be beneficial to the community?

Join us in the #pulumiverse channel on the Pulumi Slack, and we’ll be happy to help you get it over to the Pulumiverse.

We’ll see you soon! 👋

Migrating My Infrastructure From Terraform to Pulumi

Erik Näslund — Tue, 22 Feb 2022 12:36:23 -0600

Pulumi community member Erik Näslund shares his thoughts on how to migrate from Terraform to Pulumi. Read on to learn all the details of his experience!

I’ve been using Terraform for a couple of years and overall I’ve been quite happy with it. However there’s a few things that started to bother me more and more recently.

Terraform uses a language called Hashicorp HCL to define the infrastructure. It’s a relatively simple declarative language, but it’s something I had to learn along the way. Just like any language it has it’s little quirks, and I often found myself spending more time than I wanted to figure out how to do certain things. As I’m doing all the infrastructure myself I really wanted to be able to use a language I’m familiar with, to make things simple.

I had written my own Terraform modules to organize my code. Modules are a great help when it comes to organizing your code in a logical way, and I strongly recommend it if you haven’t tried using them already. However, after using it for a while I realized that the Terraform type system left something to be desired. There are a few basic types available, but not much more than that. This caused bad “micro interactions” between me and Terraform. It was commonplace for me to submit an MR to my Gitlab CI where I passed in a security group name instead of an arn or id.

The third and perhaps most important point is that I think a custom language like HCL is the wrong way to go long term. Don’t get me wrong - I love the people at Hashicorp and I’m forever grateful for Terraform because it’s the first product that made me enjoy writing IaC. That being said, I think it’s a better choice to write libraries for existing languages, instead of trying to create a configuration language of your own. There’s a lot of constructs that I’m used to having in Python that I sorely miss having available in Terraform. In my opinion there’s no reason to develop your own configuration language since it’s a time consuming task, and I don’t find it better than a general purpose language anyways.

Why did I choose Pulumi?

I have this thing called “Fun time Friday” where I spend the first half day on Fridays exploring new things I’ve heard about. Most of the things I look at during that time end up being scrapped, and there’s zero pressure to push things into production.

However after exploring Pulumi I felt that it might be significantly better for me than Terraform. I decided to take a day to see if I could migrate my infrastructure.

Could I have used something else, like AWS CDK? I most probably could. The nice thing about running a company of one is that there’s no need for analysis paralysis! At this point I believed that Pulumi was a better choice for me than Terraform, so I went for it.

Personally I struggle a bit with CDK being mostly Amazon-centric. I sometimes use my IaC to configure things outside of AWS as well. The other issue for me is that CDK by default uses AWS CloudFormation to provision resources. I’ve been bitten a few times by CF. Some times it’s been because of buggy behavior, e.g. a stack that refused to terminate. Other times it’s been the fact that CF often lags behind in capabilities, so it’s common to have to wait a few months for a new feature to be usable through CF.

Sure, I could use CDK for Terraform to have CDK generate Terraform HCL for me. However, now we’re quickly getting away from the simple system I wanted in the first place.

Apologies to those of you who are screaming “FFS - CDK is the best thing ever!" at your monitors right now. You might be 100% right! From what I’ve seen CDK seems pretty awesome for the most part. I didn’t end up with Pulumi because CDK seemed bad. I ended up with Pulumi because it seemed like a slightly better fit for my needs :).

Pulumi in 3 minutes

You might already know a bit of Pulumi, but here’s a very quick overview plus some advice on where to go from here.

Pulumi allows you to write your IaC using TypeScript, Javascript, Python, Go or C#.
You define resources which have inputs and outputs. As an example, a resource could be an RDS database Instance. One of it’s inputs would be the database version to use. One of the outputs would be the hostname / address of the server.
The outputs are lazily evaluated, and their exact value is often not known until after you’ve executed your code. I.e. you wouldn’t have the db_instance.address until the instance had been created. Think of outputs as promises / futures.
You can use component resources to group your resources into logical groups - similar to how Terraform modules work. If you don’t do this there’s a high risk you’ll end up with a very messy setup further down the line.
Read Pulumi’s own overview and naming docs before attempting to use Pulumi yourself. I’d say that’s the most important bit to know about.
Pulumi are using the open core model where most features are free, but some extra goodies are on a paid tier. You can use the open source version and do state management yourself, but you miss out on certain features.

So what does Pulumi code look like? Below is a minimal extract of some of the Python code I’ve written (slightly altered).

# __main__.py
...
prod_redis_server = RedisServer(
 identifier='redis-prod',
 route53_zone=example_com_route53_zone,
 route53_name='redis-prod.example.com',
 security_groups=[my_ecs_service.task_security_group]
)
...

# components/redis_server.py
class RedisServer(ComponentResource):
 def __init__(
 self,
 identifier: str,
 route53_zone: aws.route53.Zone, # Route53 Zome where the DNS record will be created.
 route53_name: str, # Full DNS name of the CNAME record, e.g. 'redis.example.com'.
 security_groups: List[aws.ec2.SecurityGroup], # Security groups which should be granted access to the server.
 opts=None
 ):
 super().__init__('vt:RedisServer', identifier, None, opts)

 # Prefix with `self.` if you want to access the security group from outside this component.
 security_group = aws.ec2.SecurityGroup(
 identifier,
 ingress=[
 aws.ec2.SecurityGroupIngressArgs(
 from_port=6379,
 to_port=6379,
 protocol='tcp',
 security_groups=[sg.id for sg in security_groups],
 )
 ],
 opts=ResourceOptions(parent=self),
 )

 # Create all the other sub resources needed.
 ...

As you can see it’s really simple and clean for me to set up a Redis server now.

There’s also the fact that I can use the Pulumi Console (web interface) to check the state of my infrastructure really quickly. I found myself doing this instead of logging in to the AWS console or using the AWS cli.

How I went about the migration

After reading the migration guide I immediately tried out what seemed to be the obvious option - tf2pulumi. This is supposed to be able to convert Terraform code directly to Pulumi. In my case it simply errored out and was not able to convert my Terraform code. To be fair my TF code was using slightly more “advanced” features like custom written modules, so it wasn’t the easiest thing to handle for tf2pulumi. I recommend you try it out, because it sounds great in theory. However it wasn’t an option for me, so I had to go for a more manual approach. In retrospect I’m actually quite happy I did, because it gave me the opportunity to organise my code in a way that I preferred.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

Don’t worry though - “manual” imports using Pulumi is still rather easy to perform, albeit a bit time consuming. Here’s how I did it.

I started by testing out the AWS getting started guide to get a feel for how Pulumi worked. This was a great move and I recommend everyone else to do the same.

I was then ready to start importing my existing resources. The general workflow was to look at my existing Terraform code, rewrite it using Pulumi, and then import the existing resources. It was a bit time consuming in the beginning, but I got a lot faster at it after a few hours.

Since Pulumi starts out “empty” you can rewrite and import a piece at a time. I’d recommend doing this because you will most probably do some mistakes initially, and it’s good to get early feedback.

You can check the API reference for details on how to import each specific resource. The instructions on how to do it is usually at the bottom of the reference page for each resource type.

Once I had everything migrated to Pulumi I set up a Gitlab CI/CD flow which allowed me to preview and apply the infrastructure changes. Up until then I’d been doing it locally from my own computer. If possible, I recommend doing it that way. The extra time taken by involving a separate CI/CD pipeline would make your migration take a lot longer. It’s perfectly fine when everything is done and dusted, however when learning Pulumi you’ll often need to use the pulumi command to look at the resources.

Issues, workarounds and general tips

Use component resources to group resources

When your infrastructure starts growing it becomes hard to manage unless you group your resources. Pulumi’s solution to this are the so called component resources. They are “logical components” which you define yourself.

$ pulumi stack
...
─ vt:RedisServer vt-redis-prod
 │ ├─ aws:elasticache/parameterGroup:ParameterGroup vt-redis-prod
 │ ├─ aws:ec2/securityGroup:SecurityGroup vt-redis-prod
 │ └─ aws:elasticache/cluster:Cluster vt-redis-prod
...

As you can see I’ve defined a component called “vt:RedisServer” which I use to group all the resources associated with my Redis server. You’ll get a graph representation of your whole infrastructure in the Pulumi Web Console .

It might seem like a small thing, but it really helps to have a logical grouping of your resources as your infrastructure grows. It also help because you can suddenly decide which resources your component exposes and which ones are “internal” to the component. Check out the code sample from earlier in the article to see how to do this using self..

Importing a resource that has a parent was a bit confusing at first

Each resource in pulumi has a globally unique URN. You can show the URNs for the infrastructure you’ve codeified in pulumi by issuing pulumi stack --show-urns

$ pulumi stack --show-urns
...
├─ vt:RedisServer vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod
 │ ├─ aws:elasticache/parameterGroup:ParameterGroup vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/parameterGroup:ParameterGroup::vt-redis-prod
 │ ├─ aws:ec2/securityGroup:SecurityGroup vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:ec2/securityGroup:SecurityGroup::vt-redis-prod
 │ └─ aws:elasticache/cluster:Cluster vt-redis-prod
 │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/cluster:Cluster::vt-redis-prod
...

Let’s have a look at one of the URNs in more detail, shall we?

urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/parameterGroup:ParameterGroup::vt-redis-prod

stack name: prod
project name: VT
parent resource type: vt:RedisServer
resource type: aws:elasticache/parameterGroup:ParameterGroup
resource name: vt-redis-prod

All URNs need to be unique, which means every resource of the same type, with the same parent (in the same stack and project) needs to have a unique resource name.

Normally I would have written and imported my parameter group definition something like this:

# components/redis_server.py
class RedisServer(ComponentResource):
 """
 Redis server running on ElastiCache (single node).
 """
 def __init__(self, identifier: str, opts=None):
 super().__init__('vt:RedisServer', identifier, None, opts)

 ...

 parameter_group = aws.elasticache.ParameterGroup(
 identifier,
 name=identifier,
 family='redis6.x',
 opts=ResourceOptions(parent=self),
 )

# __main__.py
vt_prod_redis_server = RedisServer(
 identifier='vt-redis-prod'
)

And I would then try to import it like this:

# Executing the import (THIS WON'T WORK, DON'T TRY IT)
pulumi import\
 aws:elasticache/parameterGroup:ParameterGroup\
 vt-redis-prod\
 vt-redis-prod

The first vt-redis-prod in the command above is the identifier used by the aws.elasticache.ParameterGroup. This is actually called “resource name” using Pulumi-speak. I prefer to call it identifier in my own code though, because “resource name” is too similar to the actual name it has to AWS for my liking.

The second vt-redis-prod is the actual name of my parameter group on AWS (remember we’re importing existing stuff).

However, the above won’t work for one simple reason - the URN won’t match because we haven’t told Pulumi that this parameter group resource we’re importing is a child of our RedisServer component. What’s the solution? Pretty simple, but somewhat obscure at the same time:

# Executing the import (the right way)
pulumi import\
 --parent vt_prod_redis_server=urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod\
 aws:elasticache/parameterGroup:ParameterGroup\
 vt-redis-prod\
 vt-redis-prod

You tell Pulumi about the parent by using the --parent argument (this is the simple part). The parent argument is a key=value pair, and the key is actually the python variable name you’ve used in your code. If you scroll back up you can see that I stored my RedisServer instance in a variable called vt_prod_redis_server. This part took longer than I’d like to admit for me to figure out.

The value is the URN of the parent component, which you can get with pulumi stack --show-urns.

You can’t import a child without having the parent in place first

That last one was a bit heavy, let’s take a breather and go for something a bit simpler!

When importing a child component you need to import the parent first. If you simply run pulumi up it will try to import all your resources all at once, and it won’t work because the parent haven’t been created yet.

Instead simply do this first to update just a single resource:

$ pulumi up --target urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod

This will cause my vt-redis-prod resource to be created in the Pulumi state. Nothing actually happens on AWS since it’s just a logical component. Once this is done you’ll be able to import all of it’s child resources as you saw above.

Writing IAM (JSON) policies with Pulumi outputs

Imagine having an IAM policy that referenced an ECR repository, just as an example. You might want the policy to grant access only to that particular ECR repository. You might be temped to try something like this:

aws.iam.Policy(
 ...
 policy=json.dumps({
 'Version': '2012-10-17',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Action': [
 'ecr:GetAuthorizationToken',
 ...
 ],
 'Resource': ecr_repository.arn
 }
 ]
 }))

While this might look sensible, it won’t quite work. Remember early on I sad that Pulumi outputs are futures, and not variables whose values you can access just like that. If you did like the example above your policy would just end up with something similar to 'Resource': <Pulumi.Output(...)> in it. Not quite what we wanted.

Instead we have to tell Pulumi to resolve these futures, so we get the actual value for them, like this:

aws.iam.Policy(
 ...
 policy=ecr_repository.arn.apply(lambda arn: json.dumps({
 'Version': '2012-10-17',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Action': [
 'ecr:GetAuthorizationToken',
 ...
 ],
 'Resource': arn
 }
 ]
 }))

If you have more than one Output you need to access you can use Output.all, like this:

policy=(
 Output.all(origin_access_identity.iam_arn, self.s3_bucket.arn)
 .apply(
 lambda args: json.dumps({
 'Version': '2008-10-17',
 'Id': 'PolicyForCloudFrontPrivateContent',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Principal': {
 'AWS': args[0]
 },
 'Action': 's3:GetObject',
 'Resource': f'{args[1]}/*'
 }
 ]
 })
 )
)

The .apply method can also be useful for troubleshooting when you need to access the particular value of an output.

This might seem tricky at first, but digest this for a bit, then have a look at the docs for Inputs and Outputs and I’m sure it’ll all make sense.

Don’t name resources unless you really have to

Pulumi is pretty smart when it comes to naming the actual resources on AWS. If you don’t specify a name when creating a resource Pulumi will use the identifier and append a hex character suffix .

This gives two great advantages. The first is that you can have multiple stacks next to each other, e.g. a “review” and a “production” environment. If you hard coded the names this wouldn’t be possible.

The second advantage is that it makes replacement of resources much easier. If you hard code the name you leave Pulumi no choice but to tear down the existing resource before creating the replacement one. This makes a big difference if you are to perform a version update of one of your dependencies. In case you allow Pulumi to handle the naming it can create the new resource before terminating the previous one. This coupled with a DNS entry update can make for updates with virtually zero downtime.

Since I imported existing resources I had to keep hard coded names in a few places. However I’ve taken the time to get rid of the hard coding as much as possible since then.

Integrating with CI/CD (Gitlab)

I use Gitlab for my Ci/CD, and I’m really happy with it. Let me show you what my very simple Pulumi pipeline looks like right now:

stages:
 - build

preview:
 stage: build
 image:
 name: pulumi/pulumi-python:3.2.1  # Choose the right image for your language
 entrypoint: [""] # This is important for things to work
 script:
 - pulumi stack select prod
 - pulumi preview --color always --diff  # I added '--diff' to always see more details
 only:
 - merge_requests  # Only run the preview on merge requests

update:
 stage: build
 image:
 name: pulumi/pulumi-python:3.2.1
 entrypoint: [""]
 script:
 - pulumi stack select prod
 - pulumi update --color always --diff --yes  # Answer "yes" when asking to apply the changes
 only:
 - master  # Only run the update on the master branch (post merge)

As you can see my pipeline is very simple for this little project which is still in an Alpha phase. There’s just the “production” environment, and changes are automatically applied once the MR is merged.

You’ll have to make use of your gitlab-fu (or other-ci-cd-fu) to adapt this to your own needs. Read the docs from Pulumi to get started. It’s usually very simple and straight forward!

I did things quite differently from what’s described in the Pulumi docs. I used the ready made image from Docker Hub, while they build their own. I did this to make the whole pipeline quicker. It seems to work really well so far, and I haven’t found any downsides to my approach (yet).

I couldn’t get the Gitlab integration webhooks to work

You are normally able to have Pulumi post a report of the planned changes attached to your merge request .

However I couldn’t make this work because of authentication issues. With the help of their support (which was excellent btw) we figured out that it was related to the fact that I was using Gitlab groups to organise my repositories.

As things stand right now you have to create a Pulumi Organization for each Gitlab Group. There’s a bit of an issue with Pulumi’s pricing right now, because the only way to be able to get more than 1 organisation is to go for their enterprise plan.

I think Pulumi will be fixing this in one way or another in the future. For now I’m still pretty happy just checking my changes from the Gitlab job output.

What’s the verdict?

So I’ve been using Pulumi for a few weeks now, and here are my impressions.

It’s really fast compared to Terraform. I didn’t even expect this one, but for some reason Pulumi manages to at least “feel” a lot faster. I haven’t done any measurements to verify this, but it definitely feels much faster.

Their documentation is really good, although it took me a bit to learn to navigate it.

Having type annotations is great! This makes it a lot more friendly to work with as a developer. No more passing around just plain strings in your code. However the type annotations doesn’t quite work in PyCharm because of how the Pulumi code is written. I consider this a PyCharm bug though, and I expect it to be fixed in the future. For now I can simply ctrl+b the class in question, and read the type annotations from the source code.

The Pulumi (Web) Console is a neat way to look at your infrastructure. I know there are similar solutions for Terraform, but I hadn’t used them personally.

The developer ergonomics are great. Using your language of choice helps a lot. When running a company of one, like myself, it’s really important to limit the number of things you need to learn as much as possible. While HCL is simple it’s yet another language to remember. If you’re working at a larger company it’s a great way to encourage developers to do more infrastructure work.

Pulumi is an open core type product, and you will have to pay to access the features on the higher tiers. Their current plans are very fairly priced. I myself is using the free community plan, which works great for me. There’s obviously the risk that they will remove the free plan in the future. However even if that happens I could either pay the fee, or transfer to state storage to AWS S3, and retain most of the product functionality. This feels like an acceptable level of risk for me.

My goal was to do the whole migration in one day. In reality it ended up being closer to two days. However, given that I learned Pulumi and migrated ~130 resources in that amount of time I’d say that Pulumi is quite easy to get started with.

All in all, I’m very happy with making the transition to Pulumi! I became way more productive managing my infrastructure. At the same time I ended up enjoying the work of doing so a lot more - which is really important too.

This article was originally posted at https://blog.ekik.org/my-experience-migrating-my-infrastructure-from-terraform-to-pulumi. Reposted here with permission.

2021 December Hackathon: Introduction

Laura Santamaria — Fri, 07 Jan 2022 09:39:28 -0600

Pulumi’s hackathon tradition continued in the last weeks of 2021 with our 2021 December hackathon. For one solid week, we had teams from across the company focus on improvements across the Pulumi ecosystem, and we brought in people from outside the engineering org to get perspectives on different needs. While there were some projects that were focused on internal work, there were still quite a few open-source projects that we can talk about publicly. We’ll get more details from some of those teams over a few more posts. In this post, however, we’re going to explore a bit about how we worked.

Setting up for success

We set up the teams for success before the hackathon began by starting discussions around project ideas in a Slack channel asynchronously. In in-person hackathons, teams often decide on projects the day of the start of the hackathon. By starting discussions early, we let folks decide on projects they’d be interested in contributing to and drove early architectural discussions so teams could hit the ground running. People also started to get excited about working on the various projects, and they were able to start connecting with folks outside of the engineering team to find time on their calendars early.

Using a survey, we gathered what sorts of projects or areas of the ecosystem people were interested in working on, their timezone, and how they would like to collaborate. By using this data to sort people into teams, we ensured that people who were interested in learning about new areas of the ecosystem could find the right projects, that people who needed to work in a more synchronous environment could be paired with others in their timezone, and that people who worked similarly could find people who worked like them. It was an interesting experiment.

Working remotely

One cool thing that we tracked this round is all of the different ways we collaborated across the different projects. In general, Pulumi is a remote workplace, and none of the teams were co-located for this hackathon. As a result, we had to consider how teams might work across timezones and outside of the classic physical co-located team dynamic. Each team got to pick the best ways for them to collaborate. Our teams’ preferences ranged from daily standups over Zoom to collaboration solely on Slack to extensive pair programming and Zoom hangouts. In all, each team used multiple ways to communicate and engage depending on which parts of their projects they were working on. Teams working on explorations of possible solutions often found that pair programming remotely was a great way to brainstorm, for example. Once teams had a solid sense of what needed to be done to get to the next part of a project, many still attended a hackathon-wide Zoom hangout session daily to have someone to hang out with and discuss any issues that arose. The whole org also had the opportunity to get to know one another a little better, and it did not have the feeling of lost time or forced socialization that weekly virtual watercoolers can have after a couple of meetings as many have found during this period of forced virtual interactions.

Showing off what we did

During the demo at the end of the hackathon, we asked teams to self-report which strategies they found the most helpful, what they learned, and obviously what the results of their hackathon project were. We found that everyone level-set to how their team chose to work, and the pairing of people based on location and work preferences helped a lot to have teams succeed.

We’ll have more from each team soon! Keep watch on this space for new posts about hackathon projects.

If you like our way of working and are interested in a new position, we have a lot of positions open right now and will have more in the future! Please head over to our careers page to find out how to apply.

Kubecon NA 2021 Roundup

Kat Cosgrove — Mon, 08 Nov 2021 22:50:38 +0000

KubeCon North America 2021 is over, but the recordings are now online! Every talk you wanted to attend and couldn’t is available on YouTube, so here’s some highlights—cloud native trends, updates from projects and SIGs, and a few of my favorite talks!

SBOM is Coming: Why You Should Care and How You Can Help - Frederick Kautz & Allan Friedman

A major trend throughout the conference was supply chain security and software bill of materials (SBOMs). There were quite a few talks on these subjects, but Frederick Kautz and Allan Friedman give a great overview of exactly what an SBOM is, and why it’s important.

Notary: State of the Container Supply Chain - Justin Cormack, Docker & Steve Lasker, Microsoft

On supply chain security, Steve Lasker presents updates to Notary v2. It’s changed quite a bit since v1 and there are more changes on the horizon. With the rising concerns surrounding supply chain security, I expect Notary and tools like it to be increasingly important. No worries if you aren’t already familiar with Notary—Steve explains it well.

A Vulnerable Tale About Burnout - Julia Simon, CloudOps

Taking things to a more human place, burnout was a common theme this year. It’s something that we don’t talk about as much as we should, but close to two years of isolation is forcing us all to confront it in ways we may not have expected. This talk by Julia Simon is important to watch not just so that you can learn to recognize these problems in yourself, but so that you can recognize them in your peers.

PodSecurityPolicy Replacement: Past, Present, and Future - Tabitha Sable & Tim Allclair

PodSecurityPolicy (PSP) has been deprecated. Many of you may be relying on this, and deprecations frequently sound scary, but Tabitha Sable and Tim Allclair are here to soothe your fears. They go to great lengths explaining what PSP is, why it was deprecated, what exactly this means for people relying on it, and where we’re going next.

SIG Contributor Experience Deep Dive - Alison Dowdney, Weaveworks & Christoph Blecker, Red Hat

An open source project lives or dies by its contributors, both technical and non-technical. Kubernetes is huge and requires a lot of contributors to keep it growing, so we have SIG Contributor Experience to make sure they stay happy and the community stays healthy. Alison Dowdney and Christoph Blecker go into just how SIG Contributor Experience operates, plus updates. If you’re in a leadership position for an open source project, there are some lessons to be learned from this talk.

Kubernetes Exposed! Seven of Nine Hidden Secrets That Will Give You… Ian Coldwater & Brad Geesaman

No Kubecon is complete without Ian Coldwater, this time joined by Brad Geesaman (and, briefly, a second Ian Coldwater) to go into some not-so-obvious but oh-so-important pitfalls and hidden security concerns within Kubernetes. Even if you caught this talk live, it’s worth watching again to make sure you’re secured against these issues, or you might find a goose haunting your cluster.

Beyond Kubernetes Security - Ellen Körbes, Tilt & Tabitha Sable, Datadog

Finally, it would be remiss of me to not mention Ellen and Tabitha’s cinematic contribution to this year’s Kubecon North America. Beyond Kubernetes Security is the spiritual successor to their talk from the last Kubecon Europe, once again going into how some security concepts and exploits work in Kubernetes, but presented as a short film involving crime, time travel, breakups, and Ellen’s collection of old electronics.

The quality of the content was incredible across the board this year, whether the speakers were live or presenting virtually, forcing some difficult decisions around selecting my schedule for talks to attend and how many I could feasibly put in this blog before it got way too long. Fortunately, all of them are on the CNCF YouTube channel now, including talks from the various co-located events. Dig through!

Kubecon North America was simultaneously a strange and familiar event. It’s been nearly two years since most of us have been at a conference, and while that certainly made things awkward at times while we all readjusted to seeing other humans in the flesh, our slow return to something vaguely familiar seems to have been a success. I sincerely hope to see you all in person at the next Kubecon EU in Valencia.

Honk <3,

Kat

The Industry Comes Together for the Cloud Engineering Summit

Isaac Harris — Wed, 06 Oct 2021 06:00:00 -0700

When we launched the first-ever Cloud Engineering Summit last year, we were humbled by the generosity and support from partners and sponsors across the industry who helped to make the summit a success. For this year’s event, we’re equally thrilled to have an amazing group of sponsors participating in the summit and it’s worth a quick ‘guided tour’ of the teams that are helping us to pull off this amazing event.

Amazon Web Services

AWS was an inaugural sponsor for the Cloud Engineering Summit with a talk from Alex Head about cost optimization. The AWS team returns again this year with four sessions: Emily Freeman will talk about Rethinking the SDLC, Rahul Sharma will share his team’s experience building the brand new Cloud Control API in How AWS architects APIs for hyper-scale, and Jenna Pederson will show you how to Bring software development practices to your infrastructure. Finally, Ken Exner will be chatting with Joe Duffy about the future of tooling for modern cloud infrastructure.

We’ll cap off the summit in a free, post-event workshop with Marina Novikova and Laura Santamaria showing you how to get started on AWS.

Apple

New to the Cloud Engineering Summit this year, cloud services experts Fariba Khan and Stephen Van Gordon will share their experiences managing infrastructure at scale in Lifecycle of a Pulumi Program. They’ll talk about building a custom state backend, using CI/CD tooling, and the Pulumi Automation API to empower teams to provision secure-by-default configurations automatically.

Google Cloud

Last year we hosted a fantastic talk from Jason (Jay) Smith who showed attendees how to stand up a serverless platform. This year, we’re proud to include Google Cloud as a sponsor and feature a keynote from Kelsey Hightower with Joe Duffy.

After the summit, you can also join Kat Cosgrove for free hands-on training showing you how to get started with Google Cloud.

Microsoft

The 2020 Cloud Engineering Summit featured Amanda Silver in a Future of Cloud Engineering panel and Adora Nwodo showed everyone how to manage apps and infrastructure in a single CI/CD pipeline. Adora will return to the summit again this year to talk about Software engineering in infrastructure engineering.

For those looking to expand their cloud engineering skillset after the summit, Matt Stratton will host a workshop to show attendees how to get started with Azure.

NGINX

We’ve spent a lot of time with the NGINX team over the past year in their quest to make it easier than ever to get to production with Kubernetes. Join first-time summit speakers Damian Curry and Elijah Zupancic for a lightning talk followed by NGINX office hours to learn about Building a modern app reference architecture for Kubernetes.

Then, if you’re ready for some hands-on Kubernetes help, join our post-event workshop From Zero to Production in Kubernetes.

Bridgecrew

New to the summit this year, Bridgecrew’s Steve Giguere is joining the Build track to share Taking a Kubernetes Deployment from Default to Secure.

CircleCI

Long-time collaborator Angel Rivera from CircleCI joins the Deploy track to talk about Building Arm Compatible CI/CD Pipelines and show you how to take advantage of Arm architectures in both your pipelines and workloads.

GitHub

The GitHub team participated in three sessions last year including the Architectures and Platforms and Security and Policy panels. This year, the team returns with Rizel Bobb-Semple delivering a plenary session showing you how to get From Code to Cloud without learning new tools.

Honeycomb

If you missed Charity Major’s summit session last year about Testing In Production, stop reading and go watch it. Then, register for Liz Fong-Jones’s Build track session this year: What is OpenTelemetry.

NS1

Last year, Johnathan Sullivan joined the Culture, Process, and Tools panel, and we’re excited to welcome Chris Buijs to this year’s Deploy panel.

…and many more

Beyond the teams and sessions mentioned above, the Cloud Engineering Summit wouldn’t be possible without help and support from a long list of sponsors and collaborators including Codefresh, Split, and Uffizzi. Watch the replay to see what all the fuss is about.

Cloud Engineering Summit Build Track

Kat Cosgrove — Thu, 16 Sep 2021 13:47:25 -0500

The Cloud Engineering Summit 2021 is coming up fast, and the speakers are out! To get you ready to attend, let’s take a look at the sessions for the Build track.

The Cloud Engineering Summit’s three tracks are built around three concepts: Build, Manage, and Deploy. I’m Kat Cosgrove, and I was responsible for selecting your speakers for the Build track! For us, that means building cloud applications and infrastructure with Modern Infrastructure as Code using general purpose programming languages. We embrace the fact that modern cloud applications have blurred the lines between the application and the infrastructure, and that success requires at least some level of proficiency in both. Whether you’re full stack or lean more towards one area, all cloud engineers apply a software engineering mindset and practices to building and testing applications and the underlying cloud infrastructure. This includes using standard programming languages, applying software principles such as reusability and abstractions and testing, and leveraging the rich ecosystem of software development tools.

Without further ado, let’s take a look at each of the talks I’ve selected for you!

Starting us off is the inimitable Kris Nóva with Infrastructure as Software. She’s going to be talking about why Turing complete config management is an antipattern, and why we should be putting the Dev back in DevOps. Nóva is an incredible speaker and engineer. Any talk she gives promises to be hot fire, so I wouldn’t miss this one.

Don’t like repeating yourself? Yeah, me neither. Fortunately, we have Sam Cogan to present his talk, Reuse, Don’t Repeat - Creating an Infrastructure as Code Module Library. Personally, I’m a huge fan of reusable modules for any purpose, when they’re built in such a way that it reduces overall toil and makes your codebase easier to understand. This talk should be useful for everyone.

Helping us ask and answer the right questions to make organizational and technological decisions is Joep Piscaer, presenting Kubernetes isn’t the answer, but what was the question again? I love Kubernetes, but it can be complex, and sometimes that complexity means it’s not the right solution for you (yet.) If you struggle with finding the right questions to ask about your tech needs, this talk could help.

I’m kind of nosy. I like knowing as much as possible about what’s going on inside of my applications and infrastructure. It’s hard to present all of that data in a way that’s easy to consume rather than overwhelming, though. Fortunately, we have Liz Fong-Jones to help out with her talk, What is OpenTelemetry? Don’t miss this one if you’re curious about a better way to gain insights into your distributed systems.

You’d think that when a particular tool is widely accepted as the industry standard for accomplishing a task, there’d be a wealth of information out there on how to accomplish most important tasks in an accessible, actionable way. Unfortunately, that’s not really the case. Fortunately, we have Steve Giguere presenting Hands-on: Taking a K8s Deployment from Default to Secure, wherein he’ll take us through securing an NGINX deployment!

Curious about how a big product gets built? Rahul Sharma is with us to talk about How AWS architects APIs for hyper scale. He won’t just be talking technology; we’ll also hear about the cultural changes required internally to launch a product, and how engagement with customers and partners helps shape the course of development.

Last but certainly not least is Heidi Waterhouse presenting The Future Should Be Uneven, a talk about personalization and accessibility in web design. Accessible design benefits us all, whether we actually need the accommodation or not. The sooner we all understand and embrace that, the better, so Heidi will walk us through some common accessibility settings that are really just good design for everyone.

Explore all of the rest of the great sessions, and don’t forget to register for the cloud engineering summit!

Preview of the Deploy Track at Cloud Engineering Summit 2021

Laura Santamaria — Wed, 15 Sep 2021 09:00:00 -0500

Cloud Engineering Summit 2021 is almost here! We’ve got a great line up this year.

Our tracks are built around the three pillars of cloud engineering: Build, Deploy, and Manage. I’m your track chair for the Deploy track, the track focused on automating and managing infrastructure. Deploy is all about unifying systems so everything in a cloud-based system is shipped together from the same automated, auditable process, reducing human error and improving quality across the board. That could mean focusing on automated testing and linting of infrastructure as code, providing shared services platforms for others to use in their pipelines, or exploring how unified and automated infrastructure changes engineering culture in an organization.

In no particular order, let’s go explore the Deploy track lineup.

Trying to get a handle on way too many environments across your pipelines? Adora Nwodo, Software Engineer at Microsoft, will explore bringing standard software engineering practices to solve the problem of maintaining many environments and configurations in a repeatable fashion. Adora has a very accessible way of explaining complex concepts in her blog and videos (find her at AdoraHacks), and I’m really looking forward to getting her perspective.

How can we improve the security of these unified processes? Alyssa Miller, Business Information Security Officer (BISO) at S&P Global Ratings, will discuss how threat modeling can improve CI/CD pipelines and general security. I’m excited to have Alyssa join us to help us understand how security should be part of the Deploy pillar; her Pitmaster’s Guide to Security is really good and I recommend watching it!

When you’re trying to share what you’re doing, you could share your code, but how do you share your infrastructure? Peter McKee, Head of Developer Relations and Advocacy at Docker, will demonstrate how to build into pipelines the ability to share features and changes in applications with others as a continuous preview. Peter’s a fantastic teacher who always has a great hands-on presentation in his back pocket, so I’m really looking forward to finding out what he has in store for us.

Data builds up while you’re building out, deploying, and managing cloud-based systems. How can you manage data while deploying your cloud infrastructure? Andy Dang, Co-Founder and Engineering Lead at WhyLabs, will talk about data integrity, observability, and monitoring with an open-source project called whylogs. Always love having a preview of an open-source project in the program, and I think data’s going to continue to be a big question for cloud engineering, so Andy’s talk was very welcome!

What about ARM-based systems? Apple certainly got into the game with their M1 chip, so it’s a platform you have to consider when doing deployments. Angel Rivera, Senior Developer Advocate at CircleCI, will explore how to make pipelines and workloads ARM compatible and capable of supporting ARM on various products. ARM-based systems are some of my favorites (yay Arduino and Raspberry Pi!), and they’re also really important considerations if you’re in the mobile or embedded space, so I can’t wait to hear Angel’s take.

How about some focus on doing all of this at scale? Jenna Pederson, Developer Advocate at AWS, will explore how bringing all of the best practices from software development to pipelines will help make working with cloud-based resources consistent and secure. I’m so happy Jenna could join us! Her content is wonderfully accessible and really explains technical concepts and complex systems clearly for all levels (which is my jam!). You don’t want to miss her talk!

Finally, how does all of this stuff work in a real life team? Evan Bradley, Software Engineer, and Tom Carrio, Senior Software Engineer, from Dynatrace are going to take us through bringing it all together with a real-life team, discussing the experiences they’ve had over time as they’ve implemented everything. I’m really excited to have folks bring their stories of what it’s all like so we can share pitfalls and solutions to move forward together. I think you’ll enjoy Evan and Tom’s talk, too!

Explore all the sessions, and don’t forget to register for Cloud Engineering Summit. I can’t wait to chat with you virtually at Cloud Engineering Summit 2021!

Preview of the Manage Track at Cloud Engineering Summit 2021

Matty Stratton — Mon, 13 Sep 2021 08:00:00 -0500

The Cloud Engineering Summit 2021 is coming up fast, and the speakers are out! To get you ready to attend, let’s take a look at the sessions for the Manage track.

The Cloud Engineering Summit’s three tracks are built around three concepts: Build, Manage, and Deploy. I’m Matt Stratton, and I’m your charismatic track chair for Manage. For us, that means managing cloud applications and infrastructure with Policy as Code, visibility, and access controls. For example, managing infrastructure with policies that detect configuration drift, enforce best practices, and even prevent compliance violations before deployment. It means building visibility across your cloud infrastructure so that you always understand its current and past states, including detailed audit history. Finally, you ensure the right guardrails and controls are set in place so that distributed teams can securely develop.

Without further ado, let’s take a look at each of the talks I’ve selected for you!

Overwhelmed by permissions and authorization? Me, too. Fortunately, we have Joy Ebertz presenting Authorization: Ensuring Only Ada Can Access Her Files. Joy is going to cover the various authorization models available to us, with use-cases for each. Authorization can be overwhelming if you don’t have a good grasp of the options available to your organization and an understanding of what works when, so I’m excited for this one.

Taking a higher-level view, Quintessence Anx will be presenting DevSecOps and Secure Incident Response. The introduction of security to the DevOps equation means some shifts in the way we think about the software development lifecycle, and Quintessence is uniquely qualified to talk about why that’s true. A look inside PagerDuty’s 14 Step Secure Incident Response process promises to be interesting for anyone involved in either development or operations.

I don’t think we have the ability to predict the future, but J. Paul Reed wonders if we can develop the ability to forecast incidents just as we have learned to have staggeringly accurate weather short-term forecasts. With Forecasting the Future: Creating a Radar for Risk, Paul will share with us how Netflix uses Risk Radar to collect and make sense of risk in their sociotechnical systems.

When we talk about infrastructure as code, there’s more to it than just the code! Fariba Khan and Stephen Van Gordon will take us on a journey with Lifecycle of a Pulumi Program to share how they leverage a custom state backend with SSO, RBAC, and programmatically configurable pipelines powered by CI/CD tooling and the Pulumi Automation API to drive IaC at their organization.

Snowflake’s Charles Xu will be here to talk about Day-two Operation of Multi-cloud Kubernetes and Vault, a very cool overview of Snowflake’s Kubernetes infrastructure and how Pulumi helps, from blue-green updates, to managing Hashicorp Vault, to rotating TLS certificates. This is a pretty complex topic, so I’m very interested to see where it goes!

Gremlin’s Jason Yee comes to us with Trailblazers: exploration, discovery, & navigating failure - a set of topics that I personally find fascinating and essential! I think we are constantly in a state of discovery and understanding of our complex systems, and learning how to explore them - and work through the inevitable failures - helps us build and operate even more reliable systems.

Rounding things out is the always incredible Tim Banks, with Toxic Velocity: Speed Kills. In this talk, Tim is going to cover the various stressors that put pressure on both organizations and individuals, how they affect us, and what it looks like when things start to spiral out of control. I’m excited for this one because while crashes do happen, we’d all like them to happen less often (or not at all!), and being able to identify some of these things before they become a real problem is better for our businesses and our people.

Explore all of the rest of the great sessions, and don’t forget to register for the cloud engineering summit!

Demos from the Multi-Language Component Hackathon

Lee Zen — Thu, 24 Jun 2021 00:00:00 +0000

At Pulumi, we have a tradition of hosting hackathons every so often to play with concepts and ideas that we may not typically encounter in our day-to-day product building activities. This past week, we’ve had two separate back-to-back hackathons. Our first hackathon, which was open to the community, focused on using our new multi-language component capabilities. Multi-language components allow developers to author reusable infrastructure abstractions in one language and make them available to others in all the languages that Pulumi supports. We were really excited to see what everyone would build!

We had a kickoff meeting on Tuesday morning where Lee Briggs walked participants through the basic steps for bringing up a multi-language component. If you’re interested in seeing the kickoff meeting, we’ve recorded it and have made it available for viewing on Pulumi TV. After the walk-through, teams got down to business!

We had some great ideas, including but not limited to components for a simplified Google Cloud project setup, an Azure web app service container, and an Amazon SageMaker model with a frontend for serving it.

As part of the hackathon, we received some great feedback from all the participants on improving the multi-language authoring experience, including reducing boilerplate code, ways to improve on the schema authoring experience, and documentation suggestions. We’re keen to continue to refine the experience.

Thank you to all the participants! If you’d like to see some of the components in action, you can watch our demo session. We saw the following component demos:

“Run my d#!n container” which demonstrated a component to run a container on any of the major cloud providers.
A component that brings up a new Google Cloud Project.
A component for creating an AWS Lambda function fronted by API Gateway
A component for creating an Amazon CloudFront distribution backed by an S3 bucket
A component that runs an AWS Lambda function as a cron.
API Gateway component that simplifies creating Amazon API Gateway resources
A serverless component that consumes Serverless configuration
Replicated S3 Bucket component to simplify cross-region object replication
Miniflux component that deploys the Miniflux service
.NET Boilerplate for multi-language components

Stay tuned for our next post on our general hackathon results. In our general hackathon, the Pulumi team tried out new ideas and investigated ways to improve upon the Pulumi experience!

You can watch the hackathon including the intro multi-lang components, the hackathon kickoff, and the demo session.

Introduction to Building Multi-lang Components

Multi-lang Hackathon Kick-Off

Hackathon Demos!

Hitesh's Intern Experience at Pulumi

Hitesh Boinpally — Fri, 11 Jun 2021 09:53:45 -0700

Hi everyone, I’m Hitesh Boinpally, a junior studying Computer Science at the University of Washington. I was offered the opportunity to intern for Pulumi over the past three months, and here’s how it looked!

Technical Work

My overall project was to build a test harness for Pulumi’s converter tools such as arm2pulumi and tf2pulumi. The project involved several pieces, all of which meant learning about technologies I hadn’t encountered before.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

A program that ran the test itself and generated results in a program readable way, which I wrote in Go.
Running that program to generate historical data over time through GitHub Actions consistently.
The cloud infrastructure part, which was uploading the test results into an AWS Redshift table for human-accessible analysis. Here was where I got to utilize Pulumi and better understand infrastructure as code.

All of these were fascinating to learn about and stretched my knowledge in directions I hadn’t previously explored.

I began with setting up the harness for arm2pulumi. The Go program had already been written, so I needed to focus on the GitHub Actions for CI and cloud infrastructure aspects. I set up the CI quickly, but the cloud infrastructure took some time. Wrangling with AWS’s different IAM roles, cross-account permissions, and how to utilize the different tools was complex and challenging. However, I learned a lot about these different spaces and gained valuable experience. Further, I appreciated the advantages of Pulumi, as I could quickly undo/redo changes incrementally, rather than making manual changes in the AWS Console.

Once I built the arm2pulumi coverage tracker end-to-end, I shifted to tf2pulumi. Unlike arm2pulumi, there was no code to inherit, and I had to write out the entire workflow from scratch. The Go program was the real challenge, as the CI work and cloud infrastructure I had written was reusable. I got a much better hang of Go as a language and the advantages that came with it. I also got to leverage Pulumi’s Automation API and the power of having multiple, configurable stacks through Pulumi to significantly simplify the process of deploying tf2pulumi’s (and any other converter’s) coverage report cloud infrastructure. Overall, I explored various new technologies, many of which I’ve used in personal projects since, and plan to continue to do so.

Day to Day Work

Due to the ongoing pandemic, my internship was done fully remote. Having come from 4 quarters of remote learning in college, I wasn’t particularly ecstatic about more remote work. Pulumi completely exceeded my expectations, though! It was hard to meet new people, and I missed out on the daily interactions you might expect in a traditional office environment. Regardless, efforts were made to overcome that, and they were pretty successful, with biweekly random 1:1s and fun team events to better integrate with the team. On the other hand, I work better at night and am a bit of a late sleeper, so I could adjust my schedule to my strengths more than I would if I were going into an office each day. Additionally, Pulumi has adapted nicely to the remote environment and made my experience much better than I thought possible.

The team itself was great, with everyone being super friendly and supportive. It took me longer than I expected to get the confidence to ask questions in general Slack channels, but each time I did, I was met with a helpful response and got the assistance I needed to succeed. Despite being the only intern, I never felt cast aside and was guided throughout my project in a balanced way, still allowing for some independence. Everyone is also so knowledgeable that I would learn something new perusing Slack every day, whether internal to Pulumi or an article about some new way of doing things.

Thanks to the high level of transparency at Pulumi, I also got to see many different levels of the company, rather than just the work that I was doing. As a result, I learned more about how a company works fundamentally, and a typical development process.

Closing Thoughts

Pulumi was my first ever internship, and I couldn’t have asked for a better experience. I learned about a ton of new technologies, some of which I may not have explored yet but at least am aware of for future projects. In addition, I met some truly incredibly talented, hardworking, and interesting people. I’m stoked to continue using Pulumi and am excited to see how the company evolves and grows!

Thank Yous

First off, a huge shout out to the leadership at Pulumi for offering this opportunity to begin with! Thanks to Lee, especially for organizing the internship and your help throughout it!

Thank you to Levi for being an awesome, approachable mentor and always willing to help out! I asked a ton of – oftentimes pretty random – questions, and you helpfully answered each of them.

Thank you to Vivek for guiding me through many technical details and approving my PRs. Your guidance was really useful, and I appreciate it a lot!

Lastly, thanks to all those who helped me out along the way and the rest of the team for building such a friendly, helpful environment!

Testable IAM Policy Documents

Thierry de Pauw — Wed, 12 May 2021 00:00:00 +0000

I was relieved to find Pulumi. Finally, we have testable Infrastructure as Code. We can write fast unit tests that we can execute locally without needing the cloud. However, I was a bit disappointed. Pulumi does not have a full representation of IAM Policy documents. Fortunately, it was relatively easy to build a library that did this!

Policy documents are assigned using JSON objects that should follow the AWS IAM JSON Policy syntax.

const policy = new aws.iam.Policy("policy", {
 description: "My test policy",
 policy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [{
 Action: ["ec2:Describe*"],
 Effect: "Allow",
 Resource: "*",
 }],
 }),
});

However, it is perfectly possible to pass an invalid IAM Policy document because there is no validation. You will only notice if it is invalid the minute the policy is applied in the AWS cloud. That creates an unreasonably long feedback loop, incurring a significant amount of waiting and time to correct.

To avoid this, I prefer to write my policies as Policy as Code. It avoids common syntax errors, reduces the feedback cycle, and increases your delivery throughput.

Having to pass a JSON as a policy document did not feel optimal.

I work in the financial industry, and compliance is important. So, I searched for something that allowed me to easily unit test IAM Policy documents, preferably at the Statement level,to help us adhere to security requirements.

Before reinventing the wheel, I searched for existing packages in JavaScript for manipulating IAM Policy documents.

Pulumi has the aws.iam.getPolicyDocument API. That looked interesting because it allows writing the policies as Policy as Code. But you cannot properly unit test the IAM Policy document produced by aws.iam.getPolicyDocument function. When Pulumi runs in testing mode, that function is not available unless you mock it. Huh. That is not very helpful.

I dug further to find Node.js packages for manipulating IAM Policy documents. Not much. Except for AWS CDK. But you must drag the whole CDK Node.js package into your project only to handle IAM Policy documents. But, the AWS CDK was a good starting point for designing @thinkinglabs/aws-iam-policy.

A simple identity-based policy

Let us look at the code sample on pulumi.com for aws.iam.Policy.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const policy = new aws.iam.Policy("policy", {
 description: "My test policy",
 policy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [{
 Action: ["ec2:Describe*"],
 Effect: "Allow",
 Resource: "*",
 }],
 }),
});

Using @thinkinglabs/aws-iam-policy, that would look as follows.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import {PolicyDocument, Statement} from "@thinkinglabs/aws-iam-policy";

export const policy = new aws.iam.Policy("policy", {
 description: "My test policy",
 policy: grantEC2Describe(),
});

function grantEC2Describe() {
 return new PolicyDocument([
 new Statement({
 effect: "Allow",
 actions: ["ec2:Describe*"],
 resources: ["*"],
 }),
 ]).json
}

To test if the IAM Policy is a valid identity-based policy, we can use PolicyDocument.validateForIdentityPolicy(), which returns an array of string error messages. If it returns an empty array, the IAM Policy is valid.

import {expect} from "chai";
import "./mocks";
import * as pulumi from '@pulumi/pulumi';
import {PolicyDocument, Statement} from '@thinkinglabs/aws-iam-policy';

import * as sut from "../src/index";

const get = <T>(output: pulumi.Output<T> | undefined): Promise<T | undefined> | undefined =>
 output ? (output as any).promise() as Promise<T> : undefined;

describe("IAM Policy", function() {

 it('should be a valid identity-based policy', async () => {
 const doc = await get(sut.policy.policy) as string
 const policy = PolicyDocument.fromJson(doc);

 expect(policy.validateForIdentityPolicy()).to.be.empty;
 });
});

A more complex resource-based policy

As a regulated industry, we are required to closely control who has access to what. What scares us most is to inadvertently grant a right to someone that could result in non-compliance, for instance, granting delete S3 bucket rights or granting access to confidential information stored in an S3 Bucket.

To avoid this, we make extensive use of S3 Bucket policies composed of several statements granting:

admin access to administrators,
usage access to users
and denying delete bucket rights for everyone.

Let us create an S3 Bucket with a Bucket Policy having multiple Statements.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import {PolicyDocument, Statement, AnonymousUserPrincipal, RootAccountPrincipal} from "@thinkinglabs/aws-iam-policy";

const adminRole = new aws.iam.Role("admin-role", {
 assumeRolePolicy: createAssumeRolePolicy(),
});
const userRole = new aws.iam.Role("user-role", {
 assumeRolePolicy: createAssumeRolePolicy(),
});
const bucket = new aws.s3.Bucket("bucket", {acl: "private"});
const bucketPolicy = new aws.s3.BucketPolicy("bucketPolicy", {
 bucket: bucket.id,
 policy: createS3BucketPolicy("0123456789012", bucket, [adminRole], [userRole]),
});

function createAssumeRolePolicy() {
 return new PolicyDocument([
 new Statement({
 effect: "Allow",
 principals: [new RootAccountPrincipal("123412341234")],
 actions: ["sts:AssumeRole"],
 }),
 ]).json;
}

export function createS3BucketPolicy(
 accountId: string,
 bucket: aws.s3.Bucket,
 bucketAdmins: aws.iam.Role[],
 bucketUsers: aws.iam.Role[],
) {
 return pulumi.all([
 bucket.arn,
 bucketAdmins.map((role) => role.uniqueId),
 bucketUsers.map((role) => role.uniqueId)
 ]).apply(([bucketArn, bucketAdminUniqueIds, bucketUserUniqueIds]) => {
 return new PolicyDocument([
 new Statement({
 sid: "Allow access for Bucket Administrators",
 effect: "Deny",
 principals: [new AnonymousUserPrincipal()],
 actions: [
 "s3:PutBucketPolicy",
 "s3:GetBucketPolicy*",
 "s3:DeleteBucketPolicy",
 ],
 resources: [bucketArn],
 conditions: {
 StringNotLike: {
 "aws:userId": [accountId]
 .concat(bucketAdminUniqueIds.map((uniqueId) => `${uniqueId}:*`)),
 },
 },
 }),
 new Statement({
 sid: "Allow use of the bucket",
 effect: "Deny",
 principals: [new AnonymousUserPrincipal()],
 actions: ["s3:ListBucket*", "s3:Get*", "s3:PutObject*", "s3:DeleteObject*"],
 resources: [bucketArn, `${bucketArn}/*`],
 conditions: {
 StringNotLike: {
 "aws:userId": [accountId]
 .concat(bucketAdminUniqueIds.map((uniqueId) => `${uniqueId}:*`))
 .concat(bucketUserUniqueIds.map((uniqueId) => `${uniqueId}:*`)),
 },
 },
 }),
 new Statement({
 sid: "Deny delete bucket",
 effect: "Deny",
 principals: [new AnonymousUserPrincipal()],
 actions: ["s3:DeleteBucket"],
 resources: [bucketArn],
 })],
 ).json;
 });
}

To test if the S3 Bucket Policy allows access for bucket administrators, we need to check if a Statement is present in the Policy and test the content of that single Statement.

@thinkinglabs/aws-iam-policy provides the ability to retrieve a single Statement by its Sid if one was provided.

const statement = policy.getStatement("MyFancySID");

Let us test if the S3 Bucket Policy grants admin rights for administrators by checking if the Policy contains the Statement “Allow access for Bucket Administrators”.

import {expect} from "chai";
import "./mocks";
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import {PolicyDocument, Statement, AnonymousUserPrincipal} from "@thinkinglabs/aws-iam-policy";

describe("S3 Bucket Policy", function() {
 const accountId = "123456789012";
 const bucket = new aws.s3.Bucket("a-bucket", {});
 const adminRole = new aws.iam.Role("admin-role", {
 assumeRolePolicy: "aPolicy",
 });
 const userRole = new aws.iam.Role("user-role", {
 assumeRolePolicy: "aPolicy",
 });

 const doc = sut.createS3BucketPolicy(accountId, bucket, [adminRole], [userRole]);

 let policy: PolicyDocument;
 before(async () => {
 policy = PolicyDocument.fromJson(await get(doc) as string);
 });

 it("should allow access for Bucket Administrators", function() {
 const statement = policy.getStatement("Allow access for Bucket Administrators");
 expect(statement).to.deep.equal(new Statement({
 actions: [
 "s3:PutBucketPolicy",
 "s3:GetBucketPolicy*",
 "s3:DeleteBucketPolicy",
 ],
 effect: "Deny",
 principals: [new AnonymousUserPrincipal()],
 resources: ["a-bucket-arn"],
 conditions: {
 StringNotLike: {"aws:userId": ["123456789012", "admin-role-unique-id:*"]},
 },
 sid: "Allow access for Bucket Administrators",
 }));
 });

});

The test needs some fake IAM Roles. This is achieved by including a mocks module.

import * as pulumi from '@pulumi/pulumi';

pulumi.runtime.setMocks({
 newResource: function(args: pulumi.runtime.MockResourceArgs): { id: string, state: Record<string, any>} {
 const defaultState = {
 arn: `${args.name}-arn`,
 name: args.name,
 ...args.inputs,
 };
 switch (args.type) {
 case 'aws:iam/role:Role':
 defaultState['uniqueId'] = `${args.name}-unique-id`;
 break;
 default:
 break;
 }

 const resourceId = args.id?.trim() ? args.id : `${args.name}-id`;

 return {id: resourceId, state: defaultState};
 },
 call: function(args: pulumi.runtime.MockCallArgs) {
 switch (args.token) {
 }
 return args.inputs;
 },
});

Ideas for future improvements

At the moment, Condition accepts any JSON object. Valid or not, it will serialize the object to JSON as-is. To avoid building an invalid Condition element, I am planning to add an object model for this. The API would look something like this.

new Statement({
 effect: "Deny",
 ...
 conditions: [
 new Condition("StringNotLike", "aws:userId", ["userId1", "userId2", ...]),
 ]
})

I am also planning to add validation for Sids. According to the AWS IAM documentation, a Sid only accepts alphanumerical characters [a-zA-Z0-9]. But I see that resource-based Policies for some services accept spaces for Sids. AWS does not document this. Although the documentation for S3 Bucket Policies and KMS Key Policies clearly show examples with spaces for Sids.

The library does not support NotPrincipal, NotAction, and NotResource because I did not need them at the time. At some point, I will add support for that too.

Conclusion

As you can see, you can extend a Pulumi program with existing Node modules or write your own. By testing IAM policies before deployment, we can avoid errors before deployments and provide guardrails for developers.

Cloud Engineering: The Future Is Now

Sophia Parafina — Thu, 29 Apr 2021 00:00:00 +0000

Thank you for joining the PulumiUP event. We had a stellar set of speakers and panelists discussing the future of DevOps and how Cloud Engineering is providing the tools and processes that enable faster delivery, the right mix of architecture, and foster collaboration among teams in an organization. Here are some of the highlights and takeaways from our speakers.

Putting Cloud Engineering in Practice

Our speakers all commented on how the cloud is speeding up development. Applications are dynamic, and your infrastructure should also be dynamic, according to Adrian Cockcroft, VP of Sustainability Architecture at Amazon. The future of infrastructure is a mixture of lightweight, scalable services combined with event-driven architecture. Furthermore, composing infrastructure should take advantage of well-architected frameworks such as those published by AWS and Liberty Mutual.

Underscoring this, Justin Fitzhugh, VP of Cloud Engineering at Snowflake, cited the need to disaggregate compute and storage to scale independently. Moreover, Snowflake needed to scale across three public clouds with self-managed infrastructure. In Snowflake’s experience, scaling with manual processes was not a viable option. They codified actions by following the software engineering process of code commits, tests, CI processes, and a CD pipeline for execution.

Keith Redmond, VP of SaaS Engineering at Fernergo share a perspective from the financial services industry. He reiterated the need for a deployment framework that bridged dev and ops teams and an automated build and release process to maintain integrity and reliability. He noted that this required a change in culture from teams handing off tasks to each other without input from the other to a completely integrated team where all actions with infrastructure begin with a code commit. The culture shift to an engineering-managed process also changed hiring patterns from operators and system administrators with some coding skills to software engineers interested in infrastructure.

In the current cloud engineering landscape, we will continue to see movement towards heterogeneous architectures, an increase in the use of software engineering methodologies to deploy infrastructure, and more automation for building and deploying in the cloud. This segues into the future of cloud engineering with a focus on tooling, process, and teams.

The Future of Cloud Engineering

The cloud engineering panel included Justin Fitzhugh, Dana Lawson, VP of Engineering at GitHub, Charity Majors, CTO at Honeycomb, and was moderated by Kat Cosgrove, Developer Advocate at JFrog. The session, titled “Your Peers,” discussed the future of cloud engineering.

When planning for the future of modern cloud applications, the panel agreed that the industry would continue to iterate and evolve away from domain specific languages, DSLs, and towards software engineering practices for infrastructure. They also agreed that while Go would be the most popular language for Infrastructure as Code, Ruby and JavaScript will also continue to be key languages.

Cloud engineering personas will change as new technologies such as the Internet of Things, availability of 5G, orchestration, edge computing, and future network APIs roll out. Cloud engineers should continue to focus on containers and what goes into applications. Knowing the entire lifecycle of their application from development to production is a must for cloud engineers. It underlies the concept of a single persona that can use services to compose development and services. Ultimately, the job of the cloud engineer is to reduce complexity and reduce friction in the pipeline by using patterns as needed and building platforms and tools consistently.

The panel discussed the reality of testing in production. Charity Majors highlighted that you have no choice, whether you admit it or not; there will always be known unknowns because devs or operators cannot test everything. Software will always have bugs, but the problem can be managed by having more ways to separate problems and test them in a small and controlled way. Testing in production begins with observability, starting with usual tools such as strace and other enhanced toolsets to inspect your code. Also, feature flags and deployment models help find bugs and limit their impact when applications go into production.

Finally, the panel looked backward at the impact and legacy of devops. The panel agreed that devops brought both sides to the table. Operations learned about code and applications while developers learned about infrastructure. Practitioners are software engineers, but each with a different focus. Implementing infrastructure is similar to creating a new feature. However, there is a concern for losing deep systems knowledge, and software engineers are unlikely to have a deep understanding of networking, system administration, or storage. The solution is to train people cross-functionally, i.e., software engineers on systems and operators on coding and application. Teamwork is the key to breaking down operational and knowledge silos in an organization.

Summary

Thank you for attending PulumiUP! Your response to the Pulumi 3.0 release was overwhelming. If you didn’t have the chance to catch PulumiUP on launch day, you can watch the entire event.

PulumiUP: the Event for Cloud Engineers

Wendy Smith — Wed, 14 Apr 2021 00:00:00 +0000

My professional background has included nearly ten years of managing field events and user conferences. I never thought I would say this, but I miss traveling. I even missed Vegas and AWS re:Invent this year. I miss connecting with customers and advocates in our communities. I wish we could all be looking forward to getting together in person in Seattle or Austin or insert any city here. As the year continued, it became clear we were not going back to in-person events anytime soon, and everyone in the industry pivoted to virtual programs while video conferencing became an all-day activity.

We spent a lot of time considering how to best engage and share announcements with our community and encourage the cloud engineering community’s growth. All of us here at Pulumi recognize how valuable your time is and how much time we’ve all spent on virtual event platforms in the last year. With that in mind, we made a deliberate decision to avoid any all-day or multi-day programming. Instead, we are hosting a short 2-hour keynote that we think you will find insightful and valuable, with best practices and real-life situations that you can learn from and apply to your approach to building, delivering, and managing modern cloud applications.

PulumiUP Agenda - Kicking off on 4/20 at 9 am - 11 am PST - Save your spot

Speakers	Title and Company	Subject
Joe Duffy	Co-Founder and CEO, Pulumi	The Modern Cloud Era: All Software is Cloud Software
Adrian Cockcroft & Aaron Kao	VP of Sustainability Architecture, Amazon VP of Marketing, Pulumi	Discussion: What’s different about building in a modern cloud application world
Justin Fitzhugh	VP of Cloud Engineering, Snowflake	Perspectives from Cloud Engineering Pioneers
Keith Redmond	VP of SaaS Engineering, Fenergo	Perspectives from Cloud Engineering Pioneers
Luke Hoban	CTO, Pulumi	Placing Cloud Engineering in the Hands of Every Developer
Sophia Parafina	Technical Marketing Manager, Pulumi
Komal Ali	Software Engineer

Panel: Hear from the Experts: Planning for the Future of Modern Cloud Applications	Title and Company
Moderator: Kat Cosgrove	Developer Advocate, JFrog
Justin Fitzhugh	VP of Cloud Engineering, Snowflake
Charity Majors	CTO/Co-Founder, Honeycomb
Dana Lawson	VP of Engineering, GitHub

Engineering leadership from Snowflake, the cloud data warehousing organization, and Fenergo, a client lifecycle management provider for the financial services industry, will share their perspectives and experiences practicing cloud engineering for the modern cloud era. Justin Fitzhugh, VP of Cloud Engineering at Snowflake, and Keith Redmond, VP of SaaS Engineering at Fenergo, will each share how they reduced costs while empowering every developer on their team.

Everyone at Pulumi is super excited to be sharing this material with our community. If you are interested in cloud engineering, cloud infrastructure, software development, modern applications, or Pulumi, you want to save your spot for this 2-hour event. Whether you’re a seasoned cloud engineer or just curious to learn what’s all the fuss about cloud engineering, you will learn something new about the future of building on the cloud at PulumiUP. You can also sign up for new workshops and other events occurring in April and May. We’ll miss seeing you in person but hope that changes soon! Pulumipus and I are packing our bags and hoping for the best.

Save your spot today!

Observability with Infrastructure as Code

Andy Davies — Fri, 26 Feb 2021 00:00:00 +0000

Guest Article: Andy Davies is a Senior Software Developer at Reaktor, a tech company based in Amsterdam, New York, and Finland, writing about using the Pulumi Automation API to add observability to infrastructure provisioning.

When using the Pulumi Automation API to create applications that provisions infrastructure, it is very handy to use observability techniques to ensure the application functions correctly and to find where performance bottlenecks are.

One of the applications I work on creates a VPC and Bastion host and then stores the credentials in a Vault instance. The problem is that the ‘create infrastructure’ part is an opaque blob, in that I can see it takes 129 seconds to create, but I can’t see what it’s doing or why it takes this amount of time.

So can I do better?

The Initial Application

In this example, I use Honeycomb’s Go Beeline to capture all the data I care about; durations, errors, and any context which is ‘interesting’:

func main() {

 beeline.Init(beeline.Config{
 WriteKey: os.Getenv("HONEYCOMB_API_KEY"),
 Dataset: "pulumi-demo",
 })
 defer beeline.Close()

 ctx, span := beeline.StartSpan(context.Background(), "basic-vpc")
 defer span.Send()

 name := auto.FullyQualifiedStackName(os.Getenv("PULUMI_USERNAME"), "basic-vpc", "dev")
 stack, err := auto.UpsertStackInlineSource(ctx, name, "basic-vpc", func(pc *pulumi.Context) error {

 azs, err := getAvailabilityZones(ctx)
 if err != nil {
 beeline.AddField(ctx, "err", err)
 return err
 }

 v, err := vpc.NewVpc(ctx, pc, "dev", &vpc.VpcArgs{
 Description: "dev",
 BaseCidr: "192.168.0.0/16",
 AvailabilityZoneNames: azs,
 S3Endpoint: true,
 DynamoEndpoint: true,
 })
 if err != nil {
 beeline.AddField(ctx, "err", err)
 return err
 }
 })

 if err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 if err := stack.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: os.Getenv("PULUMI_REGION")}); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 ws := stack.Workspace()
 if err := ws.InstallPlugin(ctx, "aws", "v3.23.0"); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 if _, err := stack.Refresh(ctx); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 stream := optup.ProgressStreams(os.Stdout)
 if _, err := stack.Up(ctx, stream); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 //vault code

}

Adding Infrastructure Observability

To get a handle on what happens when stack.Up() runs, I have implemented a custom io.Writer, which is passed into the ProgressStream constructor.

The custom progress stream’s Write method is called once for each line emitted, which allows us to start new spans when a resource starts being constructed, and send them when construction completes. This is currently achieved by parsing the console output text, but I gather in the future, it will be possible to get streamed JSON blobs that can be unmarshaled into go structs.

type pulumiBeeline struct {
 ctx context.Context
 contexts map[string]func()
}

func NewPulumiBeeline(ctx context.Context) *pulumiBeeline {
 return &pulumiBeeline{
 ctx: ctx,
 contexts: map[string]func(){},
 }
}

func (cw *pulumiBeeline) Write(p []byte) (n int, err error) {

 // todo: make more robust, support modifications, deletions etc.
 line := strings.TrimSpace(string(p))
 parts := strings.Split(line, " ")
 if len(parts) < 5 {
 return len(p), nil
 }

 //+ aws-vpc dev creating
 //+ <type> <name> <action>
 resourceType := parts[2]
 resourceName := parts[3]
 resourceAction := parts[4]

 if resourceAction == "creating" {
 c, s := beeline.StartSpan(cw.ctx, resourceName)
 beeline.AddField(c, "type", resourceType)
 // add other things here

 cw.contexts[resourceName] = s.Send
 }

 if resourceAction == "created" {
 cw.contexts[resourceName]()
 }

 return len(p), nil
}

Modifying the optup.ProgressStreams is the only change needed to the original application:

stream := optup.ProgressStreams(os.Stdout, NewPulumiBeeline(ctx))
if _, err := stack.Up(ctx, stream); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
}

When I rerun this program, I can see a lot more information in my Honeycomb traces, which shows me that Pulumi is highly parallelized and provides a better idea of where the time is spent when creating infrastructure. In this example, it’s the NAT Gateways:

In the future, I want to expand this to cover far more details. Among those details are why resources were created/modified/destroyed and collecting more information on what causes resources to fail.

Wrapping Up

In the end, this turned out to be much easier to achieve than I had hoped. Using Pulumi programmatically was a huge productivity boost, instead of running os.Exec directly.

I am looking forward to all the new kinds of tooling I can build to solve my user’s problems by continuing to utilize Honeycomb for my observability and Pulumi for my infrastructure.

Upcoming Workshops and Events

Sophia Parafina — Tue, 12 Jan 2021 00:00:00 +0000

It’s a new year and it’s time to level up your cloud engineering skills. Pulumi is there to get you started on your cloud engineering journey with workshops and technical sessions.

CI/CD Pipelines for Kubernetes Apps with Pulumi & Codefresh

To kickoff the year we start with a technical session that demonstrates how to continuously deliver both applications and provision infrastructure using Codefresh CI/CD. This session will demonstrate:

The basics of CI/CD using Codefresh.
How to declare cloud resources using Pulumi and your favorite programming languages.
How to define a simple pipeline for Kubernetes deployments.
Practical tests to ensure infrastructure reliability as part of your pipeline.

Taught by Pulumi and Codefresh engineers, you’ll receive a thorough understanding of CI/CD and infrastructure as code best practices.

Intro to Pulumi

If you’re new to infrastructure as code, this workshop is for you. You will learn the fundamentals of Infrastructure as Code through a series of guided exercises using Pulumi’s Cloud Engineering platform. The workshop covers setting up Pulumi, an infrastructure as code platform that uses programming languages to provision modern cloud infrastructure.

This workshop is designed to help users completely new to Pulumi become familiar with the core concepts needed to be effective with the Pulumi Infrastructure as Code platform. We will guide through the Pulumi platform with diagrams and a series of hands on exercises to help you understand the building blocks available in Pulumi.

The workshop will cover the following topics.

What is Pulumi?
How to install Pulumi
Configuring your development environment
Choose your Pulumi language
An introduction to the Pulumi state & the Pulumi Service
An introduction to Pulumi config
Storing sensitive data in Pulumi securely
Provision your first resources
Understanding stacks
An introduction to the Pulumi programming model

We have multiple workshops scheduled from January to March. Register for one today.

Jan 26: Intro to Pulumi, 9 AM PST Feb 9: Intro to Pulumi, 9 AM PST Feb 23: Intro to Pulumi, 9 AM PST March 9: Intro to Pulumi, 9 AM PST

Deploying Microservices with Pulumi and AWS Lambda

This serverless workshop demonstrates how Pulumi can accelerate provisioning of cloud infrastructure. The workshop shows how to build a set of microservices utilizing AWS’s newest Lambda features. Attendees will be guided through the process of provisioning a set of Lambda resources in AWS and see real time examples of how Pulumi’s innovative programming model helps turbocharge cloud engineering.

If you’re an active Pulumi user and building serverless applications this hands-on workshop is for you. You will learn:

How to provision a Lambda function in AWS.
How to provision infrastructure such as DynamoDB & API Gateway to support your microservice applications.
How to take advantage of the newest Lambda features.

Building a Kubernetes Platform in Amazon EKS with Pulumi

Kubernetes is one of the major cloud technologies and in this hands-on workshop, we’ll examine how Pulumi works with Kubernetes through real-world examples of managing Amazon EKS clusters. We’ll focus on building a user-friendly Kubernetes platform, installing software that makes Kubernetes easy to use for application developers.

Attendees will be guided through the process of provisioning an Amazon EKS cluster and installing platform friendly software such as the aws load balancer controller. We’ll cover the following:

How-to provision a production-ready Amazon EKS cluster with key features enabled using Pulumi
Provisioning the aws-load-balancer-controller to automate ingress creation
Install an example application to show the end-to-end user experience for users.

Get a solid foundation for provisioning Kubernetes clusters and applications by signing up.

Zephyr's Summer Intern Experience with Pulumi

Zephyr Zhou — Fri, 18 Sep 2020 00:00:00 +0000

Hi, I am Zephyr Zhou, a senior Computer Science student at the University of Washington. I spent this past summer interning at Pulumi. This is my first internship ever in my life. Thanks to Pulumi for providing this opportunity even in this difficult time of the Covid-19 epidemic. Despite the sad truth that I couldn’t get in touch offline, I believe this will be one of my most precious memories.

Time goes by so fast, but before saying goodbye to my internship, I would like to share the story.

Personal Growth

Honestly, before my internship started, I was so nervous that I couldn’t sleep. Previously, I had taken only one data management course that contains a tutorial about setting up Azure SQL Database. Other than that, I knew nothing about the cloud. Although I have read the Getting Started documentation and learned Go, I was worried that my understanding was not good enough. Also, I transferred from a community college to the University of Washington as a junior student and was admitted to the Computer Science major after that. Therefore, I felt relatively new to programming than my peers and less confident working in the industry, even less so working remotely.

However, I am glad that I was brave enough to make the first step out of my comfort zone to learn about new things and improve my understanding while practicing. That was one of the important reasons for choosing to intern at Pulumi, a startup with great potential. Also, due to the nature of startups, Pulumi has smaller but more connected teams. Personally, I felt more comfortable discussing issues and thoughts in small groups. Also, I was able to work with different teams and people. I was also able to broaden my horizons and improve my collaboration skills.

I found that the more familiar I got with the company and the job, the more confident I was. My confidence was boosted in large part from the support of team members. The warm greetings and kind help gave me a lot of powers and courage. Working remotely with Pulumi was much more interactive than I thought. The one-to-one meeting with my mentor and site supervisor has boosted my working efficiency because I could compile my questions and updates, talk about them all together, and get informed about if I am on the right path. I also loved the fact that I was able to join other team meetings so I could learn about the different projects other people are working on.

In addition, I absolutely loved the various team-building games, especially the virtual escape room and the pair-up chatting with other engineers in the team. These interesting events and relaxed atmosphere made me feel more welcomed and less stressful. I’ve been moved and realized it is okay to be imperfect and improve step by step. Despite having less coding experience, I was still welcomed and cared for by everyone at Pulumi.

Moreover, I am so glad that I could work on my individual internship project: building the content management system (CMS) for Pulumi’s website. I took control of the whole development phase of the Content Management System project. I was able to research and write up proposals, implement the project, plan the next steps, and improve. In the end, the CMS was published and running. I have contributed back to the Netlify CMS, the underlying technology for the CMS. I have built an open-source example and written blog posts on deploying the CMS application and its OAuth Access Token Server to AWS.

When I heard the CMS project would be a great help for the marketing team during our meetings, I felt indescribable happiness and proud of myself. Although far from perfect, I was finally able to prove myself with this unforgettable experience.

Confidence is the greatest gift that Pulumi gave me during this Summer internship. I felt a lot more confident in entering the technology industry and pursue my career goal as a software engineer. Thanks to this internship experience, I have grown braver and stronger than before.

Technology Skills

I have learned a lot of skills that I have never learned in the classrooms.

First of all, I learned a lot about Pulumi and cloud technology.
- Before my internship, I read about how Pulumi provides a platform for building infrastructure. At first, I thought the benefit would be to reduce the amount of work to click through different settings on the provider’s website. However, during the CMS development process, I found writing infrastructure as code can do many more things than that. For example, I have implemented Github Actions, which automatically deploys the infrastructure when I push or merge a pull request. It saves time to upload and deploy again and prevent any manual error by automating the process. Also, if we want to remove the cloud resources completely, Pulumi enables us to do a simple pulumi destroy rather than manually going through and worrying about missing any resources.
- I also learned to build the cloud infrastructure for a static website by configuring AWS S3 bucket, Route 53, CloudFront, and Certificate Manager. I also learned to configure AWS Fargate Service and create a Dockerfile.
- I learned more concepts, including assume role, CDN (Content Delivery Network), target groups, and load balancers.
Besides cloud technology, I also got to work with the Continuous Integration/Continuous Delivery concept and have practiced it with GitHub to work with the team.
- I learned more details about git command including, the difference between git rebase and git merge.
- I learned to create GitHub projects, issues, and pull requests and link them for better references.
- I got to know about GitHub Actions and Workflow and implemented one for my internship project to increase efficiency in deploying.
- I learned about GitHub OAuth Applications and more about GitHub API while working on the OAuth Server for the CMS application.
I learned to create a static website using plain HTML and CSS in the classroom. During my internship, I was able to get in touch with modern front-end technology including Netlify CMS, Hugo, Markdown, and React, while building CMS and the website’s preview.

Reflection and Next Steps

Although the CMS application’s basic infrastructure is working and provides a content management interface for updating the webinars and events pages, there are more things I want to improve. During the last meeting with the marketing team, we discussed creating a custom widget for shortcodes and image storage system. I created the necessary GitHub issues and added them to a GitHub project but didn’t have the time to implement it.

I realized I should have had a clearer time schedule of development when writing the proposal and coming up with a priority list of the features I wanted to develop initially. During my meeting with the marketing team in the beginning we only discussed the potential improvement but failed to count the time it takes to develop each feature and the priority of them. If I could communicate about that better and write a better-timed plan, I would be able to implement more new features.

Also, my mentor and manager taught me to dig deeper into the problem to understand the issue better. It also helped me organize my time better when looking into issues.

In general, these lessons and experience will benefit me a lot, especially after I enter the industry after graduation in Spring 2021. I feel more prepared now. I have been fascinated by the concept of cloud and cloud infrastructure as code. I feel the potential of cloud technology and believe I will develop my career goal in this direction.

Soon I will return to my senior year’s schoolwork. However, please feel free to keep in touch with me via my Linkedin.

Special Thanks!

First of all, I offer my sincere appreciation to Pulumi leaders, Joe Duffy and Luke Hoban, for offering this internship opportunity despite their busy schedules in the middle of our internship.

Words can’t express my thanks to Praneet Loke, my mentor, who spent every day having sync meetings with me to guide my internship pathway and help me move forward, step by step. I can’t count how many times he saved me from daunting problems and patiently explained obscure knowledge.

I cannot express enough thanks to Lee Zen, our manager, for organizing this internship, ensuring our internship experience, entrusting me with the exciting CMS project, and guiding me towards the correct path during our weekly meeting.

Moreover, my completion of the internship could not have been accomplished without:

Chris Smith. Thank you for helping me to understand the concept of assuming a role and setting up the credentials and domains for my CMS project.
Sophia Parafina. Thank you for proofreading my three blog posts and carefully fixing my expression errors.
Isaac Harris, Lindsay Marolich, and Jay Wampold. Thank you for spending time learning and trying out the CMS and sharing suggestions for improvement.
Zack Chase. Thank you for informing me about the components of Pulumi’s website and trying out my CMS application.
Freddy Hernandez. Thank you for ensuring our compensation and organizing various team building games, which made my life in quarantine fascinating.
Everyone at Pulumi. Thank you all for helping my fellow interns and me out and contributing to my wonderful internship experience!

Reflections of a Pulumi Intern

Albert Zhong — Fri, 18 Sep 2020 00:00:00 +0000

Hi everyone! I’m Albert, a soon-to-be sophomore studying computer science at the University of Washington. Today marks my last day as a Pulumi intern, so I figure I’d reflect on my experiences up until this point.

Joining Pulumi

I heard of Pulumi for the first time when they visited my school’s career fair in January. As I rounded the corner into the CSE1 atrium, I saw a banner titled “Modern Infrastructure as Code,” with some lines of TypeScript of what appeared to be the creation of an S3 bucket. I found it interesting because it looked so simple and modular. In contrast, when I had played around with AWS in the past to deploy the occasional Django web app, it had always involved following a series of exact instructions (usually go to this dashboard, copy this YAML file, run this command, and pray it works).

I walked up to the banner’s adjacent booth and chatted with Sean, a software engineer at Pulumi. He explained how Pulumi was building an open-source platform that allows developers to manage cloud infrastructure using familiar programming languages. To be honest, I didn’t have too much experience with the cloud, but I did sympathize with the pain of wrangling with YAML files. The concept sounded compelling, so I told him I was interested in interning there. He gave me the Pulumi recruiter’s email, who I later emailed that day with my resume and a brief bit about my background. After a few email and phone exchanges, I came over to the Pulumi office for an in-person interview (which was unfortunately also the only time I ever saw it, due to COVID-19.

A couple of weeks later the recruiter called me and extended an offer for the summer intern role! I was incredibly excited about the opportunity to work at such an interesting startup, and especially one working with the cloud, an industry that seemingly dominates my hometown of Seattle. I accepted it a few weeks later since I believed Pulumi had the most fascinating technical challenges to work on relative to the other companies I applied to. From then until summer, I was nervous that my position would be canceled since the coronavirus was forcing many companies to reconsider their budgets. Thankfully Pulumi confirmed that it would shift to a virtual setting, and I started in late June.

The First Week

I had very little idea of what the internship would look like. In addition to it being virtual, it was my first software engineering gig, so I had no prior experience to base it off. I wasn’t sure what my role would be—would I just be assigned a single project and be told to work on it? Would I have a mentor? How does the process of submitting code look like? Fortunately, these questions were answered fairly quickly.

For the first week, the other interns and I converted Pulumi examples from one language to another. TypeScript seemed to be the most common language, so I converted a few from that language to some of the relatively less popular languages, such as Python and Go. I thought that this was an excellent way to help us interns ramp up since it allowed us to get some baseline experience with Pulumi and the various cloud providers and programming languages it supports. By Friday, I was pretty familiar with how to navigate the various Pulumi repositories and documentation, which was a huge benefit as I launched into my project.

My Project: `crd2pulumi`

On the following Monday, I joined a Zoom meeting with Levi and my mentor Justin. They had written up a preliminary design doc on generating SDK types for Kubernetes CustomResources, which would be my project for the summer. I had a minimal background with Kubernetes, so I couldn’t fully conceptualize what the end product would look like until a few days later.

You can define custom fields of a Kubernetes CustomResource using CustomResourceDefinitions (CRDs). Pulumi already allows you to create CustomResources using the apiextensions package. However, there’s no way to inherently know the exact fields of a CustomResource, since it’s set by some CRD. Therefore we have to set the CustomResource fields to some generic type, such as map[string]interface{} in Go or pulumi.Input<any> in TypeScript. This makes filling out the fields for complex CRDs such as cert-manager or Istio very cumbersome since they contain thousands of nested properties. Being able to leverage IDE type-checking and auto-complete makes the process much quicker and less error-prone.

Throughout the summer, I progressively built crd2pulumi, a CLI tool that generates a typed CustomResource from a CRD. I then leveraged this tool to generate libraries for various Kubernetes Operators, which you can find here. I still feel like there are additional features to build out in this area though, so I plan to submit a PR here and there even after today.

School vs. Work

Before joining Pulumi, all my programming experiences had come from either school projects or practicing coding challenge type problems. So far, these scenarios have been fairly atomic, in the sense that they are self-contained and started and finished in a relatively discrete chunk of time. A typical coding project at school is straightforward—I sit down, read the instructions, code everything, and don’t think about it until my grade comes back. Most problems (not relating to the difficulty of the course content itself) can usually be resolved by looking more closely at the instructions or consulting the Piazza board.

In contrast, real software development at Pulumi was far more multi-person, rich, and dynamic. It was challenging, but also far more rewarding. In the first couple of weeks, I was seemingly running into issues at all layers—with the pulumi/pulumi codegen, with the Go compiler, with the Pulumi CLI, with the Python SDKs, or with my Macbook overheating and crashing whenever I tried opening up the examples repository in VSCode. Getting stuff to “just work” often required fixing many nested sub-problems until the root problem could finally be resolved. After a while, I internalized a lot of general debugging heuristics, and developed better skills in squashing bugs and getting code to “just work.”

I thought that Pulumi gave me an excellent balance between guidance and autonomy. My co-workers gave me an outline of what my project should achieve, and some pointers on how which tools to use, but other than that it was up to me to implement it.

One aspect that I liked about Pulumi was that its main audience is, of course, developers, so it was easier to design crd2pulumi since I am a developer. Being able to directly interact with other programmers that use my tool and hear their feedback was immensely satisfying. I loved the energy that came from working with a fast-growing startup that provided real value to real people. It was exciting to come to Monday’s team meeting and see Pulumi’s user count continuously rise, and hear about the latest deal that the sales team had just closed.

Pros of the Virtual Life

I hugely appreciated the flexibility of virtual work-from-home. In conjunction with the two summer classes I was taking, this allowed me to occasionally optimize my schedule. For example, if I had stayed up late on Tuesday night submitting a problem set and I knew I had no meetings tomorrow, I could wake up later on Wednesday morning, but just work a few more hours in the evening to compensate. However, I tried my best to avoid this to keep in sync with the majority of Pulumi employees that are on from 9 am to 5 pm PST.

After a few days, I settled into a routine of:

Wake up around 9-10 am and drink some water
Fire up my Macbook, check Slack and emails, attend meetings if scheduled, read through the reviews to my PR, and write some code
Eat brunch around 1-2 pm (almost always a double-wrapped Chipotle burrito with brown rice and chicken)
Write some more code, fix some bugs, push some changes until 5-6 pm

I feel that the quality of the technical practice I received would have been the same, had the internship been in-person. I usually knew which feature or fix to accomplish, and could troubleshoot most technical errors along the way. If I did run into a bug with my code or had questions on how to implement some function, I could always message a co-worker or set up a meeting to resolve it pretty quickly. I also picked up on a lot of knowledge regarding Pulumi, our tech stack, and best practices by just listening in on meetings and browsing the company Slack channel.

Cons of the Virtual Life

For me, the largest drawback to work-from-home was diminished social opportunities. Everybody I met at Pulumi was so interesting, talented, and kind, so I’m disappointed that I couldn’t meet them in person. There’s no online replacement for actually seeing a person for 8 hours a day and 5 days a week. I feel that so much of life is disproportionately formed via one-off, chance encounters with people, so not being able to go down to a city such as Seattle and work in an office limited my exposure to the envelope of serendipity. Looking back, I should’ve set up more 1-1 meetings with people at Pulumi, just to chat with them and learn from their experiences over lunch.

Nevertheless, it was still enjoyable, and Pulumi tried its hardest to keep us engaged with remote social activities. For example, Lee set up a couple of virtual escape room games which were entertaining.

Wrapping Up

In my view, one of the best lessons I got was that now I know how much I don’t know, which is far better than not knowing how much I don’t know. I was exposed to many different areas of tech—Kubernetes, code generation, TypeScript, Go, infrastructure as code, AWS, GCP, and so much more. I’ve only begun to scratch the huge surface area of the cloud ecosystem, and I’ll always be grateful to the awesome people at Pulumi who helped me get started.

In two weeks I’ll start (also remote) fall quarter at college. If you want to chat, feel free to reach out to me via any of the links on my website. Thanks for reading!

The Pulumi Intern Experience

Sashu Shankar — Wed, 16 Sep 2020 00:00:00 +0000

What is the cloud? Three months ago, that one word simply meant a bunch of water suspended in the atmosphere, but now it means more than that.

Hi, I’m Sashu Shankar, a second-year computer science student at the University of Washington, and this is my life as a Pulumi intern!

My Pulumi Experience

Getting to Know the Team

Let’s be real, onboarding with a new team in a remote situation was tough; the world was quite literally in a state of chaos, and each day was filled with talking to faces on the screen, but everyone was in the same situation, and that made it just a little more bearable. Coming into my first full software engineering internship, I was a nervous wreck, and I had millions of questions. What is the remote culture of the company going to be like? What is their standard for the quality and timeline of work, and am I going to be able to meet them? However, the team was always supportive, and it didn’t take long for me to feel like I was integrated with them and comfortable with the people around me. No matter the time of day, someone was always available to talk!

Asking Questions

“There is no such thing as a dumb question” was a phrase I heard a lot, but it never fully registered until recently. The “cloud” as we know it is quite vast. Setting it up takes hours, and fully understanding what is going on could take tenfold. Pulumi simplifies the process; however, there is still so much going on even when doing a single thing. This experience truly taught me that there really is no such thing as a dumb question and searching the internet to find answers is an expectation! Nobody knows all the answers, and that is completely okay.

Remote Lifestyle

I’m definitely not a fan of the remote environment, however, I’m the kind of person that is more productive at night, so it allowed me to shift my work to times when I knew I would get things done. That said, face to face interaction was something I missed a lot, as it’s understandably harder to get to know people over a computer. However, since Pulumi already has many remote engineers, it was a very smooth process to get set up and start working right away.

The Project Experience

Introduction to the Pulumi Model

I worked on the Platform team for the last 3 months, and they focused their work on the Pulumi SDK. The first few weeks were a huge learning curve, as the interns were tasked with converting Pulumi deployment examples from Typescript to Python, C#, and Go. This intro project taught me a lot about cloud resources, how they are deployed, and introduced me to helpful debugging tools through this process. I also learned how to write tests to validate output for a deployment which proved helpful as I learned how the entire process was working.

kube2pulumi Tooling

I built a CLI tool, and an open-source library to convert Kubernetes YAML manifests to Pulumi in a user-defined language check out kube2pulumi!. This was done so that Kubernetes users could easily migrate to Pulumi, and use familiar programming languages to tame YAML’s complexity. I truly learned a lot from this experience as it was my first time designing and implementing a project end to end in a set time frame. Utilizing existing open-source libraries, code generation techniques, and compilers theory were just some of the many things I got to delve into.

UI Interface

Another feature that came out of this was a UI layer to access it directly on the web; this forced me to think about different error scenarios and how they would be handled to make it a polished experience. Additionally, this really exposed the programmatic usage of kube2pulumi when writing the web service. It allowed me to further improve the overall experience in whichever form the user decides to use. To top it all off, going through a feature launch with the team was extremely exciting! It’s one thing to consistently work on an internal tool, but it’s another ball game when you are shipping your work directly to the community.

Final Thoughts

Coming into industry work right out of my freshman year of college was a daunting task; however, I wouldn’t have wanted to intern at any other place other than Pulumi! The mentorship, team camaraderie, and, most importantly, the exposure to new technologies and learning made this the best possible experience I could’ve imagined. I’m super proud of what I could accomplish with the team, and I’m excited to see what the future holds for the company!

What is the cloud? Three months later, I can say that the cloud, to me, is a vast tool for people to build their visions and dreams, and Pulumi is the place I would come to help me get that started :)

Shoutouts :)

Thank you Albert, Zephyr, and Vova for being incredible, kind, and helpful intern buddies!

Thank you Evan for being an amazing mentor by being so approachable, and always being there everyday. You were always so patient with me and answered virtually any question I threw at you :)

Thank you Lee and and Luke for being such organized, motivating, and empathetic managers!

Thank you to Levi, Lee Briggs, Komal, Pat, and Mikhail for being so open and friendly and always being available and sharing your experience when I needed anything at all!

And finally, thank you to everyone at Pulumi and the whole platform team for giving me this opportunity!

Supporting Kubernetes with Faster, Easier Test Environments

Scott Lowe — Thu, 30 Apr 2020 00:00:00 +0000

Scott Lowe is a 20+ year veteran of the IT industry and a Staff Kubernetes Architect at VMWare. He’s a prolific author (seven books) and blogger. His technology-focused blog covers a range of topics that include cloud computing (AWS, Azure, and Kubernetes), virtualization (KVM, VMware vSphere), open-source tools (Terraform, Ansible, Vagrant, and others), and networking (Open vSwitch, Linux networking).

For this guest post, Scott demonstrates how he uses Pulumi to deploy AWS test environments across multiple regions to help with testing various Kubernetes tools and projects, including the Cluster API project.

Cluster API

The Cluster API project is a Kubernetes project for implementing a declarative API to manage the lifecycle of a Kubernetes cluster from creation, configuration, and maintenance. The project lives outside of the Kubernetes core project but follows best practices for using a declarative API. Ultimately, it seeks to define everyday operations, provide a default implementation, and the ability to swap between implementations seamlessly.

Cluster API allows you to define Kubernetes clusters as a Kubernetes resource. It lets you use the Kubernetes API to manage Kubernetes clusters, which sounds a little bit like inception, but it greatly simplifies lifecycle management both on-prem and in the cloud.

Benefits of Infrastructure as Code

Pulumi is well suited for enabling me to help test projects like Cluster API. I’m not a developer (not yet), but by helping to test projects like Cluster API as they are developed, I can provide useful feedback. Pulumi makes it possible to define, test quickly, and deploy infrastructure components, whether those components are used to test new builds, verify functionality, assist with capturing information in documentation, or something else entirely.

Infrastructure as Code in Practice

To help with testing parts of Cluster API’s functionality, as well as to help with testing other configurations or tools related to Kubernetes, Pulumi helps manage my test environment or sandbox. In this instance, I’m using a Pulumi project written in TypeScript. This Pulumi project manages a consistent configuration across four AWS regions worldwide. This is set up as a single Pulumi project with four stacks; each stack defines infrastructure for a different AWS region worldwide.

All the TypeScript code is parameterized, so all you have to do is change the stack and then spin it up in your desired region. It looks for resources in that region like the right AMIs, etc. Right now, the code only accommodates four specific regions, but adding new regions would be pretty trivial.

It then spins up a VPC with both private and public subnets, attaches an internet gateway, creates security groups for various hosts, and creates one instance for an SSH bastion host that only allows SSH traffic into the cluster. This bastion host serves as the gateway by which you reach everything else inside the VPC.

Maps provide different configuration values for each region, such as different IP address values per region and different “friendly names” for each region.

A second Pulumi project, again with four stacks, is used to spin up AWS EC2 instances, which are used as Cluster API management clusters. This second project creates instances inside the infrastructure that was spun up by the first project. A stack reference is used to refer to a stack in the first project; this connection is what enables the second project to place its resources inside the infrastructure created by the first project.

Once there are a couple of instances running in the region, native Kubernetes tools bootstrap the initial Kubernetes cluster for that region. This first Kubernetes cluster becomes the management cluster, and from there, it’s possible to create additional Kubernetes clusters as needed using Cluster API.

Aside from helping create clusters in multiple regions, Pulumi makes it easy to spin up other, independent new environments for testing. A third Pulumi project creates a VPC with all the necessary added components (subnets, route tables, route table associations, NAT gateways, and an Internet gateway) for testing other Kubernetes-related technologies or projects. Because this environment is ephemeral, it isn’t parameterized for multiple regions and is typically used only within a single AWS region.

It’s not a complex environment: you create the base AWS infrastructure then add Pulumi stack references to build on top of that infrastructure to add additional instances. That enables you to spin up instances inside those environments as needed, and then tear them down when testing completes.

For example, I recently needed to refresh my AWS lab infrastructure. There were a variety of reasons, but in the end, I had a set of changes I needed to apply to all four regions where my Pulumi project maintains infrastructure. After making the necessary changes to the TypeScript code, I was able to tear down all my lab infrastructure and re-create it in a matter of 10 to 15 minutes with a simple pulumi destroy followed by pulumi up. Making these changes without an infrastructure-as-code tool like Pulumi would have taken a lot longer.

You can watch Scott demonstrate his testing environment.

Your Infrastructure as Code Story

We appreciate Scott taking the time to share his infrastructure as code story. Let us know you’re using Pulumi on Twitter or on Slack.

Life of a Pulumi Intern

Tasia Halim — Thu, 02 Apr 2020 00:00:00 +0000

👋I’m Tasia, a Computer Science student at the University of Washington and Pulumi’s very first intern. Read on to learn about some of my thoughts and experiences from these past few months!

Why Pulumi?

I’ve interned at a few different companies before, but for my last internship, there were several things I was looking for:

A start-up. All the companies I worked at previously had at least a couple thousand people, and I wanted to see first-hand the difference in both engineering and culture between larger, more established companies and smaller, newer ones.
A knowledgeable engineering team. Although I knew I wanted to work at a start-up, I also wanted access to the knowledge and expertise that usually comes with more senior engineers.
A product I can get behind. There’s a lot of start-ups out there, and not all of them work on things that I think are viable or exciting. Working on something that I can see myself and others using makes getting up for work a little more bearable.
Competitive compensation. Even though I don’t expect smaller companies to be as financially well-off as the other companies I worked for, I still wanted to be paid comparably.
Opportunities to learn something new. The main goal for all of my internships was basically just to learn something new. I don’t really know what I want to do for the rest of my career, but the best way I think to figure that out is to try a bit of everything!

If you haven’t guessed already, Pulumi checks all of these boxes! Although Pulumi was only founded a few years ago, their team is rich with experienced engineers and their product has taken a foothold in companies all over the world. Especially with the release of 2.0 right around the corner, I’m glad I joined at such an exciting time.

Working Across the Company

Pulumi engineering is currently split up into two teams: the Service Team and the Platform Team. The service team primarily works on the Pulumi Service, whereas the platform team primarily works on the Pulumi SDK. Although I’m technically a part of the Service Team, one of the benefits of working as an intern in a small company is the flexibility to work on projects across the whole company.

Within the Service Team, I added support for tags within the Pulumi Service and created new pages to view a member’s specific stack permissions. Within the Platform Team, I added support for both transformations and Docker images in Go. Lastly, in a cross-team effort, I’ve been helping out with updating our code generation tools for Pulumi 2.0. I’m sparing the nitty gritty details here for the sake of time and space, but most of the repos I worked in are open-source, so feel free to check them out!

Simply put, I took on the work of a full-time engineer, only with slightly more flexibility and a lot less experience. Prior to working at Pulumi, I had basically zero knowledge on Angular, cloud providers, and code generation. Now, I’m still nowhere near an expert on any of these, but I feel like I have a stronger grasp on where the software engineering industry is headed, thanks to the support of everyone I worked with!

Things I’ve Learned

Earlier I mentioned that one of my main goals for any internship is to learn. Here are just three key things I’ve learned about myself and software engineering during this one:

Remote life is not the life for me. Due to unforeseen circumstances, I spent the last month of my internship in my apartment. I always thought I was a bit of a homebody, but it turns out that staying in one place for more than one day makes me stir crazy, and I’m nowhere near as productive in my room as I think I am. On the upside, Pulumi already had several remote engineers, so all the infrastructure was already in place to continue the actual work side of things fairly smoothly.
Writing tests before writing code is actually a good strategy. I remember learning this in one of my classes and thinking “Yeah that seems like a good idea, but what person would actually do that?”. Apparently, that person would be me. Often times, I would find myself starting to code something without really understanding possible use cases. Making myself write tests beforehand clears that up pretty quickly.
Using “the cloud” is anything but simple. To be completely fair, I already knew this. My previous experience with cloud providers involved reading through multi-page documents just to set everything up, and clicking a handful of buttons on the console without really understanding why. Pulumi hugely streamlines that process, but even still there’s so much that goes into doing a single thing on a cloud. In other words, I have a newfound appreciation for people working on cloud infrastructure and I feel lucky to be entering the industry when resources like Pulumi already exist.

Final Thoughts

Overall, my time at Pulumi wasn’t quite the chaotic startup experience I was expecting. I came into Pulumi with this idea that start-ups consisted of stressed-out people, working around the clock, and disorganization across the board. My time here at Pulumi, however, was anything but. Looking back, the only major differences I felt while working at Pulumi (both engineering-related and otherwise) compared to my previous companies were the lack of free meals (which as a college student, I’m very partial to) and intern events (which as the only intern, I didn’t expect any of), which I think really speaks to the maturity of their engineering and leadership teams. I’m proud seeing what I’ve accomplished, and excited to see the future of Pulumi!

Conclusion

I’m soon returning to (virtual) classes (thanks, coronavirus!), but feel free to reach out to me via LinkedIn to learn more about me and/or my experiences!

Refactoring Pulumi Code with `aliases`

Ringo De Smet — Thu, 26 Mar 2020 00:00:00 +0000

Guest Article: Ringo De Smet, Founder of Cumundi, standardizes on Pulumi for writing infrastructure as reusable code libraries for his customers. Pulumi enables him to rapidly iterate through the build-test-release cycle of these building blocks.

Cumundi helps companies adopt cloud infrastructure in a more integral way. It has found that 75% of companies’ needs are covered by ‘vanilla’ infrastructure patterns. However, due to a shortage of people and time, there has been limited investment to take full advantage of cloud-native configurations - which can lead to inefficiency, poor performance, and higher costs.

How Pulumi Code Keeps Up with Changing Requirements

At Cumundi, we build reusable libraries for our customers to set up their infrastructure while following best practices. These best practices include a range of “non-functional requirements” - infrastructure patterns - which often receive less focus than application features. Where feasible, Cumundi encapsulates these non-functional requirements into infrastructure code libraries. This enables Cumundi to serve clients using all major clouds while keeping application-focused infrastructure libraries clean and modular.

It may come as no surprise that we also use Pulumi to manage Cumundi’s own infrastructure. As a young company (founded January 1st!), it is hard to foresee how our infrastructure needs will evolve. With Pulumi, we write code in a modern programming language, and, good coding practice is not to optimize prematurely because the future is unpredictable. Another coding best practice is red-green-refactor or formally known as Test-Driven Development. You take the current code, write a test for the new requirement, which initially fails (red). Next, you implement the code in the most straightforward way to make the test succeed (green) and you complete the cycle by refactoring the code to keep the design in proper shape.

This blog post demonstrates a TDD cycle for Pulumi code with a reduced version of the code used to configure the internal infrastructure we provide for each customer. This example focuses on one specific Pulumi resource property: aliases.

All the code is available on Github if you want to follow along with a full project setup. Every step described here is committed as a separate branch with the starting point on master, the default branch.

The Starting Point

Every Pulumi project starts empty, so ours is no exception. This is how our main Pulumi code file looks for now.

import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config('gitlab')

Applying the above code does nothing, which you can see in the Resources tab on the hosted Pulumi platform:

Adding the Git Repository

We create a separate repository for each of our customers:

const config = new pulumi.Config('gitlab')

const gitlabNamespace = config.getNumber('namespace')

const firstCustomer = new gitlab.Project("FirstCustomer",
 {
 name: "first-customer",
 description: "First Customer code",
 namespaceId: gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
)

We wrote the code in the simplest way to get the job done. After pulumi up, the Gitlab project is created:

A Customer Needs Cloud Infrastructure

For our second customer, I just duplicated the creation of the Gitlab repository and created a Google Cloud project and service account.

const secondCustomer = new gitlab.Project("SecondCustomer",
 {
 ... // properties here
 }
)

const secondCustomerProject = new gcp.organizations.Project("SecondCustomer",
 {
 projectId: 'second-customer',
 name: 'Second Customer Infrastructure'
 }
)

const serviceAccountSecondCustomer = new gcp.serviceAccount.Account("ServiceAccountSecondCustomer",
 {
 accountId: 'secondcustomer',
 displayName: 'Service Account for Second Customer project',
 project: secondCustomerProject.projectId
 }
);

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 }
);

const secondCustomerGitlabCIVariable = new gitlab.ProjectVariable("SecondCustomerGCPAccess",
 {
 project: secondCustomer.id,
 variableType: 'file',
 key: "GOOGLE_APPLICATION_CREDENTIALS",
 value: serviceAccountSecondCustomerKey.privateKey.apply(value => Buffer.from(value, 'base64').toString('utf-8')),
 protected: true,
 environmentScope: "*"
 }
)

These resources are all added with the Stack as their parent in the resource view, as seen below. This is getting clumsy.

Visualization goes a long way, but a flat list of resources does not clearly show what belongs together.

Find Relationships between Resources

In the current state of the code, we created a Google Cloud service account and a key. Since a key can’t be created without a service account, we also created a Gitlab CI project variable for the Gitlab project of the second customer. Two cases of a parent-child relationship.

Although we have this relationship between our resources, the Pulumi state graph doesn’t display it. How can we change this without affecting the real resources on Gitlab and Google Cloud?

We can pass CustomResourceOptions as the last argument to every Pulumi resource that we want to create and use parent and aliases for refactoring.

To link the key to the service account, we set the parent property to the service account resource. If you run pulumi preview, Pulumi wants to recreate the key. It wants to do this because it searches for the key as a child resource of the service account. In your last applied Pulumi state, that is not the case.

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 },
 {
 parent: serviceAccountSecondCustomer
 }
);

How can we tell Pulumi that the existing key resource fulfills the expectation in our code? The aliases property comes to the rescue!

If you want to rename resources in a Pulumi state or change the parent-child relationships, then the aliases property is your friend. Let’s indicate in our code that the existing key resource, linked to the Stack, is the key of interest.

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 },
 {
 parent: serviceAccountSecondCustomer,
 aliases: [
 { parent: pulumi.rootStackResource }
 ]
 }
);

Before going forward, we also changed the relationship between the Gitlab repository and the Gitlab CI variable. When you run pulumi up, nothing changes on Gitlab and Google Cloud. But after applying the changes, refreshing the resource view on the Pulumi platform displays the following state graph.

Clean Up Duplication

In the previous step, I copied the code for the Gitlab repository from our first customer to our second customer. In this step, we resolve the duplication issue, and ensure that customer projects can conditionally create the Google infrastructure needed.

I introduced a ComponentResource subclass, which encapsulates moving the code for the individual resources into the customer resource class. As we set up Google Cloud resources for other customers, we create a Google Cloud project and related resources only when required. We can use a regular Typescript if conditional for this.

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as gitlab from "@pulumi/gitlab";
import * as util from 'util';

export interface ProjectArgs {
 // The name of the customer, e.g. "First Customer"
 customer: pulumi.Input<string>;
 // Indication whether we need to create Google Cloud infrastructure for this customer
 needsGoogleInfra: boolean;
 // Namespace in Gitlab to create the repositories in.
 gitlabNamespace?: pulumi.Input<number> | undefined
}

// Pulumi custom resource representing a customer project
export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
 )

 if (args.needsGoogleInfra) {
 ... // Google Cloud and Gitlab CI Variable code here.
 }
 }
}

Now that I have our custom resource, I can use it for our existing customers. Our refactored main Pulumi file looks like this:

const firstCustomer = new customer.Project("FirstCustomer",
 {
 customer: 'First Customer',
 needsGoogleInfra: false,
 gitlabNamespace: gitlabNamespace
 }
)

const secondCustomerProject = new customer.Project('SecondCustomer',
 {
 customer: 'Second Customer',
 needsGoogleInfra: true,
 gitlabNamespace: gitlabNamespace
 }
)

When we apply this to our infrastructure, two custom resources are created and saved to your Pulumi state.

However, we have not updated the parent-child relationships.

Fix the Remaining Relationships

We have a representation of a customer project in our Pulumi state graph, but we want to see the resources of each customer as child resources of this abstraction, right?

The same trick with parent and aliases is used to re-parent our resources. Our resources are created within the constructor of our Project class. In Typescript, the language used in this example, we can use the keyword this to point to the Project instance.

export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 },
 {
 parent: this,
 aliases: [
 { parent: pulumi.rootStackResource }
 ]
 }
 )

While we are changing parent-child relationships, the service account resource is also updated to have the Google Cloud project as its parent. The result should look like the state graph below.

If another developer reads this code, they might not immediately have a clear picture of how everything is wired together. However, looking at the Pulumi resource visualization, the following properties can immediately be deduced:

our infrastructure has 2 customer projects
one customer only has a Gitlab repository, while the other also has Google Cloud infrastructure
if we create a Google Cloud project, we create a service account and a key together with it
if we create a Google Cloud project, we also set a Gitlab CI variable. It’s a pity though I can’t link the Gitlab CI variable to the key.

Next Steps

You probably noticed by now that the aliases property accepts a list. You can provide entries like:

{ name: <some old name> }
{ parent: <some previous parent> }
{ name: <some old name>, parent: <some previous parent> }

Note that putting the first two in the aliases list is not the same as putting the third one as the single entry. I leave it up to the reader to test it and understand the difference.

The take away from this example is: values for name in the aliases property can not be Inputs as they need to be resolvable at preview time, similar to the name of the resources themselves.

Conclusion

The Pulumi programming model well designed with regards to the evolution of an infrastructure. With ComponentResource and the parent and aliases properties, refactoring your infrastructure code is a breeze with visualizing abstractions on the Pulumi platform.

Pulumi Sweeps into KubeCon

Sophia Parafina — Mon, 25 Nov 2019 14:03:19 -0600

We had a fantastic time at KubeCon in San Diego. At the event, the Pulumi team released two technology previews: Pulumi Crosswalk for Kubernetes and Pulumi Query for Kubernetes.

Crosswalk for Kubernetes is a set of common patterns compiled in playbooks. These patterns reduce the complex Kubernetes API syntax by providing trusted defaults with idiomatic Kubernetes. Checkout a quick introduction to Crosswalk for Kubernetes in this blog post.

Sara Novotny defined observability as “the ability to ask of your system and learn from it” during her keynote with Liz Fong-Jones. Query for Kubernetes enables observability programmatically by exposing resource processes through either batch or streaming queries. Learn more about Query for Kubernetes in our blog post.

To see both Crosswalk and Query for Kubernetes in action, watch the demo video!

Pulumi engineers, Erin Krengel and Sean Holung, presented Are You About to Break Prod? Acceptance Testing with Ephemeral Environments.

Addressing a packed room, they showed how to use ephemeral environments (short lived environments that mimic production) to automate testing. Kudos to the CNCF for making the presentation videos available quickly, you can watch Erin and Sean’s presentation on YouTube.

Joe Duffy sums up our experience below. We hope to see you next year at KubeCon 2020!

Three Infrastructure as Code Blog Posts You Should Read

Sophia Parafina — Thu, 24 Oct 2019 00:00:00 +0000

We are always excited when people join the Infrastructure as Code community and write about their experiences. Pulumi can be used for a range of common tasks such as standardizing VPC builds, building VSphere virtual machines, or deploying your infrastructure from a CI/CD pipeline. Whether it’s TypeScript, JavaScript, or Python you can build your infrastructure with your language and tools of choice. Here are three new blog posts that show how to use Pulumi with code examples to perform these tasks.

CircleCI on Google Kubernetes Engine (GKE)

Angel Rivera shows how to set up CircleCI on GKE with a step-by-step tutorial. Angel demonstrates how to integrate Infrastructure as Code solutions, such as Pulumi, into CI/CD pipelines. The tutorial builds a container with a Pulumi project with a sample app written in Python that deploys Kubernetes. With Pulumi, your infrastructure is part of your CI/CD. The code for the tutorial is available on GitHub.

HATech Compares Pulumi to Terraform

Jon Hathway of HATech published a white paper comparing Pulumi to Terraform. Changes in Terraform prompted the comparison. Below, Jon explained why HATech is evaluating Pulumi:

“Take for example the teams that are trying to collaborate and drive reliably deployed and maintained product innovations. Is it better to have languages that support each individual team’s needs, or is it better to have one language to support the entire workflow/pipeline?

How about this thought experiment: Imagine your developers only spoke English, your QA team only spoke French, and the operations team only spoke Spanish? It would be carnage!

You’d never hire that way. So why, when choosing DevOps tools, do people find it acceptable to have different languages, DSLs that only key individuals understand?”

VMWare VSphere and Pulumi

Cody De Arkland demonstrates how to create a VMWare VSphere Virtual Machine by adding a ResourcePool, DataStore, NetworkStore and the virtual machine template with Pulumi. He also sets the configuration values such as DNS settings, default gateway, domain, and hostname all in code. All of this runs on the ESXi virtualization server and vCenter (which is why the Datacenter ID is needed for configuration). His example is available on GitHub. We like Cody’s caveat that his post is not an endorsement but call to explore new tools. In his conclusion, he nails home the point why Infrastructure as Code is the future of DevOps.

“Infrastructure as Code tools like Pulumi (and others in the space, i.e. Terraform from HashiCorp) add a ton of value on top of traditional deployment methods. Being able to store your infrastructure deployments in source control (i.e. GitHub/GitLab) opens up the door to a number of collaboration concepts. When you partner this in with some form of a CI/CD tool (i.e. CircleCI) - things get really interesting. Being able to submit a PR to change an infrastructure deployment, and having something like CircleCI automatically deploy the resources when changes are merged creates a pretty powerful platform for delivering infrastructure to end-users.”

Bloggers Welcome

If you write about your experience using Pulumi, let us know by tagging us on Twitter.

Pulumi Meetup: APIs, Custom Resources and GitHub Webhooks

Aydrian Howard — Tue, 16 Jul 2019 00:00:00 +0000

Last Wednesday, we invited members of our local Seattle community to Pulumi HQ for the July Pulumi Up meetup. The evening began with some networking time wherein our guests met some Pulumi engineers and users they may have only ever interacted with over Pulumi’s Community Slack while enjoying free pizza and beverages. This month’s meetup featured two talks by Pulumi engineers.

Application code isn’t the only code that can have APIs

Unfortunately, due to travel issues, Paul Stack wasn’t able to join us in person, but graciously agreed to present remotely… from Europe… at 4:00 in the morning. He presented “Application code isn’t the only code that can have APIs” and went over how programming languages help in building the best infrastructure code. During the talk, he presented a simple Pulumi program for creating an RDS instance and walked through the simple programming constructs he utilized to help create APIs that anyone could code against. Here’s a recording of his talk, with the video also available on the Pulumi TV YouTube Channel.

Automate your troubles away with Webhooks and Pulumi

Next up, Matt Ellis presented “Automating your troubles away with Webhooks and Pulumi.” Matt isn’t a fan of confrontation, but as the lead of the Pulumi API team, he needs to ensure that our changelog is kept up-to-date with every pull request. In his talk, he walked through his solution that leveraged GitHub Webhooks and Pulumi in building a small bot that enforced some policies on pull requests. You can read more about it in his previous blog post on Pulumi Webhooks, or watch the recording of his talk, which is also available on the Pulumi TV YouTube Channel.

After the talks, there was time for more networking and a last chance to get one more slice of pizza. After some more discussions inspired by the speakers, we closed out the evening.

If you’re in the Seattle area, be sure to join the Pulumi - Seattle meetup group and join us for our next Pulumi Up. Otherwise, subscribe to the Pulumi TV YouTube Channel and watch the presentations at your leisure.

Happy Birthday to Pulumi open source!

Joe Duffy — Tue, 18 Jun 2019 00:00:00 +0000

One year ago today – on June 18, 2018 – we open sourced Pulumi, a new approach to multi-cloud infrastructure as code using your favorite languages. And what a year it has been!

Here are some highlights we’ve added in partnership with the community since launching:

Over 100 examples, tutorials, and a brand new Getting Started guide.
A native Kubernetes provider with 100% Kubernetes API/version coverage.
A steady stream of improvements across AWS, Azure, and Google Cloud providers.
Pulumi Crosswalk for AWS, a framework with built-in AWS infrastructure best practices.
Over 20 additional providers, including CloudFlare, Digital Ocean, and MySQL.
Brought our Python 3 SDK to parity with our Node.js-based JavaScript and TypeScript SDKs.
Team and Enterprise SaaS editions for teams managing infrastructure in production.
GitHub, GitLab, Atlassian, and SAML/SSO identity providers.
CI/CD integrations with GitHub, GitLab, Codefresh, CircleCI, major clouds, and more.
Pluggable secrets management and transitive state encryption.
Pluggable state backends for AWS S3, Azure Blob Store, and Google Cloud Store.
Tools for managing complex, multi-stack environments, including Terraform integration.
Numerous engine reliability and performance improvements, including parallelism.
75 blogs, increasingly focused on end to end solutions we see working with customers.

In addition to the steady stream of product improvements, the community has grown fast:

1,000s of cloud engineers trying out Pulumi across over 1,000 companies.
1,000 end users collaborating in our Community Slack – including emerging MVPs!
Open source contributions from over 100 community members.
Nearly 5,000 commits, across over 100 open source Git repositories.

What we’ve enjoyed most is hearing about the innovation you, the end user and customer, have accomplished thanks to Pulumi’s unique approach. These experiences range from enabling better collaboration between application and infrastructure engineers; vastly improved productivity; more secure, robust, and reliable delivery pipelines; and an improved ability to go to production in multiple clouds, including by leveraging the cloud native Kubernetes stack.

A few notable success stories include:

Mercedes-Benz R&D North America used Python infrastructure as code to break down team silos between developers and infrastructure engineers.
Tableau was able to get up and running with Kubernetes in AWS, using Amazon EKS, and make progress on their journey to bringing continuous delivery to their organization.
Learning Machine went from 25KLOC YAML to 500 lines of JavaScript, got containers running in AWS and Kubernetes, and halved their cloud bill along the way. Read more.
Mapbox built a scalable IoT tracking solution with serverless Lambdas. Read more.
Multiple early stage startups, including Jargon, a TechStars company, have used Pulumi to get up and running quickly in the cloud – doing in days what used to take weeks.
Dozens of more great stories that we can’t wait to share in the weeks to come!

We want to sincerely thank our community and our customers and partners – without you, the year wouldn’t have nearly been as successful. The best part is that we’re only just getting started. Expect another year of more innovation, growing community engagement, customer success, and more focus on end to end solutions borne out of experience.

Whether it’s adopting infrastructure as code on your favorite cloud, deploying containers to Kubernetes, or architecting and executing on your multi-cloud journey, Pulumi is here for you!

Sincerely,

Joe and the Pulumi Team

Upcoming AWS + Pulumi Webinar on Feb 5

Erin Xue — Wed, 09 Jan 2019 00:00:00 +0000

Pulumi is hosting a webinar with AWS Fargate on February 5th, 10AM PST (register here). We’ll be chatting about how to implement cloud native infrastructure across your organization using AWS and Pulumi: general purpose programming languages to deliver everything from VMs to Kubernetes to Serverless.

Register Now

Registration link. Specifically, you will learn:

How using general-purpose programming languages provides familiar and powerful programming concepts to infrastructure-as-code challenges
How Pulumi can help build a library of code packages to enhance efficiency when implementing standard policies, network best practices, and more
Best practices for becoming a deployment-focused organization, rather than configuration-focused, to continuously deliver new cloud-native infrastructure with containerization

Joining us is Pulumi customer Learning Machine. Adopting Pulumi helped Learning Machine reduce 25,000 LoC of ad-hoc DevOps scripts to 500 LoC of JavaScript, improving time to ship code, time to provision and onboard new customers, and unify their Development and DevOps teams and techniques.

Download the case study

This webinar will be useful to developers, System Architects, IT managers and DevOps managers. (register here).

Ask any questions you have to our speakers, Trevor Hansen (Startup Partner Solutions Architect), Luke Hoban (CTO, Pulumi), and Kim Hamilton Duffy (CTO & Principal Architect, Learning Machine).

To learn more about the webinar, check out our e-book and case study.

Meet the Pulumi team at AWS re:Invent

Marc Holmes — Thu, 15 Nov 2018 00:00:00 +0000

Heading to AWS re:Invent? Concerned about how you’ll manage to get that much YAML into your carry on bag? Or maybe you just like purple.

Whatever the reason, the Pulumi team will be there all week at **Booth 316, Startup Central, Aria Quad, **and we’d love to chat with you about AWS and Pulumi.

Catch up with us on serverless functions, containers and Kubernetes, managed services and any other cloud native infrastructure as code, and see how you can more productively manage your AWS cloud resources with general purpose programming languages. We can even help you migrate your CloudFormation to Pulumi.

If you want to grab a specific time to talk through your needs, then use this link, otherwise we’ll just see you at the booth!

Data science on demand: spinning up a Wallaroo cluster

Marc Holmes — Fri, 02 Nov 2018 00:00:00 +0000

This guest post is from Simon Zelazny of Wallaroo Labs. Find out how Wallaroo powered their cluster provisioning with Pulumi, for data science on demand.

Last month, we took a long-running pandas classifier and made it run faster by leveraging Wallaroo’s parallelization capabilities. This time around, we’d like to kick it up a notch and see if we can keep scaling out to meet higher demand. We’d also like to be as economical as possible: provision infrastructure as needed and de-provision it when we’re done processing.

If you don’t feel like reading the post linked above, here’s a short summary of the situation: there’s a batch job that you’re running every hour, on the hour. This job receives a CSV file and classifies each row of the file, using a Pandas-based algorithm. The run-time of the job is starting to near the one-hour mark, and there’s concern that the pipeline will break down once the input data grows past a particular point.

In the blog post, we show how to split up the input data into smaller dataframes, and distribute them among workers in an ad-hoc Wallaroo cluster, running on one physical machine. Parallelizing the work in this manner buys us a lot of time, and the batch job can continue processing increasing amounts of data.

Sure, we can handle a million rows in reasonable time, but what if the data set grows by orders of magnitude? By running our classifier on a local Wallaroo cluster, we were able to cut the processing time of a million rows to ~16 minutes, thus fitting within our allotted time slot of one hour. But if the input data is, say, 10x more, we’re going to have a hard time processing it all locally.

Let’s see how we can keep up with the data growth by launching a cloud-based Wallaroo cluster on-demand, running the job, collecting the data, and shutting down the cluster, all in a fully automated fashion.

Tools of the trade

Wallaroo’s big idea is that your application doesn’t have to know whether it’s running on one process, several local processes, or a distributed system comprising many physical machines. In this sense, there’s no extra work involved in ‘migrating’ our classifier application from the previous blog post.

We will need some tools to help us set up and manage our cluster in the cloud. Wallaroo can work with a lot of different tools. Our friends at Pulumi provide an excellent tool that removes the headaches involved in provisioning infrastructure. We’ll use Pulumi to define, set up, and finally tear down our processing cluster in this example.

We’ll also need Ansible to start, stop, and inspect the state of our cluster, and, last but not least, we’ll need an AWS account where our machines will live.

Let’s jump into it!

A sample run

First of all, if you’d like to follow along (and spend some money provisioning EC2 servers), please download and set up Pulumi.

Next, clone the wallaroo blog examples repo and navigate to provisioned-classifier. If you followed along with the previous Pandas blog post, you’ll find our old application nested away here, underclassifier. What’s more interesting are the two new directories: pulumi and ansible.

Without delving into details, let’s see how to run our application on a freshly-provisioned cluster in the EC2 cloud:

make up run-cluster get-results down INPUT_LINES=1000000 CLUSTER_SIZE=3

Let’s break that down and see what’s really going on here.

make up CLUSTER_SIZE=3 configures the cluster to consist of 3 machines, and delegates to pulumi up the actual business of spinning up the infrastructure. Our physical cluster will contain 3 nodes for processing, and one extra metrics_host node for hosting our Metrics UI, and collecting results.
Once provisioning is complete, the next make task: run-cluster INPUT_LINES=1000000 uses our Ansible playbooks to upload application code from classifier/* to all 3 machines provisioned above, and then start up a Wallaroo cluster with 7 worker processes per machine.
Next, Ansible starts sending 1 million lines of our synthetic CSV data, and waits for 1 million lines to arrive at the data_receiver process. When those lines arrive, they are compressed, and the cluster is shut down.
make get-results pulls the compressed result file to output/results.tgz,
And finally, make down destroys the cloud infrastructure that was used to power our computation.

The Pulumi cluster definition

Let’s take a look at how our infrastructure is defined. This is the core of the definition:

 function instance(name) {
 return new aws.ec2.Instance(
 name,
 {associatePublicIpAddress: true,
 instanceType: instanceType,
 securityGroups: [secGrp.name],
 ami: ami,
 tags: {"Name": name},
 keyName: keyPair.keyName})
 }

 let metrics_host = instance("classifier-metrics_host");
 let initializer = instance("classifier-initializer");
 let workers = [];
 for(var i=0; i<clusterSize-1; i++){
 workers.push(instance("classifier-"+(i+1).toString()));
 }

As you can see from the above, our little instance() function encapsulates the common settings for every machine that we want to provision.

The metrics_host and initializer are ec2.Instance objects with descriptive names, while the workers are ec2.Instances that are distinguished solely by their ordinal number. Pulumi lets us define – in code – things like Security Groups, SSH keypairs, and practically every other aspect of cloud infrastructure.

An example of this capability is the keyPair object that’s used to access the instances via SSH. Our Makefile ensures that an ssh key is generated on-the-fly for our cluster, and Pulumi knows how to use it to set up SSH access for newly-provisioned nodes:

let pubKey = fs.readFileSync("../ssh_pubkey_in_ec2_format.pub").toString();
let keyPair = new aws.ec2.KeyPair("ClassifierKey", {publicKey: pubKey});

With the relevant bits of our computing infrastructure thus defined, we can tell Pulumi to take action in the real world and make it conform to our definition:

When we run make up CLUSTER_SIZE=3, we’ll see Pulumi output something like the following:

Performing changes:
+ pulumi:pulumi:Stack classifier-classifier-demo creating
+ aws:ec2:KeyPair ClassifierKey creating
(...)
+ aws:ec2:Instance classifier-2 created
---outputs:---
metrics_host: [
[0]: {
name : "classifier-metrics_host"
private_ip: "172.31.47.236"
public_dns: "ec2-54-245-53-87.us-west-2.compute.amazonaws.com"
}
(...)
]
info: 7 changes performed:
+ 7 resources created
Update duration: 1m59.524017637s

We can use Pulumi’s output to stitch together an Ansible inventory, which will let us interact programmatically with our provisioned instances.

If we want to modify our cluster, we can edit pulumi/index.js, and then rerun make up CLUSTER_SIZE=N. If the changes don’t require restarting the instances (for example if they only concern the Security Group), Pulumi will do the right thing and not disturb the rest of the infrastructure.

If you examine the Pulumi definition file in depth, you’ll see that it relies on a mystery AMI: ami-058d2ca16567a23f7. This is an experimental ubuntu-based image, with Wallaroo binaries (machida, cluster_shutdown,cluster_shrinker, and data_receiver) pre-loaded. For now, it only exists in the us-west-2 AWS region, but we hope to make Wallaroo AMIs available for experimentation in all regions starting with the next Wallaroo release.

Running the computation

Now that we know what magic powers conjured up our AWS infrastructure, let’s take a look at how we use it to run our task. Fundamentally, the components of our cluster can be described as follows:

The data source – In our case it’s the file send.py, which can generate and transmit randomly-generated CSV data for our computation to consume.
The machida processes: one Initializer and a bunch of Workers. The distinction between the two is only relevant at cluster startup.
The data receiver – A process that listens on a TCP port for the output of our computation. This is the data_receiver, provided as part of a Wallaroo installation.
The metrics UI – Our Elixir-powered realtime dashboard.

Our Ansible playbook takes care of coordinating the launch of the various components and making sure that their input, output and control ports match up. In particular, that the cluster initializer starts up knowing the total number of workers in the cluster, and every other worker connects to the initializer’s internal IP and control port.

This is what ends up running on the servers when we launch our Ansible playbooks:

Once the cluster is up and running, and the initializer node’s tcp source is listening for connections, we start up the sender and instruct it to send a stream of data to the TCP Source. In a realistic batch scenario, this sender could be implemented as a Connector that reads a particular file from a remote filesystem or S3. For our purposes, we’ll simulate this by generating a set number of CSV lines on-demand, and then shutting down.

While the work is being performed, we can take a look at the metrics URL printed out on the screen to find out how the work is being distributed.

To see the cluster’s real-time metrics, please visit http://ec2-54-200-198-6.us-west-2.compute.amazonaws.com:4000

In the screenshot above, you can see that the Initializer and B03a909b23nodes are processing about 4k messages per second each, and all the other workers have the classification work split evenly among them. Don’t be surprised that two of the workers are processing orders of magnitude more messages! Remember our pandas application pipeline?

 ab.new_pipeline("Classifier",
wallaroo.TCPSourceConfig(in_host, in_port, decode))
ab.to_stateful(batch_rows, RowBuffer, "CSV rows + global header state")
ab.to_parallel(classify)
ab.to_sink(wallaroo.TCPSinkConfig(out_host, out_port, encode))
return ab.build()

The CSV rows come in one-by-one, but we batch them and convert them to dataframes of a hundred, so that our classification algorithm can tackle more than one row at a time. Worker B03a909b23 just happens to be the worker where our state named “CSV rows + global header state” lives. Let’s take a look at its metrics:

Indeed, we can see that this worker is processing about 4k/sec messages in the “Batch Rows Of Csv, Emit Dataframes” step. Every other worker is busy classifying! Let’s see the breakdown for a different, random worker:

Looking good. All that’s left for us to do is wait until the job completes and we receive our zipped data onto our disk.

The numbers!

As a reminder, let’s take a look at the numbers we obtained by running our classifier on a single c5.4xlarge instance in AWS:

# SINGLE-MACHINE RUNNING TIMES (NO PROVISIONING)
CSV rows 1 worker 4 workers 8 workers
----------- ---------- ----------- -----------
10 000 39s 20s 11s
100 000 6m28s 3m16s 1m41s
1 000 000 1h03m46s 32m12s 16m33s

Now, let’s see how much speedup we can achieve from scaling out with our provisioned-on-demand infrastructure.

# MULTI-MACHINE RUNNING TIMES (PROVISIONING + COMPUTATION)
----------------------------------------------------------
CSV rows 4 machines/ 8 machines/ 16 machines/
28 workers 56 workers 112 workers
------------ -------------- -------------- ---------------
10 000 2m13s 2m36s 2m43s
100 000 3m38s 3m42s 3m48s
1 000 000 7m38s 6m41s 5m56s
10 000 000 40m56s 33m10s 23m24s
30 000 000 > 2h 1h45m 1h12m
----------------------------------------------------------

Even though there is some constant overhead involved in spinning up the required infrastructure (clearly too much overhead to justify spinning up clusters for less than 1 million rows), we’re now able to classify a hefty 10 million rows of CSV data in under half an hour, and 30 million in a little over an hour. This gives us some perspective on when our application will need extra resources, or perhaps some performance optimizations.

As long as the data fits in the 1 million : 10 million range, it seems that a cluster of 4 machines represents a sweet-spot between price and performance – we can process the data and still fit in the hour-long window allotted for our batch process, but not have to incur unnecessary infrastructure costs if we don’t need the extra speed.

Conclusion

The above figures illustrate how Wallaroo can be used as an ad-hoc compute cloud, using Pulumi and Ansible to provision, run workloads remotely, and shut down the infrastructure once the results are in.

In the case or our batch job, we can leverage this pattern to scale horizontally on-demand, even when the incoming workloads exceed the capacity of one physical machine – all while running the exact same Wallaroo application that we run locally as part of our regular development. Wallaroo handles the scale-aware layer of our program, so we can focus on the business logic and flow of our data.

If you’re hitting limits when running your hourly, daily or nightly batch jobs and are looking into scaling out horizontally, don’t hesitate to reach out or drop in to our IRC channel. We’d love to chat!

Pulumi Blog: Community

KubeCon EU 2026 Recap: The Year AI Moved Into Production on Kubernetes

From training to inference: the big pivot

The CNCF donations that will reshape AI on Kubernetes

Agentic AI gets an identity layer

AI gateways and inference routing

Platform engineering absorbs LLMOps

Sovereignty shapes infrastructure architecture

What this means for your team

Adding PostgreSQL State Backend Support to Pulumi: A Community Contribution Journey

The Need for Database-Backed State Storage

The Community Contribution Process

Key Technical Challenges

Performance and Limitations

PostgreSQL wins over a plain S3 bucket

Gaps you’ll still close only with Pulumi Cloud

Getting Started

Community Impact

What’s Next?

Conclusion

Introducing the Pulumi Puluminaries 2.0 Program

What is the Pulumi Puluminaries 2.0 Program?

What does it mean to be a Pulumi Puluminary?

Benefits of Joining the Program

How to Get Involved

Integrating DevOps and Security in Platform Engineering

In this article:

The Core Pillars of Platform Engineering

Aligning DevOps and Security for Secure Platform Engineering

Shift Left Security

Embrace Automation and Standardization

Prioritize Observability and Monitoring

Foster a Culture of Collaboration

Challenges of Integrating Security in Platform Engineering

Balancing Autonomy and Control

Driving Adoption and Changing Mindsets

Adapting to Evolving Needs and Technologies

Measuring Success in Secure Platform Engineering

The Future of Secure Platform Engineering

PulumiUP 2024: Dive Into the Future of Cloud, Platform Engineering, and AI/ML

PulumiUP at Glance

Main Track: Keynote, Cloud Culture Talks, and More

AI/ML Track Highlights

Platform Engineering and DevOps Track Highlights

Cloud and IaC Track Highlights

Workshops and Hands-On Learning

Amazing Partners and Sponsors

Build Your Schedule Now

Platform Engineering: Cloud-Native, Maturity Models, and Platforms as Products

On this platform engineering article:

What is a Platform?

Defining Cloud Native Platforms

The Platform Maturity Model: Charting the Path to Success

Platforms as Products: Driving Success

Conclusion

Frequently Asked Questions

What is the CNCF Platform Working Group?

Who is Dominik Kress?

How can I participate in the Platform As A Product research?

Platform Engineering & DevOps Series Kickoff Announcement

Upcoming In-Person Events

First Platform Engineering & DevOps in-person event: Berlin - July 16

Second In-Person Event: Boston, July 24

More coming soon!

Virtual Workshops

Announcing PulumiUP Conference 2024

What is PulumiUP?

Call for Speakers: Your Story Matters

Key Dates to Remember

Selection Criteria

Talk Formats and Themes

Join Us on September 18!

How AI is Transforming DevOps: AI Talks for DevOps Insights

How will AI Impact the Future of DevOps

What is DevOps in AI

What is AI in DevOps

How AI is Currently Used in DevOps

Streamlining Development Workflows

Enhancing Monitoring and Security

How Can a DevOps Team Take Advantage of Artificial Intelligence