Pulumi Blog: Gavin Johnson

Pulumi for AWS: Automate, Secure, and Manage Your Cloud

Aaron Kao — Mon, 02 Dec 2024 09:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi is excited to be at AWS re:Invent this week, where we’re showcasing our broad and deep support for AWS across all our products. From automating infrastructure with Pulumi IaC to securing secrets with Pulumi ESC to managing cloud assets with Pulumi Insights, Pulumi makes AWS a competitive advantage. Whether you’re a developer, DevOps pro, or platform engineer, Pulumi delivers the tools you need to build and manage modern cloud applications with ease.

Stop by the Pulumi re:Invent booth #370 this week to chat with experts on the Pulumi team. If you can’t make it to re:Invent, join our workshop, Accelerating Platform Engineering with Pulumi on AWS, on December 11, 2024, to see how Pulumi can enhance your cloud operations on AWS.

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

Why Pulumi for AWS?

Pulumi empowers your organization to automate AWS cloud infrastructure through code, tame secrets sprawl through centralized secrets management, and manage cloud assets and compliance with the help of AI. Pulumi encourages infrastructure, platform, development, DevOps, and security teams to collaborate and accelerates time to market with greater control and minimized risk.

Pulumi will speed up your deployments and time to value, providing you with an approach to open source and community that gives stability and choice. It will also increase your productivity and velocity through better tooling. Lastly, Pulumi uses software engineering to tackle the scale of an infinite combination of cloud architectures. Read more about why engineers and developers switch to Pulumi.

Pulumi for AWS Over the Years

Pulumi was founded back in 2017 to tackle the mounting challenges of modern cloud applications, which have grown increasingly intricate, spanning multiple environments and relying on numerous cloud resources and SaaS services. This complexity creates significant challenges for managing infrastructure. Legacy tools are designed for “two virtual machines with a database” and not thousands or millions of resources across many clouds and accounts. Teams waste time wrestling with excessive YAML configurations, resorting to copy-paste solutions, or using limited domain-specific languages (DSLs) that fail to scale effectively.

The fragmentation between development, infrastructure, and security teams only compounds these challenges. Pulumi’s founding vision addresses these pain points by enabling teams to build and manage cloud infrastructure through infrastructure as code using familiar programming languages and tools. Pulumi Infrastructure as Code (IaC) was launched with strong support of the AWS cloud. We wanted to be the best way for you to manage and scale your AWS infrastructure through code. Over the years, we launched many AWS-specific features to simplify how to manage AWS infrastructure at scale.

Well-Architected AWS Best Practices as a Component

One of the first innovations made for AWS was Pulumi Crosswalk for AWS, a collection of libraries that automatically use well-architected best practices to make common infrastructure-as-code tasks in AWS easier and more secure. Released in 2019, Crosswalk for AWS supports “day one” tasks, such as creating your initial container-based workloads, using Amazon Elastic Container Service (ECS) – including Fargate or Kubernetes (EKS) – and creating serverless workloads using Amazon API Gateway and AWS Lambda. Secure and cost-conscious defaults are chosen so that simple programs automatically use best practices for the underlying infrastructure, enabling better productivity with confidence.

Crosswalk for AWS also supports “day two and beyond” tasks, such as scaling your workload, securing and integrating it with your existing infrastructure, and going to production in multiple complex environments. This includes Amazon Virtual Private Cloud (VPC) for network isolation, AWS Auto Scaling for dynamic scaling, and AWS Identity and Access Management (IAM) for securing your infrastructure.

Earlier this month at KubeCon 2024, we launched the Pulumi Elastic Kubernetes Service (EKS) Provider v3.0.0, which is one of the Crosswalk libraries that makes Kubernetes management on AWS even easier. This release simplifies multi-cluster deployments, improves resource handling, and offers enhanced compatibility with the latest AWS EKS features.

AWS Policies as Code

We released Pulumi CrossGuard to enforce your organization’s cloud governance — security, compliance, cost controls, and more. This enables Policy as Code within your organization so that you can define guardrails for your infrastructure, ensuring engineers are following best practices and putting security first. CrossGuard helps your organization prevent mistakes before they occur and respond rapidly to any incidents.

Building Controls Planes

Many AWS customers build internal developer platforms (IDP) for developers to self-service infrastructure. In 2020, Pulumi Automation API was created to help automate infrastructure for these IDP control planes. Automation API exposes the full power of infrastructure as code through a programmatic interface, instead of through CLI commands, and it lets you use the Pulumi engine as an SDK, enabling you to create software that can create, update, configure, and destroy infrastructure dynamically. This enables you to use Pulumi to build IDPs that are custom-tailored to your team, organization, or customers.

AWS CDK for Pulumi

In 2022, Pulumi brought the ability to use AWS Cloud Development Kit (CDK) constructs from within a Pulumi deployment. For users already using AWS CDK, this provides Pulumi as a new option for orchestrating deployments in place of CloudFormation, offering improved deployment speed and integration with the full set of features of the Pulumi Platform. Even better, you can also now combine AWS CDK and Pulumi resources in a single Pulumi infrastructure as code project - passing outputs from Pulumi resources into AWS CDK constructs, and outputs from AWS CDK constructs into Pulumi resources.

AWS Cloud Development Kit (CDK) on Pulumi is now generally available. This powerful integration includes full support for CDK features like assets, custom resources, aspects, and context values/methods. It provides access to the full ecosystem of over 1,200 CDK Construct libraries, and it also supports all 180+ Pulumi providers, allowing for CDK applications to include resources outside of AWS.

Cloud Control API

In 2021, AWS launched Cloud Control API as a standard set of APIs to Create, Read, Update, Delete, and List (CRUDL) resources across hundreds of AWS Services. Pulumi supported that launch with a native provider called the Pulumi Cloud Control Provider, formerly known as AWS Native. The provider is now generally available, enabling day 1 support for new AWS resources, seamless compatibility with Pulumi IaC projects that currently use the Pulumi AWS provider, and extended support for third-party resources.

Beyond Pulumi IaC

Over the last few years, our customers have shared a clear message: IaC alone is insufficient. The complexities of modern cloud environments demand more than just IaC to manage cloud resources, improve security, and ensure compliance. In response, Pulumi’s offerings have expanded into a comprehensive suite of tools that go beyond IaC.

Pulumi Insights: Intelligent Cloud Management

The complexities of modern cloud environments create significant challenges in making sense of the millions of resources and resource updates per month, across hundreds of clouds, regions, and accounts. Identifying cost saving opportunities is difficult amidst soaring cloud expenditures, and maintaining compliance and security is mission critical because errors can cause catastrophic damage.

Pulumi Insights empowers organizations to tackle cloud complexity by delivering visibility, actionable insights, and AI-assisted optimization for their infrastructure. With Pulumi Insights, companies can scan and sync their entire AWS resource inventory – including resources not managed with Pulumi IaC – ensuring a complete picture of their cloud assets. This Account Discovery capability is now in public preview.

Compliance checks can also be run against standards such as PCI DSS, ISO 27001, and CIS benchmarks, helping maintain regulatory compliance and mitigate risks. The AI-powered Pulumi Copilot provides an interactive assistant to gain visibility into a team’s activity, discover cost saving opportunities, get compliant, and debug cloud failures. Pulumi Insights enables companies to discover, understand, manage, and improve their cloud infrastructure on their journey towards intelligent infrastructure management.

Pulumi ESC: Centralized Secrets Management

Cloud complexity creates significant challenges in managing sensitive credentials, API keys, and configurations across development, testing, and production environments. As organizations scale, they often face issues of secrets and configuration sprawl, duplication of secrets, and too many long-lived static secrets. None of the existing secrets management solutions adequately address these challenges, exposing enterprises to errors that can lead to security breaches, unintended exposure of sensitive data, and unauthorized access to critical resources.

Pulumi ESC addresses these challenges by offering seamless two-way integration – both pull and sync – with AWS Secrets Manager and other popular secrets stores, as well as the capability to consume secrets securely across any application, tool, and CI/CD platform. Developers gain easy, secure access to secrets via CLI, API, Kubernetes operators, and SDKs – eliminating the need for .env files and reducing the security risks associated with them. Additionally, Pulumi ESC can issue dynamic, short-lived AWS authentication tokens on demand via OpenID Connect (OIDC), reducing the risks that come with long-lived access tokens, while RBAC, versioning, and a detailed audit log let you trust (and prove) your secrets are secure.

Try Pulumi Cloud Today

Check out the Pulumi roadmap for upcoming features, let us know your feature requests by creating an issue on GitHub, connect with the Pulumi community on Slack, and if you haven’t yet, sign-up for Pulumi Cloud to get access to our full suite of products – Pulumi IaC, Pulumi ESC, and Pulumi Insights – so you can automate, secure, and manage everything you run in the cloud.

Meta image credit: ESO/M. Zamani

Pulumi's Latest Kubernetes Updates from KubeCon North America

Gavin Johnson — Tue, 12 Nov 2024 08:00:00 +0000

Pulumi is excited to be at KubeCon North America this week, the premier event for all things Kubernetes and cloud-native. KubeCon is the gathering place for developers, enterprises, and cloud native experts to meet and further the education and advancement of Kubernetes and cloud native computing. At Pulumi, we are strongly committed to Kubernetes and continue to support the ecosystem with infrastructure management solutions that empower teams to automate, secure, and manage Kubernetes at scale.

Come See Us at KubeCon

Stop by the Pulumi KubeCon booth R1 this week to chat with experts on the Pulumi team. If you can’t make it to KubeCon, join our workshop on November 18, 2024, to see how easy it is to manage Kubernetes with Pulumi.

Recent Pulumi Kubernetes Updates

We’ve launched several new and improved capabilities recently to serve Kubernetes users and to make it easier than ever to automate, secure, manage, and scale clusters and workloads.

Pulumi Kubernetes Operator 2.0

The Pulumi Kubernetes Operator automates the deployment and management of infrastructure by running Pulumi programs directly in Kubernetes clusters, enabling teams to manage cloud resources alongside Kubernetes-native resources. The Pulumi Kubernetes Operator 2.0 is a significant upgrade that introduces dedicated “workspace” pods for each Stack resource, effectively isolating each stack’s compute and memory resources, improving the isolation of secrets, and opening up new customization options. The operator now scales horizontally, enhancing performance and enabling teams to manage complex Kubernetes setups with greater reliability.

Pulumi ESC Secrets in Kubernetes with the External Secrets Operator

Pulumi ESC integrates with the External Secrets Operator (ESO) to securely access secrets from Pulumi ESC in Kubernetes. This enables a secure, automated way to manage and use secrets from Pulumi ESC across Kubernetes clusters, reducing the need for manual secrets maintenance and potential security risks from the local storage of secrets in manifest files.

Pulumi EKS Provider Version 3.0.0

The Pulumi EKS Provider Version 3.0.0 simplifies Amazon Elastic Kubernetes Service (EKS) cluster management with improvements in flexibility and security and the introduction of new features to enhance the Kubernetes experience on AWS. This update allows users to more easily customize networking, scaling, and security configurations for EKS clusters, streamlining deployment and management workflows.

Kubernetes-Native Support for Customer-Managed Pulumi Deployments Agents

Kubernetes-native support for customer-managed Pulumi Deployments agents allows organizations to host Pulumi Deployments agents within their Kubernetes environments enhancing the flexibility and control they have over their infrastructure deployments. Kubernetes-native support offers greater flexibility, scalability, and security for self-hosted Pulumi Deployments agents.

Improved Kubernetes await Logic in the Pulumi Kubernetes Provider

Improved Kubernetes await logic in the Pulumi Kubernetes provider enables more reliable handling of resource dependencies and synchronization when managing Kubernetes resources. This update makes it easier to ensure that resources are created or updated in the correct order, improving the overall predictability and stability of Kubernetes deployments.

Helm Chart v4 Resource in the Pulumi Kubernetes Provider

The Helm Chart v4 resource in the Pulumi Kubernetes provider introduces several new capabilities for users managing Helm charts. It allows for more sophisticated chart management, including enhanced support for dependencies between charts, applying transformations to chart resources, and the ability to handle Kubernetes namespaces more flexibly.

Revamped crd2pulumi

crd2pulumi is a CLI tool that generates strongly-typed Pulumi resources from Kubernetes CustomResourceDefinitions (CRDs), enabling better IDE support, type safety, and autocompletion for managing complex CRDs. It was revamped to enhance its handling of CRDs with multiple versions, refine the structure of its generated code, and offer more flexible output paths for smoother integration into Pulumi programs.

Learn about the crd2pulumi and its recent updates