Google Cloud Run is the latest addition to the serverless compute family. While it may look similar to existing services of public cloud, the feature set makes Cloud Run unique: Docker as a deployment package enables using any language, runtime, framework, or library that can respond to an HTTP request. Automatic scaling, including scale to zero, means you pay for what you consume with no fixed cost and no management overhead.
The Amazon Web Services (AWS) Cloud ecosystem is large and vibrant, so vast and vibrant that at times, it can be challenging to know where best to start! In the case of containers, Abby Fuller tweeted a descriptive summary about using AWS container services.
It’s been a few years since Google shut down Google Reader, and while a number of nice commercial alternatives have sprung in its wake, none of them has ever been quite the right fit for me personally.
So a while back, after far too much time spent wandering the blogsphere manually, typing URLs into address bars by hand, I decided to go looking to see whether the universe had produced an open-source solution to this problem — and to my surprise and delight, it had! Miniflux is an excellent little open-source RSS server and reader, written in Go and backed by PostgreSQL, that also happens to be packaged as a Docker container. So in this post, I’ll show how easy it is to deploy a Miniflux server of your own on AWS, using only Pulumi and a few lines of TypeScript.
Amazon offers multiple solutions for running containers in AWS, through its managed Elastic Container Service (ECS). This includes three major approaches: ECS managed automatically with Fargate, ECS backed by EC2 instances, and Elastic Kubernetes Service (EKS), delivering the full power of Kubernetes. It’s not always easy to choose between these, so in this article we provide some basic guidance on the tradeoffs you’ll encounter when choosing.
Amazon Web Services provides an incredible platform for developers to build cloud-native applications, and is used by millions of customers of all sizes. The building block services that AWS offers enable teams to offload undifferentiated heavy-lifting to AWS. To maximally benefit from these services though, cloud engineering teams must learn how to compose all of these building blocks together to build and deliver their own applications. Today, this is still too hard. Getting from your laptop to a production-ready AWS deployment frequently takes days or weeks instead of minutes or hours. And AWS building block services frequently leave you to re-implement (and re-discover) best-practices instead of providing these as smart defaults.
Pulumi Crosswalk for AWS is a new open source library of infrastructure-as-code components that make it easier to get from zero to production on AWS, easier to adopt AWS best practices by default, and easier to evolve your AWS infrastructure as your application needs mature.
Here at Pulumi, we’re big fans of Docker: at this point there is little
doubt that Docker has completely revolutionized the way that we think
about deploying software. However, even in 2019, it’s quite difficult to
get Docker containers to production.
docker run is great, and we all
love it, but unfortunately it’s quite a big leap from
docker run to
running your container in a production-ready environment.
We recently wrote a blog post about using AWS Fargate to run your docker containers with our open source packages. In this blog post we’re going to focus on another interesting aspect of Pulumi: being able to re-use your infrastructure code for both development and production!
Update: Check out the Learning Machine Case Study where provisioning went from 3 weeks to 1 hour with Pulumi and AWS.
“The impact of serverless capabilities was also transformative for the Learning Machine business. Pulumi enabled a rapid shift to Amazon ECS, AWS Fargate and AWS Lambda — the net effect of which was a 67% reduction in AWS charges. This enabled the team to spend less time focused on maintaining existing infrastructure and more time deploying new applications on AWS and adding new customers.
Pulumi is the foundational technology that allowed us to transform our organization,” said Hughes. The entire DevOps process was streamlined and in addition to realizing better productivity and higher quality, the team has new insight into their SaaS offering that they never thought possible.”
In this post, we’ll look at 12 “pearls” – bite-sized code snippets – that demonstrate some fun ways you can program the cloud using Pulumi. In my introductory post, I mentioned a few of my “favorite things”. Now let’s dive into a few specifics, from multi-cloud to cloud-specific, spanning containers, serverless, and infrastructure, and generally highlighting why using real languages is so empowering for cloud scenarios. Since Pulumi lets you do infrastructure-as-code from the lowest-level to the highest, we will cover a lot of interesting ground in short order.
Here at Pulumi we are (perhaps unsurprisingly!) huge fans of using Pulumi to manage our cloud infrastructure and services. We author our infrastructure in strongly-typed programming languages, which allows us to to benefit from rich tooling - documenting and factoring our infrastructure using the same software engineering practices we apply to our application code. This also allows us to create reusable abstractions which accelerate our ability to deliver new features and services, and our ability to standardize and refactor infrastructure patterns across our services with relative ease.
Like other users, we use Pulumi at a variety of levels of abstraction. We use Pulumi for raw infrastructure provisioning, defining the core networking layer for our AWS-based backend infrastructure. And we use Pulumi to define how our application services are deployed into ECS using just a few lines of code. Pulumi hosts and manages static content for www.pulumi.com and get.pulumi.com. We use Pulumi to define the CloudWatch dashboards connected to our infrastructure. And for monitoring, Pulumi defines metrics and notifications/alarms in PagerDuty and Slack.
Best of all, we’ve been able to take things we’ve learned from these use
cases, and others we’ve worked with beta users on over the last few
months (thank you!), and factor common patterns out into reusable
@pulumi/cloud for ourselves and
others to build upon.
In this post, we’ll do a deeper dive into each of these use cases, highlighting unique aspects of how we use Pulumi itself, and some of our engineering processes around how we integrate Pulumi into the rest of our toolchain.
Pulumi makes it easy to build cloud applications that use a combination of containers, lambdas, and connected data services and infrastructure: Colada apps.
An example of a Colada app is extracting a thumbnail from a video. A serverless function can only run for 5 minutes, so we’ll run a container in AWS Fargate to do the video processing.
In this app, a Lambda function is triggered whenever a new video is uploaded to S3. This function launches a task in Fargate that uses FFmpeg to extract a video thumbnail. A second Lambda function is triggered when a new thumbnail has been created.