Pulumi Blog: Guest post

How Skai Migrated to Amazon Keyspaces with Pulumi

Danny Zalkind — Tue, 16 May 2023 00:00:00 +0000

Danny Zalkind is the Senior Director of Infrastructure Engineering for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

As Skai continues its journey towards fully migrating to the cloud using Pulumi, we’ve taken another large bite out of the migration pie, moving our most critical data to AWS on top of Amazon Keyspaces, an Apache Cassandra–compatible database service.

This blog post will dive into how we used Pulumi and AWS Lambda to perform the migration.

Offloading a piece of the pie to AWS

First, some background on what we do. Skai helps marketers work smarter and faster with a unified platform to reach and convert shoppers. An important service we provide to our clients is optimization of their digital marketing campaign efficiency. We store billions of events per day to enable clients to continuously improve the value they get from their ad spend.

Our on-premises setup included a 100-node Apache Cassandra cluster to be able to withstand the scale of billions of events, some stored for prolonged periods of time. Maintaining and scaling this setup was a high effort from our Data Infra teams. It was clear to us it was time to offload that piece of the pie to our cloud provider.

Why Amazon Keyspaces

When looking for a solution, we were looking for a service that would mark the following checkboxes:

Fully serverless to allow us focus on our business instead of server maintenance.
Scalability should be automatic and easy to perform.
High availability and multi-region built-in.
Cost effective pay-as-you-go and commitment-based model.
Extensive support and rich documentation.
Fully automatable using Pulumi, our main Infrastructure as Code framework.

Migration method

Let’s take a look at how we set up the migration. We decided to migrate the data using a dual-write method, writing new data to both our on-prem Cassandra cluster and Keyspaces. Any historical data needed could be fetched from Snowflake, our data lake solution, using our Apache Spark pipeline.

Finally, to make sure that the data is valid and we have no data discrepancies on the Keyspaces side, we added a Lambda function to periodically sample the data and compare it to the data in our on-prem cluster.

We made a conscious decision to decouple the application code from the provisioning of the infrastructure, letting Pulumi handle all resource creation and configuration, while the application team used our standard CI/CD pipeline to deliver their code to the service owning the Keyspaces data.

Let’s start provisioning

Keyspaces requires creating a logical entity called a keyspace, which groups related tables relevant for one or more applications. After that, we iterate over our config and create each table with the relevant columns and the preferred provisioning mode — either “on-demand” or “provisioned,” depending on our workload pattern. A provisioned workload in pre-allocated and thus significantly (x7) lower in hourly price, suitable mainly for use cases where the workload is quite predictable with no spikes.

Finally we populate AWS tags in order to be able track costs and performance metrics.

The following Python code illustrates the process:

import pulumi
import pulumi_aws as aws
from pulumi_komponents.account_provider.account_provider import get_account_provider

# Define resource tagging
def __populate_tags(aws_config: pulumi.Config, component_type: str, tags: dict):
 return {
 "Project": pulumi.get_project(),
 "Environment": aws_config.require("account"),
 "Type": component_type,
 **tags,
 }

def pulumi_program():
 aws_config = pulumi.Config("aws")
 keyspaces_config = pulumi.Config("keyspaces")
 tags = keyspaces_config.require_object("tags")
 provider = get_account_provider(aws_config)

 # Define keyspace
 keyspace = aws.keyspaces.Keyspace(
 resource_name=f'keyspaces-{keyspaces_config.require("name")}',
 name=keyspaces_config.require("name"),
 tags=__populate_tags(aws_config, "keyspace", tags),
 opts=pulumi.ResourceOptions(provider=provider, protect=True),
 )

 # Define and iterate over keyspace tables
 tables = keyspaces_config.require_object("tables")

 for table_name, table_ddl in tables.items():
 columns = []
 partition_keys = []
 clustering_keys = []
 static_columns = []
 for col in table_ddl.get("schema").get("columns"):
 name, type = col.popitem()
 columns.append(
 aws.keyspaces.TableSchemaDefinitionColumnArgs(name=name, type=type)
 )
 for par in table_ddl.get("schema").get("partition_keys"):
 partition_keys.append(
 aws.keyspaces.TableSchemaDefinitionPartitionKeyArgs(name=par)
 )
 for cl in table_ddl.get("schema").get("clustering_keys") or []:
 name, order_by = cl.popitem()
 clustering_keys.append(
 aws.keyspaces.TableSchemaDefinitionClusteringKeyArgs(
 name=name, order_by=order_by
 )
 )
 for s_col in table_ddl.get("schema").get("static_columns") or []:
 static_columns.append(
 aws.keyspaces.TableSchemaDefinitionStaticColumnArgs(name=s_col)
 )

 mode = table_ddl.get("throughput").get("mode")

 aws.keyspaces.Table(
 resource_name=f"keyspaces-table-{table_name}",
 keyspace_name=keyspace.name,
 table_name=table_name,
 schema_definition=aws.keyspaces.TableSchemaDefinitionArgs(
 columns=columns,
 partition_keys=partition_keys,
 clustering_keys=clustering_keys,
 static_columns=static_columns,
 ),

 # Set capacity mode and units for read and write per Table
 capacity_specification=aws.keyspaces.TableCapacitySpecificationArgs(
 throughput_mode=mode,
 read_capacity_units=table_ddl.get("throughput").get("rcu")
 if mode == "PROVISIONED"
 else None,
 write_capacity_units=table_ddl.get("throughput").get("wcu")
 if mode == "PROVISIONED"
 else None,
 ),

 # Set general table properties such as TTL and recovery method
 default_time_to_live=None,
 ttl=aws.keyspaces.TableTtlArgs(status="ENABLED"),
 point_in_time_recovery=aws.keyspaces.TablePointInTimeRecoveryArgs(
 status="ENABLED"
 ),

 # Populate AWS resource tagging
 tags=__populate_tags(
 aws_config,
 "table",
 {
 "Table-type": "unarchived"
 if table_name.endswith("_unarchived")
 else "standard",
 **tags,
 },
 ),
 opts=pulumi.ResourceOptions(
 parent=keyspace, provider=provider, protect=True
 ),
 )

Comparison Lambda

Next we define and create a Lambda function that samples and compares data between the data newly ingested into Keyspaces, our on-premise Cassandra cluster, and our data lake Snowflake.

We created an abstract class implementation of AWS Lambda to standardize our various Lambda implementations and keep the code readable. This class returns all needed arguments, configuration, layers, etc. Here is some Python code that shows our abstraction, named AwsFunction:

class AwsFunction(AbstractAwsFunction):
 def __init__(self, args: AwsFunctionArgs):
 super().__init__(resource_type="AwsLambda:Function", args=args)

 # Set Lambda zip artificate path
 def _create_function(self) -> aws.lambda_.Function:
 code_archive_path = f"/tmp/{self.args.name}.zip"
 with zipfile.ZipFile(code_archive_path, "w") as zip_obj:
 zip_sub_folders(
 root_source=self.args.functions_absolute_path,
 destination=code_archive_path,
 selected_sub_folders=[ZIP_COMMON_DIR, self.args.code_directory_name],
 zip_obj=zip_obj,
 )

 # Get iam role, kms keys and layers
 self.function_role = self._get_function_role()
 self.kms_key = self._create_kms()
 layers_arn, layers = self.__build_layers()

 # Return function outputs
 return aws.lambda_.Function(
 resource_name=generate_resource_name(self.args.name, "function"),
 name=self.args.function_name,
 role=self.function_role.arn,
 package_type=self.args.package_type,
 architectures=[self.args.architecture.lambda_value],
 handler=self.args.handler,
 memory_size=self.args.memory_size,
 reserved_concurrent_executions=self.args.concurrent_executions,
 runtime=self.args.runtime,
 code=pulumi.FileArchive(code_archive_path),
 timeout=self.args.timeout,
 layers=layers_arn,
 tags=self._populate_tags(component="lambda", resource_type="function"),
 kms_key_arn=self.kms_key.arn,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=self.args.environment_variables.to_dict()
 ),
 vpc_config=self.args.vpc_config,
 opts=ResourceOptions(
 provider=self.args.provider, depends_on=[self.kms_key] + layers
 ),
 )

Now we instantiate the abstract class as part of our “building block” component class, create the function, triggers and invocation:

class AwsLambda(ComponentResource):
 def __init__(self, args: AwsLambdaArgs):
 super().__init__(f"AwsLambda", args.lambda_name)

 # Define the function nased on the abstract class
 aws_function = (
 AwsFunction(args.aws_function_args)
 if args.aws_function_args.package_type == "Zip"
 else AwsFunctionContainer(args.aws_function_args)
 )

 # Add triggers
 triggers = {}
 for trigger, config in args.aws_function_args.lambda_triggers.items():
 triggers[trigger] = LAMBDA_TYPES_TO_DEF[trigger](
 config, aws_function.function, args.provider, args.tags
 )

 # Add async invocation config
 FunctionEventInvokeConfig(
 generate_resource_name(args.lambda_name, "event-invoke-config"),
 function_name=aws_function.function.name,
 destination_config=FunctionEventInvokeConfigDestinationConfigArgs(
 on_failure=FunctionEventInvokeConfigDestinationConfigOnFailureArgs(
 destination=args.aws_function_args.on_failure_arn,
 )
 if args.aws_function_args.on_failure_arn
 else None,
 on_success=FunctionEventInvokeConfigDestinationConfigOnSuccessArgs(
 destination=args.aws_function_args.on_success_arn,
 )
 if args.aws_function_args.on_success_arn
 else None,
 ),
 qualifier=args.aws_function_args.qualifier,
 maximum_event_age_in_seconds=args.aws_function_args.maximum_event_age_in_seconds,
 maximum_retry_attempts=args.aws_function_args.maximum_retry_attempts,
 opts=ResourceOptions(provider=args.provider),
 )

Now we’re ready to call our class to create our comparison Lambda:

from pulumi_komponents.awslambda import AwsLambda, AwsLambdaArgs
from pulumi import Config

config = Config()

AwsLambda(AwsLambdaArgs(config))

Note: This only a partial code example, meant to demonstrate the way we use object-oriented paradigms to create reusable building-blocks of code to be consumed by various platform and R&D teams to provision AWS resources.

Putting it all together

Once we have our Keyspaces setup in place and the confidence that our dual-write process is writing the data correctly, we are now able to wait enough time to accumulate the data on the AWS side, using a feature toggle in our application to switch our backend systems to work with AWS Keyspaces.

Outcome

Following the successful migration, we have gained 50% reduction in backend process runtime, increased event processing throughput by 400% and freed up 30% of our team’s time to do other things, instead of maintaining cluster infrastructure.

I hope this post provides readers with useful techniques and ideas for migrating large-scale databases to the public cloud.

Managing NOAA Open Data across Multiple Clouds with Pulumi

Denis Willett, PhD — Tue, 08 Nov 2022 21:01:21 +0000

Denis Willett is a software engineer at the North Carolina Institute of Climate Studies who works on the NOAA Open Data Dissemination Program. His work focuses on leveraging cloud technologies for the development of data processing and machine learning pipelines. Denis did his PhD in Entomology and Nematology at University of Florida and his undergraduate and masters work in Earth Systems at Stanford University. You can read his full bio here.

NOAA Open Data Dissemination (NODD) makes environmental data freely and publicly accessible across Amazon Web Services (AWS), Microsoft Azure (Azure), and Google Cloud Platform (GCP). These data include near real-time satellite imagery, weather models, radar feeds, drought information, ocean databases, and a suite of climate data records among many others. This program supports more than 220 datasets and over 24PB of open data. Since its inception, the program has been growing rapidly, almost doubling in size over the past year.

Availability of the petabyte scale environmental data on the cloud is only one aspect of NODD. The program places a premium of performant data delivery and access. NODD data pipelines provide resilient and low, in many cases near real-time, latencies to NOAA’s core data offerings. In addition, NODD provides performant, parallel access to these data supporting billions of requests per day and accessions of individual datasets exceeding 1PB in a single day.

To support this program, the North Carolina Institute for Climate Studies (NCICS) through the Cooperative Institute for Satellite Earth System Studies (CISESS) engineers infrastructure on all three major public clouds ranging from object stores to messaging platforms, data ingest and transformation pipelines, and monitoring systems. These systems are deployed and managed by a very small team that, by necessity, must be agile, resilient and facilitate low latency data transfers at petabyte scale. Given the scale and diversity of resources managed through this initiative, we began looking for tools to facilitate multi-cloud management using infrastructure as code (IAC).

Before Pulumi

Prior to beginning to transition to Pulumi, our team built infrastructure primarily using the Command Line Interface (CLIs) and consoles of each cloud service provider. While this worked well, it required extensive Cloud Service Partner (CSP) specific training to be able to not only navigate the nuances of each provider’s offerings, but also the unique interface to deploy resources. Changing and updating infrastructure in this manner resulted in long development cycles where upgrades were mostly manual and composed of repetitive tasks.

Because of the manual nature of these deployments across three different cloud service providers, infrastructure began to sprawl. A clear snapshot of what resources were deployed where became difficult to compile and management of legacy and deprecated systems depended on manual intervention for clean-up. Infrastructure also began to drift. What started as uniform deployment of resources across multiple cloud platforms morphed as manual updates to each instance resulted in unique environments that were hard to replicate if they had to be deleted and redeployed.

As an example, our program relies on Apache NiFi deployments on AWS, GCP, and Azure to coordinate and manage petabyte scale data transfers while managing data provenance. Drift in these deployments makes it especially difficult to redeploy identical configurations (even though we were using Ansible and Containerization) if they go down.

An additional consequence of this approach is knowledge silos. The team member who deployed a group of resources on AWS had trouble (even with extensive documentation) communicating reusable architectures to another team member who was working on similar situations in Azure. While there are obvious nuances relevant to each cloud environment, our need to replicate similar infrastructure in a multi-cloud environment at scale necessitated extensive detailed communication in a team environment. Questions like: “What is the ARN of the most recent version of this service?” and “What queues do we have on GCP?” required communication and time taken by the individual responsible to answer those questions.

How and why we chose Pulumi

As our team grew and our platform scaled (more than doubling in storage size over the past year by more than 10PB), we began to explore tooling to help us better navigate and manage our multi-cloud and multi-account environment. The nature of the NOAA Open Data Dissemination program agreements with each cloud service provider (CSP) meant that we had multiple accounts with different permissions, funding, and configuration on each CSP, resulting in flexible options.

We began looking at infrastructure as code (IAC) solutions and initially considered many of the cloud-specific options: CloudFormation on AWS, Deployment Manager on Google, and Resource Manager on Azure. While these were nice for automation, we quickly discarded these as options because each had an individual learning curve for that environment; they did not translate well in a multi-cloud environment. These tools also did not have the control features of a full-fledged programming language; looping and if-then statements were challenging to implement.

We also considered other Infrastructure as Code providers. While these solutions would meet a lot of our needs, and one of our team members had extensive experience with one of these tools, the ability to use a common tool seamlessly in a multi-cloud environment was attractive. However, the learning curve associated with a domain specific language was cause for reticence. In addition, we really wanted to be able to use the features of a full-fledged programming language.

After experimenting with a few IAC options, our research institute began using Pulumi in late 2021 and immediately began implementing it to expand our infrastructure.

Benefits of using Pulumi for multi-cloud infrastructure as code

Pulumi had all the benefits we were looking for in a multi-cloud infrastructure as code platform. In terms of automation, manual checklists could be converted to code reducing both deployment time and human errors in deployment. This automation with infrastructure as code (and accompanying state management) allowed us to adopt an immutable infrastructure paradigm where old resources are replaced with new updated infrastructure. While these benefits are accessible via other IAC tools, we particularly appreciated the Pulumi interface to be able to quickly access and inspect infrastructure deployments.

We saw immediate wins for our teams in the following areas:

Drift was eliminated. Single source of truth, version controlled (and tagged) code documented exactly what was deployed where. When updates were needed, the code was updated and redeployed without manually tweaking individual resources through the console.
Management of legacy and deprecated resources were obviated. With IAC managed deployment that would automatically clean up old resources, we saved many future hours cleaning up unused cloud resources.
Team collaboration was enhanced. With a single source of truth and a visual snapshot of resources and deployments, the number of ‘What was the ARN of that resource again?” type questions dropped considerably. Team members could simply look at the deployment in one location and see all of the resources across all three major clouds. This type of friction-reduction really enhances team productivity.

In addition to these IAC benefits, we saw a number of key advantages in implementing Pulumi. First and foremost was the ability to use full-featured programming languages to deploy IAC solutions in a multi-cloud environment. This not only allowed us to fully take advantage of programming control structures (e.g., need 100 different rules on AWS? Now you do not have to manually script each one, just put it in a loop), but also flattened the learning curve for adoption. Our team already used Python extensively in our day-to-day work. Adopting Pulumi was almost as simple as importing a package. Accessing Pulumi through Python made it much easier to embrace.

This accessibility through Python also meant that learning how to create, deploy, and share reusable templates was trivial as all team members already knew how to build, share, and import modules in Python. The ability to rapidly share these templates accelerated our development and allowed for standardization around best practices. For example, we use serverless capabilities on all three CSPs. Once we had a template for deploying our function as a service (FAAS) code on one platform, we did not have to reinvent the wheel or remember to go through a checklist; it was as simple as importing our internal FAAS module and pointing it to the correct code to deploy.

Next steps

Pulumi has become an automation and collaboration tool that is accelerating our ability to provide petabyte scale data to the public in a multi-cloud environment. As we continue to scale our program, Pulumi is allowing us to leverage a small team to manage increasingly larger, more complex deployments in a dynamic environment. In addition, it is rapidly becoming a key player in a cross-cloud fabric that we are building to facilitate working across multiple platforms. By providing a similar, intuitive interface through our language of choice, Python, it is allowing us to more quickly adapt and reproducibly deploy infrastructure across AWS, Azure, and GCP.

Please reach out to us with your thoughts, reactions, and comments at NODD@NOAA.GOV.

Skai Migrates to AWS with Pulumi

Danny Zalkind — Tue, 04 May 2021 00:00:00 +0000

Danny Zalkind is the DevOps group manager for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai’s solution provides data-driven insights and optimization technology to help companies make informed decisions and scale performance across critical publishers.

Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers.

Why Skai is moving to AWS

The public cloud is not entirely new to us. Skai’s microservices have been running on AWS for a few years now. Our core monolith service is hosted in a data center running our private cloud infrastructure with its hundreds of Terabytes of data.

As our business grew, we noticed that our private cloud infrastructure could not keep pace with our business needs. Adding new capacity to our private cloud could take months of purchasing hardware, preparation, shipping, and deployment.

We decided to adopt infrastructure elastic enough to support the rate of innovation and growth that the business demanded.

We sought out the best infrastructure provider to shift as many services as possible to the cloud. AWS is a clear choice because of its unmatched managed services: RDS for our MySQL databases, Lambda functions for our scheduled processes and various automation tasks, DMS for data migration from the datacenter, Keyspaces for our Cassandra cluster, and EKS for our container workloads.

How Pulumi helped with migration

Once the team selected a cloud platform, it was time to plan and execute the migration. Skai’s DevOps group was selected to lead the project.

Why Pulumi

Skai’s private cloud infrastructure was provisioned and deployed using a combination of Puppet and Fabric. The code was long, full of customizations, not readable, and challenging to troubleshoot.

We were looking for a new solution that would:

Support cloud-native and multi-cloud architectures
Allow adding complex logic to the deployment process
Support reusable code blocks for various use cases
Allow each team to author different parts of the infrastructure
Use a modern programming language
And not create bottlenecks in the release process between the teams

We immediately fell in love with Pulumi, with its programmatic design, extensive modules, and extensibility via dynamic providers.

Project and Stack design

When the Skai team decided to standardize on Pulumi, we designed our architecture to support our use case and development cycle.

We decided to create a project for each AWS organization OUs and a stack for each instance of our application service.

For example, a project would be “platform-production” and contain all stacks and resources of individual environments such as “app1234,” where the number is an environment identifier.

This provided the optimal mix of releasing new code versions per workload type and placing them in the development lifecycle while making granular configuration changes per each instance more manageable.

Programming language

Pulumi supports several programming languages. For us, Python was an obvious choice due to its highly extensible nature, simplicity, and the team’s considerable amount of prior experience with Python. Since most existing automations were written in Python, this made adopting Pulumi very easy for the team.

Multi-team collaboration

Our organization is structured in multiple teams, with each team responsible for different areas of our infrastructure:

The Core team in charge of shared infrastructure such as provisioning AWS accounts
Multiple application teams are responsible for provisioning specific applications, and
An architect who leads designs, builds proof of concepts, and writes code.

It was essential to build our IaC projects with minimal coupling between the teams while keeping each team aligned with the same overarching project goals.

To enable this “loosely-coupled, tightly-aligned” approach, we created component classes for different infrastructure elements, which we later referenced in different places in our end-to-end code.

import pulumi
from pulumi_aws import ec2
from pulumi_aws import ebs

class KSAppServer(pulumi.ComponentResource):

 def __init__(self,
 name: str,
 args: KSAppServerArgs,
 opts: pulumi.ResourceOptions = None):
 super().__init__('kenshoo:KS:KSAppServer', name, None, opts)
 self.name = name
 self.app_instance = ec2.Instance("ks-aws-ue-" + args.ks_id + "-app",
 instance_type=args.instance_size,
 vpc_security_group_ids=args.security_groups_id,
 subnet_id=args.subnet,
 ami=args.ami,
 user_data=args.user_data_content,
 key_name=args.key_pair,
 tags={
 'Name': "ks-aws-ue-" + args.ks_id + "-app"
 },
 opts=pulumi.ResourceOptions(parent=self, delete_before_replace=True))

This practice enabled the application team to work on classes without being entirely dependent on the progress of the Core infrastructure team, which worked on classes of more low-level infrastructure such as VPCs and networking objects.

What about those internal metadata systems?

One more challenge we faced is registering each new instance of our application with internal metadata and IT systems. In our data center setup, we relied on a set of Jenkins jobs, but now, we wanted to integrate these processes with our code and make sure that we always have a consistent state of registration. We decided to use Pulumi’s dynamic providers to achieve this:

from requests import post, get

URL = 'https://krem.ourdomain.com/api/'

class KremMetadataRegistration:
 def create(self, ks_id):
 post(url=f'{URL}/onboarding/init_new_ks',
 params={'ks': f'ks{ks_id}'})

 def delete(self, ks_id):
 response_id = get(url=f'{URL}/config', params={'ks': ks_id}).json()[
 'id']
 if response_id:
 post(url=f'{URL}/onboarding/offboarding',
 params={'id': response_id})

Next, we reference the dynamic provider in another class that we call from the end-to-end code of the whole provisioning flow.

from pulumi.dynamic import *
from .metadata.krem_metadata_registration import KremMetadataRegistration

class RegistrationSchemaInputs(object):
 ks_id: Input[str]
 registered_service: Input[bool]

 def __init__(self, ks, registered_service):
 self.ks = ks
 self.ks_id = ks.lstrip('ks')
 self.registered_service = registered_service

class SchemaProvider(ResourceProvider):
 def __init__(self):
 self.metadatas = {
 'krem': KremMetadataRegistration()
 }

We can streamline the registration and deregistration processes for multiple systems as part of our provisioning lifecycle, creating a more streamlined flow with this approach.

Summary

Moving a high-scale and mature application to the public cloud presented quite a few challenges. We have selected Pulumi as our IaC framework of choice due to its native programmatic nature, an endless number of modules, and the ability to customize the solution using our dynamic providers and custom logic.

This post gave you a glimpse into how we designed and implemented provisioning of our application on AWS. Join us at the second part of the blog that follows our journey of automating data migration.

Automate Your Infrastructure with Automation API and C#

Joshua Studt — Mon, 08 Mar 2021 00:00:00 +0000

Joshua Studt is a Solutions Architect at Financial Independence Group and a Pulumi Community member who contributed the C# package for Automation API.

Currently available in public preview, Pulumi’s Automation API enables you to provision your infrastructure programmatically using the Pulumi engine. Today, we are excited to announce C# support for Automation API, enabling .NET developers to automate infrastructure deployments, create complex orchestration workflows, build custom ops tooling, and build cloud frameworks. Read more about the Automation API here.

Using Automation API in .NET

The Pulumi.Automation NuGet package exposes a LocalWorkspace for creating and managing Pulumi Stacks, and a WorkspaceStack that is a programmatic representation of a Stack for updating, refreshing, previewing, and destroying cloud resources. The Automation API makes it trivial to run Pulumi programs inline:

var program = PulumiFn.Create(() =>
{
 var bucket = new Pulumi.Aws.S3.Bucket("s3-website-bucket");

 return new Dictionary<string, object?>
 {
 ["bucket_name"] = bucket.BucketName,
 };
});

var stackArgs = new InlineProgramArgs("projectName", "stackName", program);
using var stack = await LocalWorkspace.CreateOrSelectStackAsync(stackArgs);

await stack.Workspace.InstallPluginAsync("aws", "v3.30.1");
var result = await stack.UpAsync();
var bucketName = result.Outputs["bucket_name"];

You can use your existing Pulumi projects and invoke them from the Automation API:

var stackArgs = new LocalProgramArgs("stackName", "C:\path\to\pulumi\project\dir");
using var stack = await LocalWorkspace.CreateOrSelectStackAsync(stackArgs);

await stack.UpAsync();

Since the Automation API can be invoked and debugged like any other code, it enables you to put together complex deployment workflows such as a blue-green deployment model:

using var workspace = await LocalWorkspace.CreateAsync(new LocalWorkspaceOptions
{
 Program = PulumiFn.Create<ApplicationStack>(), // use your existing Pulumi.Stack implementation
 ProjectSettings = new ProjectSettings("projectName", ProjectRuntimeName.Dotnet),
});

// your "blue" production code is already running

var green = await WorkspaceStack.SelectAsync("green", workspace);
await green.UpAsync();

// do your cutover to "green" logic
// and then teardown "blue"

var blue = await WorkspaceStack.SelectAsync("blue", workspace);
await blue.DestroyAsync();

See full Automation API examples in C#. Let us know if you’re using Automation API on Twitter or in our Community Slack.

Observability with Infrastructure as Code

Andy Davies — Fri, 26 Feb 2021 00:00:00 +0000

Guest Article: Andy Davies is a Senior Software Developer at Reaktor, a tech company based in Amsterdam, New York, and Finland, writing about using the Pulumi Automation API to add observability to infrastructure provisioning.

When using the Pulumi Automation API to create applications that provisions infrastructure, it is very handy to use observability techniques to ensure the application functions correctly and to find where performance bottlenecks are.

One of the applications I work on creates a VPC and Bastion host and then stores the credentials in a Vault instance. The problem is that the ‘create infrastructure’ part is an opaque blob, in that I can see it takes 129 seconds to create, but I can’t see what it’s doing or why it takes this amount of time.

So can I do better?

The Initial Application

In this example, I use Honeycomb’s Go Beeline to capture all the data I care about; durations, errors, and any context which is ‘interesting’:

func main() {

 beeline.Init(beeline.Config{
 WriteKey: os.Getenv("HONEYCOMB_API_KEY"),
 Dataset: "pulumi-demo",
 })
 defer beeline.Close()

 ctx, span := beeline.StartSpan(context.Background(), "basic-vpc")
 defer span.Send()

 name := auto.FullyQualifiedStackName(os.Getenv("PULUMI_USERNAME"), "basic-vpc", "dev")
 stack, err := auto.UpsertStackInlineSource(ctx, name, "basic-vpc", func(pc *pulumi.Context) error {

 azs, err := getAvailabilityZones(ctx)
 if err != nil {
 beeline.AddField(ctx, "err", err)
 return err
 }

 v, err := vpc.NewVpc(ctx, pc, "dev", &vpc.VpcArgs{
 Description: "dev",
 BaseCidr: "192.168.0.0/16",
 AvailabilityZoneNames: azs,
 S3Endpoint: true,
 DynamoEndpoint: true,
 })
 if err != nil {
 beeline.AddField(ctx, "err", err)
 return err
 }
 })

 if err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 if err := stack.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: os.Getenv("PULUMI_REGION")}); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 ws := stack.Workspace()
 if err := ws.InstallPlugin(ctx, "aws", "v3.23.0"); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 if _, err := stack.Refresh(ctx); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 stream := optup.ProgressStreams(os.Stdout)
 if _, err := stack.Up(ctx, stream); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 //vault code

}

Adding Infrastructure Observability

To get a handle on what happens when stack.Up() runs, I have implemented a custom io.Writer, which is passed into the ProgressStream constructor.

The custom progress stream’s Write method is called once for each line emitted, which allows us to start new spans when a resource starts being constructed, and send them when construction completes. This is currently achieved by parsing the console output text, but I gather in the future, it will be possible to get streamed JSON blobs that can be unmarshaled into go structs.

type pulumiBeeline struct {
 ctx context.Context
 contexts map[string]func()
}

func NewPulumiBeeline(ctx context.Context) *pulumiBeeline {
 return &pulumiBeeline{
 ctx: ctx,
 contexts: map[string]func(){},
 }
}

func (cw *pulumiBeeline) Write(p []byte) (n int, err error) {

 // todo: make more robust, support modifications, deletions etc.
 line := strings.TrimSpace(string(p))
 parts := strings.Split(line, " ")
 if len(parts) < 5 {
 return len(p), nil
 }

 //+ aws-vpc dev creating
 //+ <type> <name> <action>
 resourceType := parts[2]
 resourceName := parts[3]
 resourceAction := parts[4]

 if resourceAction == "creating" {
 c, s := beeline.StartSpan(cw.ctx, resourceName)
 beeline.AddField(c, "type", resourceType)
 // add other things here

 cw.contexts[resourceName] = s.Send
 }

 if resourceAction == "created" {
 cw.contexts[resourceName]()
 }

 return len(p), nil
}

Modifying the optup.ProgressStreams is the only change needed to the original application:

stream := optup.ProgressStreams(os.Stdout, NewPulumiBeeline(ctx))
if _, err := stack.Up(ctx, stream); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
}

When I rerun this program, I can see a lot more information in my Honeycomb traces, which shows me that Pulumi is highly parallelized and provides a better idea of where the time is spent when creating infrastructure. In this example, it’s the NAT Gateways:

In the future, I want to expand this to cover far more details. Among those details are why resources were created/modified/destroyed and collecting more information on what causes resources to fail.

Wrapping Up

In the end, this turned out to be much easier to achieve than I had hoped. Using Pulumi programmatically was a huge productivity boost, instead of running os.Exec directly.

I am looking forward to all the new kinds of tooling I can build to solve my user’s problems by continuing to utilize Honeycomb for my observability and Pulumi for my infrastructure.

Credijusto Manages Authentication with Auth0 and Pulumi

Fernando Carletti — Tue, 22 Sep 2020 00:00:00 +0000

Guest author Lead Devops Engineer Fernando Carletti, writes about using the Pulumi Auth0 provider to manage resources at Credijusto.

Auth0 allows you to simplify your authentication process. The Auth0 Provider allows you to manage the Auth0 resources, managing Applications, Databases, Social Connections, APIs, and other resources. Here at Credijusto we use it manage authentication from the front-end through all the APIs that serve that request, leveraging the complexity of the authentication to Auth0.

For this article, we will start a new Pulumi project in a fresh Auth0 account and fully configure it for a backend and a single page application and set up a connection to Github which allows you apps to authenticate with it using OAuth.

Setup the project

Create a new Pulumi program:

$ pulumi new typescript

Install the Auth0 provider SDK:

$ npm add @pulumi/auth0

Create the Auth0 resources in index.ts you created as shown in the example below.

Configure the credentials

Here we will use the credentials of the default application created by Auth0. For production use, you may want to create a new one or rename the Test Application to ensure Pulumi uses its own set of credentials.

Note: please use your domain, clientId, and clientSecret. You can find the Test Application credentials in Auth0’s Applications page.

pulumi config set auth0:domain my-account.auth0.com
pulumi config set auth0:clientId foo
pulumi config set auth0:clientSecret --secret bar
pulumi config set githubClientId github-foo
pulumi config set githubClientSecret --secret github-bar

Tip: You can create your Github OAuth Application here.

Configure the Tenant

Auth0 creates an initial tenant when a new account is created. Any tenant resource you create will end up configuring the tenant of which the Test Application is part of, even while Pulumi is saying a new resource is created, it will only set the properties on its existing tenant.

Let’s set a friendly name for our tenant:

new auth0.Tenant('default', {
 friendlyName: 'Hello Pulumi!'
});

By accessing your settings page you should be to see the Friendly Name field configured.

Create the applications

Now we need to create our applications. Since the Auth0 provider uses the same naming as the Management API, the Application is called Client.

Let’s create both the frontend and backend applications:

const backend = new auth0.Client('backend', {
 appType: 'non_interactive'
});

const frontend = new auth0.Client('frontend', {
 appType: 'spa'
});

Note that the backend is set as the non_interactive type, this is referred to as Machine to machine in the Auth0 console.

Create the Github connection

With the applications set, we need to create a social connection using Github and allow those applications to use it:

const config = new pulumi.Config();

new auth0.Connection('github', {
 strategy: 'github',
 enabledClients: [backend.id, frontend.id],
 options: {
 clientId: config.require('githubClientId'),
 clientSecret: config.requireSecret('githubClientSecret'),
 }
});

Pulumi allows you to manage your Auth0 resources allowing you to easily replicate your configuration across multiple environments. Read more about the Auth0 Provider.

How Pinpoint Manages Kubernetes Costs and Deployments

Andrew Kunzel — Mon, 17 Aug 2020 00:00:00 +0000

This guest blog was contributed by Andrew Kunzel and Michael Goode of Pinpoint. Andrew is the Director of Backend Engineering, and Michael is a Platform Operations Engineer.

At Pinpoint, Kubernetes is the most powerful tool in our arsenal. It allows us to deploy and rapidly scale our applications with speed and efficiency that continues to delight our customers. In recent years, managed services like AWS EKS have made it easier than ever to leverage the power of Kubernetes in even the smallest of organizations. Yet even with these new conveniences, managing all of this infrastructure can be a daunting task. Right out of the gate, we knew that we wanted to avoid the burden of maintaining repositories full of home-brewed deployment scripts and domain-specific languages like YAML.

Pulumi affords us the ability to streamline the development of our cloud infrastructure with open source libraries that integrate deeply into AWS service APIs and support many popular languages. This introduces powerful, high-level constructs like for-loops and strong type-checking alongside the more rigid declarative approach mandated by other available tools. And it’s not just limited to hardware! We also use Pulumi to substantially reduce the overhead required to deploy critical services that run on Kubernetes itself, which helps to consolidate our deployment configuration even further and makes updating these services a breeze.

In this blog post, we’ll share some simple and intuitive ways in which Pulumi has simplified our approach to infrastructure-as-code.

Spot Instances

As a startup, we are always budget-conscious. Every tool we use has to justify either the up-front subscription cost or development time, or both. To tamp down our AWS EC2 costs, we decided to leverage EC2 spot instances as worker nodes in our development Kubernetes cluster. Spot instances are highly cost-efficient because they use spare compute capacity within a region at a fraction of the on-demand instance price. When you request a spot instance, you must offer the maximum price you are willing to pay. This price can fluctuate based on demand in a particular region, but will always be less than or equal to the current on-demand price. Knowing this, we could use Pulumi’s JavaScript AWS library to get the current on-demand price for a given instance type and plug that directly into the spotPrice property of the node group configuration.

function getOnDemandPrice(instanceType: aws.ec2.InstanceType): pulumi.Output<any> {
 return pulumi.output(aws.pricing.getProduct({
 filters: [
 {
 field: "instanceType",
 value: `${instanceType}`,
 },
 {
 field: "operatingSystem",
 value: "Linux",
 },
 {
 field: "location",
 value: "US East (N. Virginia)",
 },
 {
 field: "preInstalledSw",
 value: "NA",
 },
 {
 field: "usageType",
 value: `BoxUsage:${instanceType}`,
 },
 {
 field: "termType",
 value: "OnDemand",
 },
 ],
 serviceCode: "AmazonEC2",
 }, { async: true })).apply(data => {
 let result = JSON.parse(data.result)
 let typeOnDemand = result.terms.OnDemand
 let skuTermCode = Object.keys(typeOnDemand)[0]
 let rateCode = Object.keys(typeOnDemand[skuTermCode].priceDimensions)[0]
 let price = typeOnDemand[skuTermCode].priceDimensions[rateCode].pricePerUnit.USD
 return price
 })
 }

 cluster.createNodeGroup(`${stack.env}-${name}`, {
 amiId: ami,
 version: kubernetesVersion,
 instanceProfile: nodeGroupInstanceProfile,
 minSize: 1,
 maxSize: 10,
 desiredCapacity: 5,
 nodeRootVolumeSize: 100,
 nodeAssociatePublicIpAddress: false,
 nodeSubnetIds: stack.privateSubnets,
 instanceType: instanceType,
 spotPrice: getOnDemandPrice(instanceType)
 })

And just like that, we can quickly provision spot instances on our cluster! Better yet, we don’t ever have to worry about setting a valid spot price or managing this process via custom scripts in a CI/CD pipeline. The entire deployment is easily managed within the same file.

Managing Third-Party Services

Pulumi also helps us to manage the services that we run on our Kubernetes clusters by allowing us to efficiently configure these services consistently that avoid drift and minimizes complexity. Pulumi is extremely flexible in this regard because it offers native support for de-facto tools like Helm and Kustomize as well as individual Kubernetes components like pods and deployments. You can even render raw YAML, which greatly simplifies integration with legacy systems or custom components.

Transformations with Helm

We prefer to use Helm for most of our deployments. It is an excellent tool for deploying third party services and customizing them with a useful but complicated templating system. These templates are great when they work, but if the chart’s maintainer doesn’t make something configurable, you’re usually unable to alter the manifest when it comes time to deploy your chart. You’d likely have to fork and manage a custom version of the chart yourself in the worst case. Luckily, Pulumi can override just about anything in a manifest by using a Transformation. This property allows you to manipulate a manifest’s content before it is rendered and applied to the cluster.

As an example, we recently discovered that some versions of the Traefik Helm chart do not include an explicit namespace definition on certain templates. This caused a few resources to install into the default namespace; without Pulumi, we would have been unable to fix this issue without forking the chart or writing a post-install script to wrangle them back. Luckily, all we really had to do was create a function to set the object metadata and then pass that function directly into the ‘transformations’ property of the Traefik Helm chart:

 function setNamespace(obj: any) {
 if (obj.metadata.namespace === undefined) {
 obj.metadata.namespace = 'platform-services';
 }
 }

 new k8s.helm.v3.Chart("traefik", {
 repo: "traefik",
 chart: "traefik",
 version: "6.0.0",
 namespace: 'platform-services',
 transformations: [setNamespace],
 values: {
 deployment: {
 replicas: 3
 },
 },
 }, { provider: provider });

This will run the transformation on every manifest used by the chart, fixing our issue in only a few code lines.

Being able to manage all of our infrastructure as code and enhance the behavior of standard Kubernetes deployment tools has dramatically improved our ability to deploy services and maintain our configuration. We look forward to expanding our use of Pulumi as we grow as an organization and empower our developers further.

Test and Optimize Elasticsearch with Pulumi

Itamar Syn-Hershko — Thu, 05 Mar 2020 00:00:00 +0000

Guest Article: Itamar Syn-Hershko, Founder and CTO of BigData Boutique shows how they use Pulumi to benchmark Elasticsearch configurations across cloud providers. Pulumi enables BigData Boutique to test deployments in parallel and gather metrics to produce performant and cost-effective solutions for its customers.

BigData Boutique can rapidly iterate through infrastructure configurations based on client requirements. Familiar programming languages significantly shortens the inner dev loop because developers and operators can use modern programming tools such as IDEs, test frameworks, and CI/CD. By increasing the velocity of infrastructure deployment across many clouds with repeatable and versioned code, Pulumi enables companies to build efficient revenue centers for their business.

How Pulumi Drives Our Elasticsearch Capacity Planning and Cost Optimization Service

Capacity Planning and Cost Optimization of Elasticsearch clusters requires a special level of expertise and automation. Here is how we use Pulumi to launch long-running benchmarks to correctly identify the right configuration for our customers’ Big Data clusters.

At BigData Boutique, we are continually challenged by our customers—whether it’s complex Big Data challenges we are asked to solve, reducing costs for sophisticated systems holding mountains of data, improving performance for high-demand systems, or challenges faced in building a data-driven system.

One of the main challenges our customers are facing with Big Data systems is deploying the right hardware at the right scale. They need to know how to correctly estimate the number of servers required to run their compute workloads and databases, so they don’t pay more than they should, while still keeping those services running without interruption and within the defined KPIs.

To support our customers and help them succeed in their Big Data projects, we provide an array of advanced services, and our unique cost-optimization service addresses this challenge directly. This article outlines how we use long-running benchmarks to correctly identify the right configuration for our customers’ BigData clusters by using the infrastructure as code tool Pulumi for automating the creation of complex infrastructures with conditionals and highly tunable configurations to orchestrate those benchmarks at scale.

Elasticsearch Cost Optimization

Let’s take Elasticsearch as an example. Elasticsearch is a real-time search and analytics engine. It is built to ingest huge amounts of data and to allow one to search through and visualize them in real-time. But to be able to sustain the required amount of data on ingestion and the desired number of queries per second, the right hardware has to be used.

To do this, we have to make some decisions: which disks to use (SSD disks? local, or network storage?), the number of servers (scaling up or out?), CPU and RAM ratios, index mappings, various cluster configurations, and more. These are not easy questions to answer, and that is where our Elasticsearch expertise comes into play. But answering those questions correctly with a high level of certainty also requires rigorous testing for validating assumptions.

Whether our customer is running on-prem or on a cloud, our Elasticsearch Capacity Planning Service exists for one purpose: to find the hardware solution with the optimum balance between cost and performance, and do it scientifically, so the answers are as accurate and as precise as possible.

Elasticsearch Sizing Process

Our process begins with a sample data request. We ask the customer for their data (or a large sample of it), existing index mappings, expected queries, and finally known KPIs and SLAs. Our goal in the initial discovery phase is to reduce the number of free variables as much as possible because the more such variables, the more permutations we will have to test. Ideally, we want very few dimensions of freedom — usually in terms of hardware and configurations—so the number of permutations to test is on the order of dozens and not thousands.

Once the initial framework has been decided, we run fully automated benchmarks to test some of the permutations our experience tells us are the most promising to deliver good results. The benchmarking process is smart enough to omit long tails and outliers, so the results are not going to lie as to the true nature of the configuration that was selected and tested.

Between benchmark iterations, our team uses its years of experience to create more test configurations that optimize various configurations we decide are important based on the results of the previous benchmarks. After several iterations of benchmarking on various configurations, we send the preliminary findings to confirm our results and proposed direction to the customer. The purpose is to communicate the results and validate the trade-offs we decide on during the sizing procedure. Finally, we provide a summary session with our team of Elasticsearch experts and customer technical and business contacts to present the results and recommendations, and to explain the reasoning behind the analysis.

Finding the best Elasticsearch configuration requires careful benchmarking and tech expertise. Yet trade-offs will come into play. For instance, it might seem that the only way to gain another 100ms of performance improvement for a certain specific query would be to spend another $500 per month on hardware. But there may be other options. Our Sizing Service takes the guesswork out of the process. We automate the entire benchmarking procedure in order to make it an exact science.

To fully automate the benchmarking process, we use Pulumi.

Using Pulumi for State-of-the-Art Infrastructure Automation

Pulumi is an infrastructure-as-code tool that allows us to define and run infrastructures using code, as opposed to doing this manually or using other tools that use configuration-based syntax. This lets us create complex infrastructures with conditionals and highly tunable configurations. The Pulumi program we created can be run numerous times with different parameters. Each result is a cluster that is built and configured differently, sometimes even on completely entirely different clouds.

At BigData Boutique, we use that concept to launch multiple infrastructures in parallel, benchmark them, and show a report that highlights which config is better along with the various trade-offs for each. After the benchmark process, the cluster is destroyed, and the “provisioner” instance reports all logs, metrics, and the benchmark report, which we then analyze and visualize.

Generating, visualizing, and analyzing dozens of reports in parallel gives us, the Elasticsearch Engineers, the ability to understand which knobs to turn and what configurations to change to make our customers succeed in their project without spending more than they are willing to.

Conclusion

In sum, leveraging Pulumi helps us to create highly configurable infrastructures to orchestrate benchmarks at scale. As one of many advanced services that we offer, the Elasticsearch Capacity Planning and Cost Optimization Service continues to meet our customers’ most important planning and optimizations needs in an as scientifically rigorous manner as possible. You can read more about it here.

The original article was published on their company blog.

Three Infrastructure as Code Blog Posts You Should Read

Sophia Parafina — Thu, 24 Oct 2019 00:00:00 +0000

We are always excited when people join the Infrastructure as Code community and write about their experiences. Pulumi can be used for a range of common tasks such as standardizing VPC builds, building VSphere virtual machines, or deploying your infrastructure from a CI/CD pipeline. Whether it’s TypeScript, JavaScript, or Python you can build your infrastructure with your language and tools of choice. Here are three new blog posts that show how to use Pulumi with code examples to perform these tasks.

CircleCI on Google Kubernetes Engine (GKE)

Angel Rivera shows how to set up CircleCI on GKE with a step-by-step tutorial. Angel demonstrates how to integrate Infrastructure as Code solutions, such as Pulumi, into CI/CD pipelines. The tutorial builds a container with a Pulumi project with a sample app written in Python that deploys Kubernetes. With Pulumi, your infrastructure is part of your CI/CD. The code for the tutorial is available on GitHub.

HATech Compares Pulumi to Terraform

Jon Hathway of HATech published a white paper comparing Pulumi to Terraform. Changes in Terraform prompted the comparison. Below, Jon explained why HATech is evaluating Pulumi:

“Take for example the teams that are trying to collaborate and drive reliably deployed and maintained product innovations. Is it better to have languages that support each individual team’s needs, or is it better to have one language to support the entire workflow/pipeline?

How about this thought experiment: Imagine your developers only spoke English, your QA team only spoke French, and the operations team only spoke Spanish? It would be carnage!

You’d never hire that way. So why, when choosing DevOps tools, do people find it acceptable to have different languages, DSLs that only key individuals understand?”

VMWare VSphere and Pulumi

Cody De Arkland demonstrates how to create a VMWare VSphere Virtual Machine by adding a ResourcePool, DataStore, NetworkStore and the virtual machine template with Pulumi. He also sets the configuration values such as DNS settings, default gateway, domain, and hostname all in code. All of this runs on the ESXi virtualization server and vCenter (which is why the Datacenter ID is needed for configuration). His example is available on GitHub. We like Cody’s caveat that his post is not an endorsement but call to explore new tools. In his conclusion, he nails home the point why Infrastructure as Code is the future of DevOps.

“Infrastructure as Code tools like Pulumi (and others in the space, i.e. Terraform from HashiCorp) add a ton of value on top of traditional deployment methods. Being able to store your infrastructure deployments in source control (i.e. GitHub/GitLab) opens up the door to a number of collaboration concepts. When you partner this in with some form of a CI/CD tool (i.e. CircleCI) - things get really interesting. Being able to submit a PR to change an infrastructure deployment, and having something like CircleCI automatically deploy the resources when changes are merged creates a pretty powerful platform for delivering infrastructure to end-users.”

Bloggers Welcome

If you write about your experience using Pulumi, let us know by tagging us on Twitter.

Mapbox IOT-as-code with Pulumi Crosswalk for AWS

Chris Toomey — Wed, 12 Jun 2019 00:00:00 +0000

Guest Author: Chris Toomey, Solution Architect Lead @ Mapbox

With 8 billion+ connected IoT devices and 2 billion GPS-equipped smartphones already online, logistics businesses are tracking assets at every step in the supply chain. At this scale and complexity, it is imperative to have a flexible way to ingest, process, and act upon this data, without sacrificing security or best practices.

To meet this need, Mapbox has created an Asset Tracking Solution that uses Pulumi’s open source JavaScript libraries (AWS, AWSX) available with multi-language support with Pulumi Crosswalk for AWS. Pulumi Crosswalk for AWS is an open source framework that streamlines creation, deployment and management of AWS services with built-in AWS Best Practices and minimal lines of code in common programming languages.

In this blog, we will show snippets of the Javascript code that embraces the power of Pulumi to program AWS service APIs to create the Mapbox solution. To see the full architecture in action with a live bike race across America, please refer to this webinar recorded on June 13th 2019 and the Mapbox asset tracking solution. Also refer to this blog of the Race across America showcased live during the webinar tomorrow.

Prerequisites: Install Pulumi; Install Node.js and Setup AWS

The diagram represents how Mapbox’s solution design on AWS services is built with Pulumi AWS and AWSX libraries in Javascript. Data is ingested in the Asset tracking IOT solution with a REST API. Mapbox uses its stream processor to perform enrichments such as geofencing, traffic-aware ETA calculations, and high-prevision elevation. This data, exposed by the API is backed by a high performance database (DynamoDB) to enable visualization in realtime on the map client.

The fun part – Pulumi Service console helps you map your cloud architecture shown above as a connected set of DAG resources so you need not remember what you deployed as your cloud environments scale:

Lets now work through the solution flow with some sample snippets of code. To get access to the complete piece of code, please connect with Mapbox solution team here and Pulumi team here. We will be happy to help you sample test this solution on AWS!

STEP 1: Create a Pulumi project using an AWS JavaScript template

$ brew install pulumi/tap/pulumi
$ mkdir asset-tracking && cd asset-tracking
$ pulumi new aws-javascript
$ yarn install
$ ls
Pulumi.dev.yaml Pulumi.yaml index.js node_modules package-lock.json package.json

STEP 2: Create the ingestion REST API with AWS Services

To visualize data the system has to accept large volumes of data from multiple sources into the rest of the system. Mapbox uses Pulumi’s AWS & AWSX libraries to build an IOT ingestion rule and forwards it to Kinesis streams. A sample index.js looks like this:

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");
const awsx = require("@pulumi/awsx");

// Create Kinesis stream for ingestion
const ingestStream = new aws.kinesis.Stream("ingestAssets", {
 shardCount: 1,
 retentionPeriod: 72
});

// Create IoT Rule to push into Kinesis stream
const iotRole = new aws.iam.Role("iotRole", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 Service: "iot.amazonaws.com"
 },
 Action: "sts:AssumeRole"
 }
 ]
 })
});

const iotRolePolicy = new aws.iam.RolePolicy("iotRolePolicy", {
 policy: pulumi.interpolate`{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "kinesis:*"
 ],
 "Resource": "${ingestStream.arn}"
 }
 ]
 }`,
 role: iotRole.id
});

const iotRule = new aws.iot.TopicRule("iotTrigger", {
 description: "Pass from IoT Core to Asset Tracking",
 name: "iotAssetIngest",
 enabled: true,
 kinesis: {
 partitionKey: "id",
 roleArn: iotRole.arn,
 streamName: ingestStream.name
 },
 sql: "SELECT * FROM 'topic'",
 sqlVersion: "2015-10-08"
});

Kinesis streams are used to ingest data into multiple AWS services defined as IOT rules, Firehose and Lambda functions. Each ingest stream has the right IAM role and policy to allow Kinesis to send the data into these AWS services.

The code above does not shown the Javascript code for Lambda functions used by Mapbox streamprocessor and Firehose all created with Pulumi AWS and AWSX libraries.

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

STEP 3: Create a consumption API for the map client to consume from a data source

Once, data is flowing through the system we build another API to query DynamoDB, transform that data, and provide it to our mapping client in geojson. With Pulumi AWSX, we create a APIGateway endpoint API that can query this data out of DynamoDB. The sample code in index.js will look like this:

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

// Create DynamoDB Table
const assetTable = new aws.dynamodb.Table("assetTable", {
 attributes: [
 {
 name: "id",
 type: "S"
 },
 {
 name: "ts",
 type: "N"
 }
 ],
 hashKey: "id",
 rangeKey: "ts",
 ttl: {
 attributeName: "expiration",
 enabled: true
 },
 billingMode: "PAY_PER_REQUEST"
 });

// Create API to read DynamoDB
const endpoint = new awsx.apigateway.API("assetQuery", {
 routes: [
 {
 path: "/",
 method: "GET",
 eventHandler: (request, ctx, cb) => {
 const AWS = require("aws-sdk");
 const ddb = new AWS.DynamoDB.DocumentClient({
 apiVersion: "2012-10-08"
 });
 const tableName = assetTable.name.value;
 const params = {
 TableName: tableName
 };
 ddb.scan(params, (err, data) => {
 const features = data.Items.map(item => {
 const point = turf.point([item.longitude, item.latitude], {
 id: item.id,
 speed: item.speed
 });
 return point;
 });
 const featureCollection = turf.featureCollection(features);
 cb(undefined, {
 statusCode: 200,
 body: Buffer.from(
 JSON.stringify(featureCollection),
 "utf8"
 ).toString("base64"),
 isBase64Encoded: true,
 headers: { "content-type": "application/json" }
 });
 });
 }
 }
 ],
 stageName: "dev"
});

Once we’ve created the API, we start defining our routes and then in-line we can define a Lambda function. It can use NPM modules and Pulumi will handle bundling all the dependencies.

This endpoint assumes we are querying Dynamo - and so we are going to use the DynamoDB Document Client. Using that straightforward scan syntax, we query the table and then start to parse our results. Since every data point has a latitude and longitude as GeoJSON, the data is transformed using Turf.js before sending to the client. Even more beautiful is how Pulumi handles all the heavy lifting of packaging, deploying, and giving you the endpoint to query immediately.

STEP 4: Hook up a map!

Now we have data coming and going from our Solution, we just have to hook it up to a Mapbox map. This is actually quite simple now that Pulumi has built us a REST API.

Following this example, we paste in the endpoint from Pulumi and pick our interval.

Additional Resources

For more information from Mapbox check out the Mapbox Blog. For more solutions from Pulumi please visit our Blog or head over to the Pulumi Examples repository. For AWS solutions visit the AWS partner blog site.

Managing your MySQL databases with Pulumi

Linio Engineering — Tue, 28 May 2019 00:00:00 +0000

One of the most critical components of an application’s infrastructure is its database, and one of the most popular databases in use in the cloud today is MySQL.

Pulumi can already be used to create managed MySQL instances in a wide variety of clouds, including AWS, Azure and GCP. In addition to this, Pulumi recently added support for managing the MySQL instances themselves to manage permissions, create databases, and other common tasks.

In this post, we’ll walk through a quick tutorial of how to use this new Pulumi MySQL provider to manage existing and new MySQL databases.

Setup

Create a new Pulumi program:

$ pulumi new aws-typescript

Install the MySQL provider SDK:

$ npm add @pulumi/mysql

Create the database in index.ts as shown in either of the two examples below.

Examples

Example 1: Using an existing MySQL server

In this scenario, the following configuration is required:

Note: please use your own endpoint, username and password.

$ pulumi config set mysql:endpoint localhost:3306
$ pulumi config set mysql:username root
$ pulumi config set --secret mysql:password foo
$ pulumi config set --secret jdoePassword hunter2

In this example, we will create a database in the previously configured MySQL server, add a user (jdoe) with the given password and grant this user SELECT and UPDATE access to it.

import * as mysql from '@pulumi/mysql';
import * as pulumi from '@pulumi/pulumi';

const config = new pulumi.Config();
const jdoePassword = config.requireSecret(‘jdoePassword’);

const database = new mysql.Database('sample', {
 name: 'sample',
});

const user = new mysql.User('jdoe', {
 user: 'jdoe',
 // Since the password is a Pulumi Secret, it will be encrypted and not stored in plaintext
 plaintextPassword: jdoePassword,
});

new mysql.Grant('jdoe', {
 user: user.user,
 host: user.host,
 database: database.name,
 privileges: ['SELECT', 'UPDATE'],
});

Now deploy the infrastructure to provision the database and user:

$ pulumi up

Example 2: Using a RDS instance

In this scenario, the following configuration is required:

$ pulumi config set mysqlUsername rootUser
$ pulumi config set --secret mysqlPassword rootPassword

In this example, we will create an AWS RDS Instance to be our MySQL server, create a “first-class” provider to manage the MySQL database, and add a user (jdoe) with the given password and grant this user SELECT and UPDATE access to it:

Note: This requires the default VPC security group to allow access from where the Pulumi deployment is being executed.

import * as mysql from '@pulumi/mysql';
import * as aws from '@pulumi/aws';
import * as pulumi from '@pulumi/pulumi';

const config = new pulumi.Config();
const mysqlUser = config.require('mysqlUser');
const mysqlPassword = config.requireSecret('mysqlPassword');

const rds = new aws.rds.Instance('sample', {
 engine: 'mysql',
 username: mysqlUser,
 password: mysqlPassword,
 availabilityZone: 'us-east-1b',
 instanceClass: 'db.t2.micro',
 allocatedStorage: 20,
 deletionProtection: true,

 // For a VPC cluster, you will also need the following:
 // dbSubnetGroupName: 'sg-db01-replication-1',
 // vpcSecurityGroupIds: ['sg-c1c63aba'],
});

const mysqlProvider = new mysql.Provider('mysql', {
 endpoint: rds.endpoint,
 username: rds.username,
 password: rds.password,
});

const database = new mysql.Database('sample', {
 name: 'sample',
}, {
 provider: mysqlProvider
});

const user = new mysql.User('jdoe', {
 user: "jdoe",
 host: "example.com",
 plaintextPassword: "password",
}, {
 provider: mysqlProvider
});

new mysql.Grant('jdoe', {
 user: user.user,
 host: user.host,
 database: database.name,
 privileges: ["SELECT", "UPDATE"],
}, {
 provider: mysqlProvider
});

Now deploy the infrastructure:

$ pulumi up

Wrapping Up Pulumi allows you to manage your MySQL cloud instances in AWS, Azure, and GCP, as well as manage MySQL databases, users and more. Together, this enables end-to-end provisioning of your application’s database infrastructure. Read more about the Pulumi MySQL provider.

We’ve only shown a little bit of what Pulumi can do. If you need any help, feel free to create an issue on GitHub or join the Pulumi Community Slack channel.

Data science on demand: spinning up a Wallaroo cluster

Marc Holmes — Fri, 02 Nov 2018 00:00:00 +0000

This guest post is from Simon Zelazny of Wallaroo Labs. Find out how Wallaroo powered their cluster provisioning with Pulumi, for data science on demand.

Last month, we took a long-running pandas classifier and made it run faster by leveraging Wallaroo’s parallelization capabilities. This time around, we’d like to kick it up a notch and see if we can keep scaling out to meet higher demand. We’d also like to be as economical as possible: provision infrastructure as needed and de-provision it when we’re done processing.

If you don’t feel like reading the post linked above, here’s a short summary of the situation: there’s a batch job that you’re running every hour, on the hour. This job receives a CSV file and classifies each row of the file, using a Pandas-based algorithm. The run-time of the job is starting to near the one-hour mark, and there’s concern that the pipeline will break down once the input data grows past a particular point.

In the blog post, we show how to split up the input data into smaller dataframes, and distribute them among workers in an ad-hoc Wallaroo cluster, running on one physical machine. Parallelizing the work in this manner buys us a lot of time, and the batch job can continue processing increasing amounts of data.

Sure, we can handle a million rows in reasonable time, but what if the data set grows by orders of magnitude? By running our classifier on a local Wallaroo cluster, we were able to cut the processing time of a million rows to ~16 minutes, thus fitting within our allotted time slot of one hour. But if the input data is, say, 10x more, we’re going to have a hard time processing it all locally.

Let’s see how we can keep up with the data growth by launching a cloud-based Wallaroo cluster on-demand, running the job, collecting the data, and shutting down the cluster, all in a fully automated fashion.

Tools of the trade

Wallaroo’s big idea is that your application doesn’t have to know whether it’s running on one process, several local processes, or a distributed system comprising many physical machines. In this sense, there’s no extra work involved in ‘migrating’ our classifier application from the previous blog post.

We will need some tools to help us set up and manage our cluster in the cloud. Wallaroo can work with a lot of different tools. Our friends at Pulumi provide an excellent tool that removes the headaches involved in provisioning infrastructure. We’ll use Pulumi to define, set up, and finally tear down our processing cluster in this example.

We’ll also need Ansible to start, stop, and inspect the state of our cluster, and, last but not least, we’ll need an AWS account where our machines will live.

Let’s jump into it!

A sample run

First of all, if you’d like to follow along (and spend some money provisioning EC2 servers), please download and set up Pulumi.

Next, clone the wallaroo blog examples repo and navigate to provisioned-classifier. If you followed along with the previous Pandas blog post, you’ll find our old application nested away here, underclassifier. What’s more interesting are the two new directories: pulumi and ansible.

Without delving into details, let’s see how to run our application on a freshly-provisioned cluster in the EC2 cloud:

make up run-cluster get-results down INPUT_LINES=1000000 CLUSTER_SIZE=3

Let’s break that down and see what’s really going on here.

make up CLUSTER_SIZE=3 configures the cluster to consist of 3 machines, and delegates to pulumi up the actual business of spinning up the infrastructure. Our physical cluster will contain 3 nodes for processing, and one extra metrics_host node for hosting our Metrics UI, and collecting results.
Once provisioning is complete, the next make task: run-cluster INPUT_LINES=1000000 uses our Ansible playbooks to upload application code from classifier/* to all 3 machines provisioned above, and then start up a Wallaroo cluster with 7 worker processes per machine.
Next, Ansible starts sending 1 million lines of our synthetic CSV data, and waits for 1 million lines to arrive at the data_receiver process. When those lines arrive, they are compressed, and the cluster is shut down.
make get-results pulls the compressed result file to output/results.tgz,
And finally, make down destroys the cloud infrastructure that was used to power our computation.

The Pulumi cluster definition

Let’s take a look at how our infrastructure is defined. This is the core of the definition:

 function instance(name) {
 return new aws.ec2.Instance(
 name,
 {associatePublicIpAddress: true,
 instanceType: instanceType,
 securityGroups: [secGrp.name],
 ami: ami,
 tags: {"Name": name},
 keyName: keyPair.keyName})
 }

 let metrics_host = instance("classifier-metrics_host");
 let initializer = instance("classifier-initializer");
 let workers = [];
 for(var i=0; i<clusterSize-1; i++){
 workers.push(instance("classifier-"+(i+1).toString()));
 }

As you can see from the above, our little instance() function encapsulates the common settings for every machine that we want to provision.

The metrics_host and initializer are ec2.Instance objects with descriptive names, while the workers are ec2.Instances that are distinguished solely by their ordinal number. Pulumi lets us define – in code – things like Security Groups, SSH keypairs, and practically every other aspect of cloud infrastructure.

An example of this capability is the keyPair object that’s used to access the instances via SSH. Our Makefile ensures that an ssh key is generated on-the-fly for our cluster, and Pulumi knows how to use it to set up SSH access for newly-provisioned nodes:

let pubKey = fs.readFileSync("../ssh_pubkey_in_ec2_format.pub").toString();
let keyPair = new aws.ec2.KeyPair("ClassifierKey", {publicKey: pubKey});

With the relevant bits of our computing infrastructure thus defined, we can tell Pulumi to take action in the real world and make it conform to our definition:

When we run make up CLUSTER_SIZE=3, we’ll see Pulumi output something like the following:

Performing changes:
+ pulumi:pulumi:Stack classifier-classifier-demo creating
+ aws:ec2:KeyPair ClassifierKey creating
(...)
+ aws:ec2:Instance classifier-2 created
---outputs:---
metrics_host: [
[0]: {
name : "classifier-metrics_host"
private_ip: "172.31.47.236"
public_dns: "ec2-54-245-53-87.us-west-2.compute.amazonaws.com"
}
(...)
]
info: 7 changes performed:
+ 7 resources created
Update duration: 1m59.524017637s

We can use Pulumi’s output to stitch together an Ansible inventory, which will let us interact programmatically with our provisioned instances.

If we want to modify our cluster, we can edit pulumi/index.js, and then rerun make up CLUSTER_SIZE=N. If the changes don’t require restarting the instances (for example if they only concern the Security Group), Pulumi will do the right thing and not disturb the rest of the infrastructure.

If you examine the Pulumi definition file in depth, you’ll see that it relies on a mystery AMI: ami-058d2ca16567a23f7. This is an experimental ubuntu-based image, with Wallaroo binaries (machida, cluster_shutdown,cluster_shrinker, and data_receiver) pre-loaded. For now, it only exists in the us-west-2 AWS region, but we hope to make Wallaroo AMIs available for experimentation in all regions starting with the next Wallaroo release.

Running the computation

Now that we know what magic powers conjured up our AWS infrastructure, let’s take a look at how we use it to run our task. Fundamentally, the components of our cluster can be described as follows:

The data source – In our case it’s the file send.py, which can generate and transmit randomly-generated CSV data for our computation to consume.
The machida processes: one Initializer and a bunch of Workers. The distinction between the two is only relevant at cluster startup.
The data receiver – A process that listens on a TCP port for the output of our computation. This is the data_receiver, provided as part of a Wallaroo installation.
The metrics UI – Our Elixir-powered realtime dashboard.

Our Ansible playbook takes care of coordinating the launch of the various components and making sure that their input, output and control ports match up. In particular, that the cluster initializer starts up knowing the total number of workers in the cluster, and every other worker connects to the initializer’s internal IP and control port.

This is what ends up running on the servers when we launch our Ansible playbooks:

Once the cluster is up and running, and the initializer node’s tcp source is listening for connections, we start up the sender and instruct it to send a stream of data to the TCP Source. In a realistic batch scenario, this sender could be implemented as a Connector that reads a particular file from a remote filesystem or S3. For our purposes, we’ll simulate this by generating a set number of CSV lines on-demand, and then shutting down.

While the work is being performed, we can take a look at the metrics URL printed out on the screen to find out how the work is being distributed.

To see the cluster’s real-time metrics, please visit http://ec2-54-200-198-6.us-west-2.compute.amazonaws.com:4000

In the screenshot above, you can see that the Initializer and B03a909b23nodes are processing about 4k messages per second each, and all the other workers have the classification work split evenly among them. Don’t be surprised that two of the workers are processing orders of magnitude more messages! Remember our pandas application pipeline?

 ab.new_pipeline("Classifier",
wallaroo.TCPSourceConfig(in_host, in_port, decode))
ab.to_stateful(batch_rows, RowBuffer, "CSV rows + global header state")
ab.to_parallel(classify)
ab.to_sink(wallaroo.TCPSinkConfig(out_host, out_port, encode))
return ab.build()

The CSV rows come in one-by-one, but we batch them and convert them to dataframes of a hundred, so that our classification algorithm can tackle more than one row at a time. Worker B03a909b23 just happens to be the worker where our state named “CSV rows + global header state” lives. Let’s take a look at its metrics:

Indeed, we can see that this worker is processing about 4k/sec messages in the “Batch Rows Of Csv, Emit Dataframes” step. Every other worker is busy classifying! Let’s see the breakdown for a different, random worker:

Looking good. All that’s left for us to do is wait until the job completes and we receive our zipped data onto our disk.

The numbers!

As a reminder, let’s take a look at the numbers we obtained by running our classifier on a single c5.4xlarge instance in AWS:

# SINGLE-MACHINE RUNNING TIMES (NO PROVISIONING)
CSV rows 1 worker 4 workers 8 workers
----------- ---------- ----------- -----------
10 000 39s 20s 11s
100 000 6m28s 3m16s 1m41s
1 000 000 1h03m46s 32m12s 16m33s

Now, let’s see how much speedup we can achieve from scaling out with our provisioned-on-demand infrastructure.

# MULTI-MACHINE RUNNING TIMES (PROVISIONING + COMPUTATION)
----------------------------------------------------------
CSV rows 4 machines/ 8 machines/ 16 machines/
28 workers 56 workers 112 workers
------------ -------------- -------------- ---------------
10 000 2m13s 2m36s 2m43s
100 000 3m38s 3m42s 3m48s
1 000 000 7m38s 6m41s 5m56s
10 000 000 40m56s 33m10s 23m24s
30 000 000 > 2h 1h45m 1h12m
----------------------------------------------------------

Even though there is some constant overhead involved in spinning up the required infrastructure (clearly too much overhead to justify spinning up clusters for less than 1 million rows), we’re now able to classify a hefty 10 million rows of CSV data in under half an hour, and 30 million in a little over an hour. This gives us some perspective on when our application will need extra resources, or perhaps some performance optimizations.

As long as the data fits in the 1 million : 10 million range, it seems that a cluster of 4 machines represents a sweet-spot between price and performance – we can process the data and still fit in the hour-long window allotted for our batch process, but not have to incur unnecessary infrastructure costs if we don’t need the extra speed.

Conclusion

The above figures illustrate how Wallaroo can be used as an ad-hoc compute cloud, using Pulumi and Ansible to provision, run workloads remotely, and shut down the infrastructure once the results are in.

In the case or our batch job, we can leverage this pattern to scale horizontally on-demand, even when the incoming workloads exceed the capacity of one physical machine – all while running the exact same Wallaroo application that we run locally as part of our regular development. Wallaroo handles the scale-aware layer of our program, so we can focus on the business logic and flow of our data.

If you’re hitting limits when running your hourly, daily or nightly batch jobs and are looking into scaling out horizontally, don’t hesitate to reach out or drop in to our IRC channel. We’d love to chat!