Pulumi Blog: Danny Zalkind

How Skai Migrated to Amazon Keyspaces with Pulumi

Danny Zalkind — Tue, 16 May 2023 00:00:00 +0000

Danny Zalkind is the Senior Director of Infrastructure Engineering for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

As Skai continues its journey towards fully migrating to the cloud using Pulumi, we’ve taken another large bite out of the migration pie, moving our most critical data to AWS on top of Amazon Keyspaces, an Apache Cassandra–compatible database service.

This blog post will dive into how we used Pulumi and AWS Lambda to perform the migration.

Offloading a piece of the pie to AWS

First, some background on what we do. Skai helps marketers work smarter and faster with a unified platform to reach and convert shoppers. An important service we provide to our clients is optimization of their digital marketing campaign efficiency. We store billions of events per day to enable clients to continuously improve the value they get from their ad spend.

Our on-premises setup included a 100-node Apache Cassandra cluster to be able to withstand the scale of billions of events, some stored for prolonged periods of time. Maintaining and scaling this setup was a high effort from our Data Infra teams. It was clear to us it was time to offload that piece of the pie to our cloud provider.

Why Amazon Keyspaces

When looking for a solution, we were looking for a service that would mark the following checkboxes:

Fully serverless to allow us focus on our business instead of server maintenance.
Scalability should be automatic and easy to perform.
High availability and multi-region built-in.
Cost effective pay-as-you-go and commitment-based model.
Extensive support and rich documentation.
Fully automatable using Pulumi, our main Infrastructure as Code framework.

Migration method

Let’s take a look at how we set up the migration. We decided to migrate the data using a dual-write method, writing new data to both our on-prem Cassandra cluster and Keyspaces. Any historical data needed could be fetched from Snowflake, our data lake solution, using our Apache Spark pipeline.

Finally, to make sure that the data is valid and we have no data discrepancies on the Keyspaces side, we added a Lambda function to periodically sample the data and compare it to the data in our on-prem cluster.

We made a conscious decision to decouple the application code from the provisioning of the infrastructure, letting Pulumi handle all resource creation and configuration, while the application team used our standard CI/CD pipeline to deliver their code to the service owning the Keyspaces data.

Let’s start provisioning

Keyspaces requires creating a logical entity called a keyspace, which groups related tables relevant for one or more applications. After that, we iterate over our config and create each table with the relevant columns and the preferred provisioning mode — either “on-demand” or “provisioned,” depending on our workload pattern. A provisioned workload in pre-allocated and thus significantly (x7) lower in hourly price, suitable mainly for use cases where the workload is quite predictable with no spikes.

Finally we populate AWS tags in order to be able track costs and performance metrics.

The following Python code illustrates the process:

import pulumi
import pulumi_aws as aws
from pulumi_komponents.account_provider.account_provider import get_account_provider

# Define resource tagging
def __populate_tags(aws_config: pulumi.Config, component_type: str, tags: dict):
 return {
 "Project": pulumi.get_project(),
 "Environment": aws_config.require("account"),
 "Type": component_type,
 **tags,
 }

def pulumi_program():
 aws_config = pulumi.Config("aws")
 keyspaces_config = pulumi.Config("keyspaces")
 tags = keyspaces_config.require_object("tags")
 provider = get_account_provider(aws_config)

 # Define keyspace
 keyspace = aws.keyspaces.Keyspace(
 resource_name=f'keyspaces-{keyspaces_config.require("name")}',
 name=keyspaces_config.require("name"),
 tags=__populate_tags(aws_config, "keyspace", tags),
 opts=pulumi.ResourceOptions(provider=provider, protect=True),
 )

 # Define and iterate over keyspace tables
 tables = keyspaces_config.require_object("tables")

 for table_name, table_ddl in tables.items():
 columns = []
 partition_keys = []
 clustering_keys = []
 static_columns = []
 for col in table_ddl.get("schema").get("columns"):
 name, type = col.popitem()
 columns.append(
 aws.keyspaces.TableSchemaDefinitionColumnArgs(name=name, type=type)
 )
 for par in table_ddl.get("schema").get("partition_keys"):
 partition_keys.append(
 aws.keyspaces.TableSchemaDefinitionPartitionKeyArgs(name=par)
 )
 for cl in table_ddl.get("schema").get("clustering_keys") or []:
 name, order_by = cl.popitem()
 clustering_keys.append(
 aws.keyspaces.TableSchemaDefinitionClusteringKeyArgs(
 name=name, order_by=order_by
 )
 )
 for s_col in table_ddl.get("schema").get("static_columns") or []:
 static_columns.append(
 aws.keyspaces.TableSchemaDefinitionStaticColumnArgs(name=s_col)
 )

 mode = table_ddl.get("throughput").get("mode")

 aws.keyspaces.Table(
 resource_name=f"keyspaces-table-{table_name}",
 keyspace_name=keyspace.name,
 table_name=table_name,
 schema_definition=aws.keyspaces.TableSchemaDefinitionArgs(
 columns=columns,
 partition_keys=partition_keys,
 clustering_keys=clustering_keys,
 static_columns=static_columns,
 ),

 # Set capacity mode and units for read and write per Table
 capacity_specification=aws.keyspaces.TableCapacitySpecificationArgs(
 throughput_mode=mode,
 read_capacity_units=table_ddl.get("throughput").get("rcu")
 if mode == "PROVISIONED"
 else None,
 write_capacity_units=table_ddl.get("throughput").get("wcu")
 if mode == "PROVISIONED"
 else None,
 ),

 # Set general table properties such as TTL and recovery method
 default_time_to_live=None,
 ttl=aws.keyspaces.TableTtlArgs(status="ENABLED"),
 point_in_time_recovery=aws.keyspaces.TablePointInTimeRecoveryArgs(
 status="ENABLED"
 ),

 # Populate AWS resource tagging
 tags=__populate_tags(
 aws_config,
 "table",
 {
 "Table-type": "unarchived"
 if table_name.endswith("_unarchived")
 else "standard",
 **tags,
 },
 ),
 opts=pulumi.ResourceOptions(
 parent=keyspace, provider=provider, protect=True
 ),
 )

Comparison Lambda

Next we define and create a Lambda function that samples and compares data between the data newly ingested into Keyspaces, our on-premise Cassandra cluster, and our data lake Snowflake.

We created an abstract class implementation of AWS Lambda to standardize our various Lambda implementations and keep the code readable. This class returns all needed arguments, configuration, layers, etc. Here is some Python code that shows our abstraction, named AwsFunction:

class AwsFunction(AbstractAwsFunction):
 def __init__(self, args: AwsFunctionArgs):
 super().__init__(resource_type="AwsLambda:Function", args=args)

 # Set Lambda zip artificate path
 def _create_function(self) -> aws.lambda_.Function:
 code_archive_path = f"/tmp/{self.args.name}.zip"
 with zipfile.ZipFile(code_archive_path, "w") as zip_obj:
 zip_sub_folders(
 root_source=self.args.functions_absolute_path,
 destination=code_archive_path,
 selected_sub_folders=[ZIP_COMMON_DIR, self.args.code_directory_name],
 zip_obj=zip_obj,
 )

 # Get iam role, kms keys and layers
 self.function_role = self._get_function_role()
 self.kms_key = self._create_kms()
 layers_arn, layers = self.__build_layers()

 # Return function outputs
 return aws.lambda_.Function(
 resource_name=generate_resource_name(self.args.name, "function"),
 name=self.args.function_name,
 role=self.function_role.arn,
 package_type=self.args.package_type,
 architectures=[self.args.architecture.lambda_value],
 handler=self.args.handler,
 memory_size=self.args.memory_size,
 reserved_concurrent_executions=self.args.concurrent_executions,
 runtime=self.args.runtime,
 code=pulumi.FileArchive(code_archive_path),
 timeout=self.args.timeout,
 layers=layers_arn,
 tags=self._populate_tags(component="lambda", resource_type="function"),
 kms_key_arn=self.kms_key.arn,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=self.args.environment_variables.to_dict()
 ),
 vpc_config=self.args.vpc_config,
 opts=ResourceOptions(
 provider=self.args.provider, depends_on=[self.kms_key] + layers
 ),
 )

Now we instantiate the abstract class as part of our “building block” component class, create the function, triggers and invocation:

class AwsLambda(ComponentResource):
 def __init__(self, args: AwsLambdaArgs):
 super().__init__(f"AwsLambda", args.lambda_name)

 # Define the function nased on the abstract class
 aws_function = (
 AwsFunction(args.aws_function_args)
 if args.aws_function_args.package_type == "Zip"
 else AwsFunctionContainer(args.aws_function_args)
 )

 # Add triggers
 triggers = {}
 for trigger, config in args.aws_function_args.lambda_triggers.items():
 triggers[trigger] = LAMBDA_TYPES_TO_DEF[trigger](
 config, aws_function.function, args.provider, args.tags
 )

 # Add async invocation config
 FunctionEventInvokeConfig(
 generate_resource_name(args.lambda_name, "event-invoke-config"),
 function_name=aws_function.function.name,
 destination_config=FunctionEventInvokeConfigDestinationConfigArgs(
 on_failure=FunctionEventInvokeConfigDestinationConfigOnFailureArgs(
 destination=args.aws_function_args.on_failure_arn,
 )
 if args.aws_function_args.on_failure_arn
 else None,
 on_success=FunctionEventInvokeConfigDestinationConfigOnSuccessArgs(
 destination=args.aws_function_args.on_success_arn,
 )
 if args.aws_function_args.on_success_arn
 else None,
 ),
 qualifier=args.aws_function_args.qualifier,
 maximum_event_age_in_seconds=args.aws_function_args.maximum_event_age_in_seconds,
 maximum_retry_attempts=args.aws_function_args.maximum_retry_attempts,
 opts=ResourceOptions(provider=args.provider),
 )

Now we’re ready to call our class to create our comparison Lambda:

from pulumi_komponents.awslambda import AwsLambda, AwsLambdaArgs
from pulumi import Config

config = Config()

AwsLambda(AwsLambdaArgs(config))

Note: This only a partial code example, meant to demonstrate the way we use object-oriented paradigms to create reusable building-blocks of code to be consumed by various platform and R&D teams to provision AWS resources.

Putting it all together

Once we have our Keyspaces setup in place and the confidence that our dual-write process is writing the data correctly, we are now able to wait enough time to accumulate the data on the AWS side, using a feature toggle in our application to switch our backend systems to work with AWS Keyspaces.

Outcome

Following the successful migration, we have gained 50% reduction in backend process runtime, increased event processing throughput by 400% and freed up 30% of our team’s time to do other things, instead of maintaining cluster infrastructure.

I hope this post provides readers with useful techniques and ideas for migrating large-scale databases to the public cloud.

Skai Migrates to AWS with Pulumi

Danny Zalkind — Tue, 04 May 2021 00:00:00 +0000

Danny Zalkind is the DevOps group manager for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai’s solution provides data-driven insights and optimization technology to help companies make informed decisions and scale performance across critical publishers.

Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers.

Why Skai is moving to AWS

The public cloud is not entirely new to us. Skai’s microservices have been running on AWS for a few years now. Our core monolith service is hosted in a data center running our private cloud infrastructure with its hundreds of Terabytes of data.

As our business grew, we noticed that our private cloud infrastructure could not keep pace with our business needs. Adding new capacity to our private cloud could take months of purchasing hardware, preparation, shipping, and deployment.

We decided to adopt infrastructure elastic enough to support the rate of innovation and growth that the business demanded.

We sought out the best infrastructure provider to shift as many services as possible to the cloud. AWS is a clear choice because of its unmatched managed services: RDS for our MySQL databases, Lambda functions for our scheduled processes and various automation tasks, DMS for data migration from the datacenter, Keyspaces for our Cassandra cluster, and EKS for our container workloads.

How Pulumi helped with migration

Once the team selected a cloud platform, it was time to plan and execute the migration. Skai’s DevOps group was selected to lead the project.

Why Pulumi

Skai’s private cloud infrastructure was provisioned and deployed using a combination of Puppet and Fabric. The code was long, full of customizations, not readable, and challenging to troubleshoot.

We were looking for a new solution that would:

Support cloud-native and multi-cloud architectures
Allow adding complex logic to the deployment process
Support reusable code blocks for various use cases
Allow each team to author different parts of the infrastructure
Use a modern programming language
And not create bottlenecks in the release process between the teams

We immediately fell in love with Pulumi, with its programmatic design, extensive modules, and extensibility via dynamic providers.

Project and Stack design

When the Skai team decided to standardize on Pulumi, we designed our architecture to support our use case and development cycle.

We decided to create a project for each AWS organization OUs and a stack for each instance of our application service.

For example, a project would be “platform-production” and contain all stacks and resources of individual environments such as “app1234,” where the number is an environment identifier.

This provided the optimal mix of releasing new code versions per workload type and placing them in the development lifecycle while making granular configuration changes per each instance more manageable.

Programming language

Pulumi supports several programming languages. For us, Python was an obvious choice due to its highly extensible nature, simplicity, and the team’s considerable amount of prior experience with Python. Since most existing automations were written in Python, this made adopting Pulumi very easy for the team.

Multi-team collaboration

Our organization is structured in multiple teams, with each team responsible for different areas of our infrastructure:

The Core team in charge of shared infrastructure such as provisioning AWS accounts
Multiple application teams are responsible for provisioning specific applications, and
An architect who leads designs, builds proof of concepts, and writes code.

It was essential to build our IaC projects with minimal coupling between the teams while keeping each team aligned with the same overarching project goals.

To enable this “loosely-coupled, tightly-aligned” approach, we created component classes for different infrastructure elements, which we later referenced in different places in our end-to-end code.

import pulumi
from pulumi_aws import ec2
from pulumi_aws import ebs

class KSAppServer(pulumi.ComponentResource):

 def __init__(self,
 name: str,
 args: KSAppServerArgs,
 opts: pulumi.ResourceOptions = None):
 super().__init__('kenshoo:KS:KSAppServer', name, None, opts)
 self.name = name
 self.app_instance = ec2.Instance("ks-aws-ue-" + args.ks_id + "-app",
 instance_type=args.instance_size,
 vpc_security_group_ids=args.security_groups_id,
 subnet_id=args.subnet,
 ami=args.ami,
 user_data=args.user_data_content,
 key_name=args.key_pair,
 tags={
 'Name': "ks-aws-ue-" + args.ks_id + "-app"
 },
 opts=pulumi.ResourceOptions(parent=self, delete_before_replace=True))

This practice enabled the application team to work on classes without being entirely dependent on the progress of the Core infrastructure team, which worked on classes of more low-level infrastructure such as VPCs and networking objects.

What about those internal metadata systems?

One more challenge we faced is registering each new instance of our application with internal metadata and IT systems. In our data center setup, we relied on a set of Jenkins jobs, but now, we wanted to integrate these processes with our code and make sure that we always have a consistent state of registration. We decided to use Pulumi’s dynamic providers to achieve this:

from requests import post, get

URL = 'https://krem.ourdomain.com/api/'

class KremMetadataRegistration:
 def create(self, ks_id):
 post(url=f'{URL}/onboarding/init_new_ks',
 params={'ks': f'ks{ks_id}'})

 def delete(self, ks_id):
 response_id = get(url=f'{URL}/config', params={'ks': ks_id}).json()[
 'id']
 if response_id:
 post(url=f'{URL}/onboarding/offboarding',
 params={'id': response_id})

Next, we reference the dynamic provider in another class that we call from the end-to-end code of the whole provisioning flow.

from pulumi.dynamic import *
from .metadata.krem_metadata_registration import KremMetadataRegistration

class RegistrationSchemaInputs(object):
 ks_id: Input[str]
 registered_service: Input[bool]

 def __init__(self, ks, registered_service):
 self.ks = ks
 self.ks_id = ks.lstrip('ks')
 self.registered_service = registered_service

class SchemaProvider(ResourceProvider):
 def __init__(self):
 self.metadatas = {
 'krem': KremMetadataRegistration()
 }

We can streamline the registration and deregistration processes for multiple systems as part of our provisioning lifecycle, creating a more streamlined flow with this approach.

Summary

Moving a high-scale and mature application to the public cloud presented quite a few challenges. We have selected Pulumi as our IaC framework of choice due to its native programmatic nature, an endless number of modules, and the ability to customize the solution using our dynamic providers and custom logic.

This post gave you a glimpse into how we designed and implemented provisioning of our application on AWS. Join us at the second part of the blog that follows our journey of automating data migration.