Pulumi Blog: Elasticsearch

Test and Optimize Elasticsearch with Pulumi

Itamar Syn-Hershko — Thu, 05 Mar 2020 00:00:00 +0000

Guest Article: Itamar Syn-Hershko, Founder and CTO of BigData Boutique shows how they use Pulumi to benchmark Elasticsearch configurations across cloud providers. Pulumi enables BigData Boutique to test deployments in parallel and gather metrics to produce performant and cost-effective solutions for its customers.

BigData Boutique can rapidly iterate through infrastructure configurations based on client requirements. Familiar programming languages significantly shortens the inner dev loop because developers and operators can use modern programming tools such as IDEs, test frameworks, and CI/CD. By increasing the velocity of infrastructure deployment across many clouds with repeatable and versioned code, Pulumi enables companies to build efficient revenue centers for their business.

How Pulumi Drives Our Elasticsearch Capacity Planning and Cost Optimization Service

Capacity Planning and Cost Optimization of Elasticsearch clusters requires a special level of expertise and automation. Here is how we use Pulumi to launch long-running benchmarks to correctly identify the right configuration for our customers’ Big Data clusters.

At BigData Boutique, we are continually challenged by our customers—whether it’s complex Big Data challenges we are asked to solve, reducing costs for sophisticated systems holding mountains of data, improving performance for high-demand systems, or challenges faced in building a data-driven system.

One of the main challenges our customers are facing with Big Data systems is deploying the right hardware at the right scale. They need to know how to correctly estimate the number of servers required to run their compute workloads and databases, so they don’t pay more than they should, while still keeping those services running without interruption and within the defined KPIs.

To support our customers and help them succeed in their Big Data projects, we provide an array of advanced services, and our unique cost-optimization service addresses this challenge directly. This article outlines how we use long-running benchmarks to correctly identify the right configuration for our customers’ BigData clusters by using the infrastructure as code tool Pulumi for automating the creation of complex infrastructures with conditionals and highly tunable configurations to orchestrate those benchmarks at scale.

Elasticsearch Cost Optimization

Let’s take Elasticsearch as an example. Elasticsearch is a real-time search and analytics engine. It is built to ingest huge amounts of data and to allow one to search through and visualize them in real-time. But to be able to sustain the required amount of data on ingestion and the desired number of queries per second, the right hardware has to be used.

To do this, we have to make some decisions: which disks to use (SSD disks? local, or network storage?), the number of servers (scaling up or out?), CPU and RAM ratios, index mappings, various cluster configurations, and more. These are not easy questions to answer, and that is where our Elasticsearch expertise comes into play. But answering those questions correctly with a high level of certainty also requires rigorous testing for validating assumptions.

Whether our customer is running on-prem or on a cloud, our Elasticsearch Capacity Planning Service exists for one purpose: to find the hardware solution with the optimum balance between cost and performance, and do it scientifically, so the answers are as accurate and as precise as possible.

Elasticsearch Sizing Process

Our process begins with a sample data request. We ask the customer for their data (or a large sample of it), existing index mappings, expected queries, and finally known KPIs and SLAs. Our goal in the initial discovery phase is to reduce the number of free variables as much as possible because the more such variables, the more permutations we will have to test. Ideally, we want very few dimensions of freedom — usually in terms of hardware and configurations—so the number of permutations to test is on the order of dozens and not thousands.

Once the initial framework has been decided, we run fully automated benchmarks to test some of the permutations our experience tells us are the most promising to deliver good results. The benchmarking process is smart enough to omit long tails and outliers, so the results are not going to lie as to the true nature of the configuration that was selected and tested.

Between benchmark iterations, our team uses its years of experience to create more test configurations that optimize various configurations we decide are important based on the results of the previous benchmarks. After several iterations of benchmarking on various configurations, we send the preliminary findings to confirm our results and proposed direction to the customer. The purpose is to communicate the results and validate the trade-offs we decide on during the sizing procedure. Finally, we provide a summary session with our team of Elasticsearch experts and customer technical and business contacts to present the results and recommendations, and to explain the reasoning behind the analysis.

Finding the best Elasticsearch configuration requires careful benchmarking and tech expertise. Yet trade-offs will come into play. For instance, it might seem that the only way to gain another 100ms of performance improvement for a certain specific query would be to spend another $500 per month on hardware. But there may be other options. Our Sizing Service takes the guesswork out of the process. We automate the entire benchmarking procedure in order to make it an exact science.

To fully automate the benchmarking process, we use Pulumi.

Using Pulumi for State-of-the-Art Infrastructure Automation

Pulumi is an infrastructure-as-code tool that allows us to define and run infrastructures using code, as opposed to doing this manually or using other tools that use configuration-based syntax. This lets us create complex infrastructures with conditionals and highly tunable configurations. The Pulumi program we created can be run numerous times with different parameters. Each result is a cluster that is built and configured differently, sometimes even on completely entirely different clouds.

At BigData Boutique, we use that concept to launch multiple infrastructures in parallel, benchmark them, and show a report that highlights which config is better along with the various trade-offs for each. After the benchmark process, the cluster is destroyed, and the “provisioner” instance reports all logs, metrics, and the benchmark report, which we then analyze and visualize.

Generating, visualizing, and analyzing dozens of reports in parallel gives us, the Elasticsearch Engineers, the ability to understand which knobs to turn and what configurations to change to make our customers succeed in their project without spending more than they are willing to.

Conclusion

In sum, leveraging Pulumi helps us to create highly configurable infrastructures to orchestrate benchmarks at scale. As one of many advanced services that we offer, the Elasticsearch Capacity Planning and Cost Optimization Service continues to meet our customers’ most important planning and optimizations needs in an as scientifically rigorous manner as possible. You can read more about it here.

The original article was published on their company blog.

Getting Started With PaC

Sophia Parafina — Tue, 11 Feb 2020 00:00:00 +0000

Modern applications have brought many benefits and improvements, including the ability to scale and rapid iterations to update software. However, this has come at the cost of complexity. Modern infrastructure is composed of many resources that require detailed configuration to work correctly and securely. Even managed solutions from cloud service providers need additional configuration to ensure that services are secure and free of defects. Cloud providers, such as AWS, do allow you to create policies to ensure that applications are secure, but they are specific to resources that are already deployed. A significant benefit of Policy as Code is the ability to verify and spot problems before deploying your infrastructure.

Leaky Buckets

What do many of the most innovative companies in the modern cloud era have in common? They’ve all had massive data breaches from leaky AWS S3 Buckets. One of the most common reasons for data breaches is misconfigured AWS S3 Buckets. Both security professionals and bad actors use tools such as slurp to find open S3 buckets by just entering a domain, keyword, or even AWS credentials with the slurp CLI. Ideally, we want to proactively prevent public access to our S3 buckets before they are created.

Pulumi’s CrossGuard helps customers adhere to security best practices by checking the final state of a deployment against a set of policies and revealing any non-compliant services. CrossGuard is an SDK for writing policies for services, and it includes best practices playbooks for cloud providers such as AWS, Azure, and Google Cloud Platform. The Getting Started with PaC covers the case of leaking S3 buckets, but let’s do a quick review with an example showing how we can detect a leaky bucket before it gets deployed.

Note: If you haven’t installed Pulumi, follow the Pulumi Getting Started Start by creating an S3 bucket with Pulumi.

$ pulumi new aws-typescript

Running pulumi new creates the project files as well as the code to create a single S3 bucket. We’ll use an S3 ACL to make the bucket public by setting it to the canned ACL public-read.

const bucket = new aws.s3.Bucket("my-bucket", {
 acl: "public-read",
});

Next, we’ll create a Policy Pack, by creating a directory and making it the current working directory.

$ mkdir policypack && cd policypack

Then create the Policy Pack using pulumi policy new.

$ pulumi policy new aws-typescript

By default, this creates a policy that checks to see if the ACL sets the bucket to be publicly accessible.

new PolicyPack("aws-typescript", {
 policies: [{
 name: "s3-no-public-read",
 description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 if (bucket.acl === "public-read" || bucket.acl === "public-read-write") {
 reportViolation(
 "You cannot set public-read or public-read-write on an S3 bucket. " +
 "Read more about ACLs here: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html");
 }
 }),
 }],
});

A policy consists of a name; a description that is displayed when the policy is violated; an enforcement level of either advisory or mandatory; and validateResource which specifies the type of resource, the resource, any arguments, and the message to report the violation.

We can test this locally using pulumi preview

$ pulumi preview --policy-pack <path-to-policy-pack-directory>

The preview will fail and return this output:

Diagnostics:
 pulumi:pulumi:Stack (test-dev):
 error: preview failed

 aws:s3:Bucket (my-bucket):
 mandatory: [s3-no-public-read] Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.
 You cannot set public-read or public-read-write on an S3 bucket. Read more about ACLs here: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

CrossGuard validated the desired state of the S3 bucket against the no-public-read policy stopped the preview. If we were deploying the S3 bucket, it would stop the deployment until the policy is met.

Exposed Databases

Like S3 buckets, exposed services are another source of data breaches. Misconfigured databases, failing to set passwords, or dev services open to the public are responsible for releases of Personal Identifiable Information (PII) that range well into the millions of customers. Many of these breaches were due to misconfigured Elasticsearch instances that were public and not secured with a password, such as the recent exposure of a Microsoft customer service database.

We’ll take the same approach that we used to secure S3 buckets. First, we’ll create an Elasticsearch deployment, write a policy, then apply it.

$ pulumi new aws-typescript

Replace index.ts with the following to deploy Elasticsearch:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.elasticsearch.Domain("example", {
 clusterConfig: {
 instanceType: "r4.large.elasticsearch",
 },
 elasticsearchVersion: "1.5",
 snapshotOptions: {
 automatedSnapshotStartHour: 23,
 },
 tags: {
 Domain: "TestDomain",
 },
});

In this example, we’ll use AWSGuard, which is a set of best practice policies for AWS. First, create a directory, change into it, and create a new policy.

$ mkdir awsguard && cd awsguard
$ pulumi policy new awsguard-typescript

The default policy enforcement level is advisory, but given the frequency of misconfigured Elasticsearch deployments, we’ll set to mandatory.

new AwsGuard({ all: "mandatory" });

When we run pulumi preview with the AWSGuard PolicyPack, we can see how it triggers violations based on best practices that can prevent data exposure.

$ pulumi preview --policy-pack ../awsguard/
Previewing update (dev):

 Type Name Plan Info
 + pulumi:pulumi:Stack elasticsearch-dev create 1 error
 + └─ aws:elasticsearch:Domain example create 2 errors

Diagnostics:
 pulumi:pulumi:Stack (elasticsearch-dev):
 error: preview failed

 aws:elasticsearch:Domain (example):
 mandatory: [elasticsearch-encrypted-at-rest] Checks if the Elasticsearch Service domains have encryption at rest enabled.
 Elasticsearch domain example-5510ef8 must be encrypted at rest.
 mandatory: [elasticsearch-in-vpc-only] Checks that the Elasticsearch domain is only available within a VPC, and not accessible via a public endpoint.
 Elasticsearch domain example-5510ef8 must run within a VPC.


Policy Packs run:
 Name Version
 pulumi-awsguard (../awsguard) (local)

Permalink: https://app.pulumi.com/spara/elasticsearch/dev/previews/bb6df58f-69aa-485a-99a4-d076bc067b84

AWSGuard flagged two recommended Elasticsearch best practices, encryption and deploying only within a VPC and not publicly. By enabling encryption at rest, unauthorized users cannot read data in Elasticseach clusters, storage, indices, and log files. By enabling Elasticsearch’s built-in encryption, your deployment follows recommended practices that comply with HIPAA and GDPR. AWS VPCs can keep the network traffic between your applications and Elasticsearch cluster within your infrastructure’s network and not over the public internet. Isolating Elasticsearch within your infrastructure can help mitigate the likelihood of data exposure.

CrossGuard available in Pulumi 2.0

The examples in this article demonstrate how to avoid common security pitfalls experienced by even large organizations. We’ve looked at the out of the box capability of CrossGuard by securing an S3 bucket and showed how best practices encoded in AWSGuard can warn of common configuration errors before deploying infrastructure. These examples provide a preview of how IaC can improve deployments by keeping your infrastructure secure.

As mentioned previously, CrossGuard is currently a technology preview. It will be GA with the release of Pulumi 2.0 with even more features, such as OPA integration, and more policy packs. We invite you to try it out and send us feedback either through Github issues or on the Pulumi Community Slack Channel.