Using Amazon MSK as an event source for Amazon Kinesis Data Firehose

Question

Pulumi · Accepted Answer

Certainly! We can use Pulumi to create an Amazon Managed Streaming for Apache Kafka (MSK) cluster and set it up as an event source for Amazon Kinesis Data Firehose. Here's how we can achieve this:

1. **Amazon MSK (Managed Streaming for Apache Kafka)**: We'll create an MSK Serverless Cluster which is a fully managed service that makes it easy to build and run applications that use Apache Kafka for processing streaming data. MSK provides all the infrastructure and software for Kafka, so you can focus on building your applications.

2. **Amazon Kinesis Data Firehose**: It's a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk. We'll set up a Delivery Stream that will be fed by the data from our MSK cluster. Data Firehose automatically scales to match the throughput of your data and requires no ongoing administration.

3. **Event Source Mapping**: To connect these two services, we'll create an event source mapping in AWS Lambda. This will trigger a function to process records from your MSK cluster and load them into your Data Firehose Delivery Stream.

Let's write a Pulumi program to set this up. We're assuming that your Pulumi setup, including AWS configuration, is already in place.

```typescript
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

// Create an MSK Serverless cluster
const mskCluster = new aws.msk.ServerlessCluster("myMskCluster", {
    // Define the properties according to your specifications
    // Here is a basic configuration example:
    clusterName: "my-msk-cluster",
    vpcConfigs: [{
        subnetIds: ["subnet-xxxxxxxxxxxxxxxxx"], // replace with actual Subnet IDs
        securityGroupIds: ["sg-xxxxxxxxxxxxxxx"], // replace with actual Security Group IDs
    }],
    clientAuthentication: {
        sasl: {
            iam: {
                enabled: true,
            },
        },
    },
});

// Define Kinesis Firehose Delivery Stream with MSK as the source
const firehoseStream = new aws.kinesis.FirehoseDeliveryStream("myFirehoseStream", {
    deliveryStreamType: "KinesisStreamAsSource",
    kinesisSourceConfiguration: {
        kinesisStreamArn: mskCluster.arn, // Using the MSK Cluster ARN
        roleArn: "arn:aws:iam::123456789012:role/firehose_delivery_role", // replace with actual Role ARN
    },
    // You will also need to specify where Firehose will deliver the messages.
    // For example, we're sending the data to an S3 bucket here:
    s3Configuration: {
        bucketArn: "arn:aws:s3:::my-bucket", // replace with actual Bucket ARN
        roleArn: "arn:aws:iam::123456789012:role/firehose_delivery_role", // the same role used above
    },
});

// Export the ARNs of the created resources
export const mskClusterArn = mskCluster.arn;
export const firehoseStreamArn = firehoseStream.arn;
```

This program sets up the necessary infrastructure. Make sure to replace the placeholders like `subnet-xxxxxxxxxxxxxxxxx`, `sg-xxxxxxxxxxxxxxx`, `arn:aws:iam::123456789012:role/firehose_delivery_role`, and `arn:aws:s3:::my-bucket` with your actual Subnet IDs, Security Group IDs, IAM Role ARN, and S3 Bucket ARN.

To run this Pulumi program:

1. Save the above code to a file named `index.ts`.
2. Ensure you have Pulumi installed and AWS credentials configured.
3. Run `pulumi up` to preview and deploy the changes.

Upon successful deployment, the MSK cluster will be streaming data to Kinesis Firehose, which in turn will deliver the data to the configured destination, such as an S3 bucket.