Private Cross-Account Event Streaming with Kafka PrivateLink

Question

Pulumi · Accepted Answer

Setting up a private cross-account event streaming solution with Apache Kafka through PrivateLink involves creating a managed Kafka cluster and configuring it to allow private access from another AWS account. You would typically use AWS PrivateLink to securely connect services across different accounts or VPCs without the traffic going over the internet.

The high-level steps involved include:

1. Provision an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster.
2. Configure the MSK cluster to allow connections via PrivateLink.
3. Set up the necessary VPC endpoint services and endpoint connections on the consumer's account to consume events from the Kafka cluster.

Below, I will guide you through a Pulumi program written in Python which sets up the requisite AWS infrastructure to accomplish these steps:

- Use the `aws-native.msk.Cluster` to create an MSK cluster.
- Use the `aws.ec2.VpcEndpointService` and `aws.ec2.VpcEndpointConnection` to establish the PrivateLink connectivity.

Here's what each part of the code does:

- **`msk.Cluster`**: Provisions a new Amazon MSK cluster, specifying settings like the Kafka version, number of broker nodes, and broker node group info including instance type, EBS volume size, subnets, and security groups.

- **`ec2.VpcEndpointService`**: Creates an endpoint service that others can use to create a connection to your services in your VPC.

- **`ec2.VpcEndpointServicePermissions`**: Grants permissions to other AWS principals to create a connection to the endpoint service.

- **`ec2.VpcEndpoint`**: Creates an interface endpoint to the MSK cluster for another AWS account or VPC.

Below is the Pulumi program. This program assumes you have already set up your AWS provider and configured your Pulumi environment.

```python
import pulumi
import pulumi_aws as aws
import pulumi_aws_native as aws_native

# Create an Amazon MSK cluster.
msk_cluster = aws_native.msk.Cluster("mskCluster",
    cluster_name="my-kafka-cluster",
    kafka_version="2.6.1",
    number_of_broker_nodes=3,
    broker_node_group_info=aws_native.msk.ClusterBrokerNodeGroupInfoArgs(
        instance_type="kafka.m5.large",
        client_subnets=[
            # Subnet IDs for the MSK cluster.
        ],
        security_groups=[
            # Security Group IDs for the MSK cluster.
        ],
        storage_info=aws_native.msk.ClusterBrokerNodeGroupInfoStorageInfoArgs(
            ebs_storage_info=aws_native.msk.ClusterBrokerNodeGroupInfoStorageInfoEBSStorageInfoArgs(
                volume_size=1000
            )
        )
    ),
)

# Create a VPC endpoint service which can be used by other accounts to connect to the MSK cluster.
endpoint_service = aws.ec2.VpcEndpointService("endpointService",
    acceptance_required=True,
    network_load_balancer_arns=[
        # ARN of Network Load Balancer to be associated with your VPC Endpoint Service.
    ]
)

# Grant permission to another AWS account to create a VPC endpoint connection to the endpoint service.
endpoint_service_permission = aws.ec2.VpcEndpointServicePermissions("endpointServicePermission",
    service_id=endpoint_service.id,
    allowed_principals=[
        # Enter AWS account ID of the consumer account here.
    ]
)

pulumi.export('msk_cluster_arn', msk_cluster.arn)
pulumi.export('vpc_endpoint_service_id', endpoint_service.id)
```

To run this Pulumi program:

1. Save this script as a `.py` file.
2. Use the Pulumi CLI to create a new stack or select an existing one.
3. Run `pulumi up` to preview and then deploy your stack.

Remember to replace placeholders like subnet IDs, security group IDs, and the AWS account ID with actual values appropriate for your environment.

This code handles the infrastructure setup on the producer's side, allowing them to stream events. The consumer side would require a corresponding setup to create VPC endpoint connections that use the endpoint service configured above.

Please keep in mind that this example might be simplified, as networking and security often demand specific configurations that align with organizational policies and practices. This would likely involve additional resource configurations for things like logging, monitoring, and fine-grained security controls.