Using aws emrserverless with logs

Question

Pulumi · Accepted Answer

In order to use AWS EMR Serverless with logs, you’ll want to create an EMR Serverless application and configure it to output logs to CloudWatch. For this, we will be using two AWS resources provided by Pulumi:

1. `aws.emrserverless.Application`: This is the AWS EMR Serverless application resource which allows you to create and manage an EMR Serverless application.
2. `aws-native.logs.LogGroup`: This is the AWS CloudWatch Logs resource that allows you to create a log group that can store logs from various AWS services including EMR Serverless.

Here is a detailed Pulumi TypeScript program that sets up an EMR Serverless application with CloudWatch Logs integration:

```typescript
import * as aws from '@pulumi/aws';
import * as pulumi from '@pulumi/pulumi';

// Create a CloudWatch Log Group for your EMR Serverless Application logs.
const logGroup = new aws.cloudwatch.LogGroup("emrServerlessLogs", {});

// Now, define the EMR Serverless Application with monitoring configuration.
// This will ensure that logs for the application are sent to the log group we just created.
const app = new aws.emrserverless.Application("emrServerlessApp", {
    type: "SPARK",
    releaseLabel: "emr-6.5.0",
    // Replace or modify these initial/max capacities per your workload requirements
    initialCapacities: [{
        initialCapacityType: "WORKER",
        initialCapacityConfig: {
            workerCount: 2, // Specify the amount of workers you want to start with
            workerConfiguration: {
                cpu: "vCPU_4",
                memory: "memory_16GB",
            },
        },
    }],
    maximumCapacity: {
        cpu: "vCPU_64",
        memory: "memory_256GB",
    },
    // Configure your network - replace the subnet and security group IDs as needed
    networkConfiguration: {
        subnetIds: ["subnet-xyz"],
        securityGroupIds: ["sg-xyz"],
    },
    // Enable auto stop configuration to control costs.
    autoStopConfiguration: {
        enabled: true,
        idleTimeoutMinutes: 10, // Auto-stop the application after 10 minutes of idleness
    },
    // Enable auto start configuration for always-on workloads.
    autoStartConfiguration: {
        enabled: true,
    },
    // Configure the monitoring with the CloudWatch LogGroup created above
    monitoringConfiguration: {
        s3MonitoringConfiguration: {
            logUri: `s3://my-emr-logs/`, // Substitute with your preferred S3 bucket URI
            encryptionKeyArn: `alias/aws/s3`, // Optional: specify your KMS key for S3 log encryption
        },
        managedPersistenceMonitoringConfiguration: {
            enabled: true,
            encryptionKeyArn: `alias/aws/s3`, // Optional: specify your KMS key for managed persistence logs
        },
        cloudWatchMonitoringConfiguration: {
            logGroupName: logGroup.name,
        },
    },
});

// Export the CloudWatch Log Group name and the EMR Serverless application id for easy access
export const logGroupName = logGroup.name;
export const applicationId = app.id;
```

This program defines an EMR Serverless Application resource with an initial worker capacity. It configures both the auto-stop to manage costs and the auto-start configurations, if you need the application always-on. Make sure to replace `"subnet-xyz"` and `"sg-xyz"` with the actual IDs of your AWS VPC subnet and security group. The monitoring configuration sets up logging for the application using the newly created CloudWatch Log Group. Also, it configures the logs to be saved to an S3 bucket of your choice.

Remember to replace `s3://my-emr-logs/` with the URI of your desired S3 bucket for logs. If you want to add encryption for logs, specify an appropriate KMS key ARN instead of `alias/aws/s3`. Ensure you have the right permissions and that the resources reside within the same region.

After running this Pulumi program successfully, you will have an EMR Serverless application that logs to CloudWatch. You can access these logs through the AWS console or AWS CLI, and configure alarms or create dashboards as needed to monitor the health and performance of your EMR serverless application.