How do I analyze AWS ECS container logs with Splunk?
In this guide, we will set up an AWS ECS cluster and configure it to send container logs to Splunk using Pulumi. We will define the necessary resources including the ECS cluster, task definition, and log group. We will also set up the Splunk HEC endpoint to receive the logs.
Key Points:
- Create an ECS cluster and task definition.
- Configure CloudWatch log group for ECS.
- Set up Splunk HEC endpoint for log ingestion.
- Define IAM roles and policies for ECS and CloudWatch.
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
// Create an ECS cluster
const cluster = new aws.ecs.Cluster("ecs-cluster");
// Create a CloudWatch Log Group
const logGroup = new aws.cloudwatch.LogGroup("ecs-log-group", {
retentionInDays: 7,
});
// Define the Splunk HEC endpoint and token
const splunkHECEndpoint = "https://splunk-hec-endpoint:8088";
const splunkHECToken = "your-splunk-hec-token";
// Create a Task Definition with Splunk logging configuration
const taskDefinition = new aws.ecs.TaskDefinition("task", {
family: "ecs-task-family",
containerDefinitions: JSON.stringify([{
name: "app",
image: "nginx",
memory: 512,
cpu: 256,
logConfiguration: {
logDriver: "splunk",
options: {
"splunk-token": splunkHECToken,
"splunk-url": splunkHECEndpoint,
"splunk-insecureskipverify": "true",
"splunk-format": "json",
"splunk-source": "ecs",
"splunk-index": "main",
},
},
}]),
requiresCompatibilities: ["FARGATE"],
networkMode: "awsvpc",
cpu: "256",
memory: "512",
executionRoleArn: new aws.iam.Role("ecsTaskExecutionRole", {
assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "ecs-tasks.amazonaws.com" }),
}).arn,
taskRoleArn: new aws.iam.Role("ecsTaskRole", {
assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "ecs-tasks.amazonaws.com" }),
}).arn,
});
// Create an ECS service to run the task
const service = new aws.ecs.Service("ecs-service", {
cluster: cluster.arn,
taskDefinition: taskDefinition.arn,
desiredCount: 1,
launchType: "FARGATE",
networkConfiguration: {
subnets: ["subnet-12345678"],
assignPublicIp: true,
},
});
Conclusion
In this guide, we created an AWS ECS cluster and configured it to send container logs to Splunk using Pulumi. We defined an ECS task with a Splunk log driver, set up CloudWatch log group, and created necessary IAM roles. This setup ensures that your ECS container logs are efficiently sent to Splunk for analysis.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.