1. Answers
  2. Analyzing AWS ECS container logs with Splunk

How do I analyze AWS ECS container logs with Splunk?

In this guide, we will set up an AWS ECS cluster and configure it to send container logs to Splunk using Pulumi. We will define the necessary resources including the ECS cluster, task definition, and log group. We will also set up the Splunk HEC endpoint to receive the logs.

Key Points:

  • Create an ECS cluster and task definition.
  • Configure CloudWatch log group for ECS.
  • Set up Splunk HEC endpoint for log ingestion.
  • Define IAM roles and policies for ECS and CloudWatch.
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create an ECS cluster
const cluster = new aws.ecs.Cluster("ecs-cluster");

// Create a CloudWatch Log Group
const logGroup = new aws.cloudwatch.LogGroup("ecs-log-group", {
    retentionInDays: 7,
});

// Define the Splunk HEC endpoint and token
const splunkHECEndpoint = "https://splunk-hec-endpoint:8088";
const splunkHECToken = "your-splunk-hec-token";

// Create a Task Definition with Splunk logging configuration
const taskDefinition = new aws.ecs.TaskDefinition("task", {
    family: "ecs-task-family",
    containerDefinitions: JSON.stringify([{
        name: "app",
        image: "nginx",
        memory: 512,
        cpu: 256,
        logConfiguration: {
            logDriver: "splunk",
            options: {
                "splunk-token": splunkHECToken,
                "splunk-url": splunkHECEndpoint,
                "splunk-insecureskipverify": "true",
                "splunk-format": "json",
                "splunk-source": "ecs",
                "splunk-index": "main",
            },
        },
    }]),
    requiresCompatibilities: ["FARGATE"],
    networkMode: "awsvpc",
    cpu: "256",
    memory: "512",
    executionRoleArn: new aws.iam.Role("ecsTaskExecutionRole", {
        assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "ecs-tasks.amazonaws.com" }),
    }).arn,
    taskRoleArn: new aws.iam.Role("ecsTaskRole", {
        assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "ecs-tasks.amazonaws.com" }),
    }).arn,
});

// Create an ECS service to run the task
const service = new aws.ecs.Service("ecs-service", {
    cluster: cluster.arn,
    taskDefinition: taskDefinition.arn,
    desiredCount: 1,
    launchType: "FARGATE",
    networkConfiguration: {
        subnets: ["subnet-12345678"],
        assignPublicIp: true,
    },
});

Conclusion

In this guide, we created an AWS ECS cluster and configured it to send container logs to Splunk using Pulumi. We defined an ECS task with a Splunk log driver, set up CloudWatch log group, and created necessary IAM roles. This setup ensures that your ECS container logs are efficiently sent to Splunk for analysis.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up