Monitoring ECS with Datadog using service definition YAML

Question

Pulumi · Accepted Answer

Monitoring an Amazon ECS (Elastic Container Service) cluster with Datadog via service definition YAML involves using Pulumi to create the necessary infrastructure and configuration. You would typically need Datadog integration set up with AWS to monitor the ECS cluster. This is usually done by creating a `ServiceDefinitionYaml` resource for Datadog and configuring the ECS service to send its metrics to Datadog.

Below is a Pulumi TypeScript program walking you through the setup. First, we will define the AWS ECS cluster and an associated service. Next, we will integrate Datadog by defining a service definition YAML to capture the monitoring aspects.

Here's how you can set this up with Pulumi in TypeScript:

1. **Define AWS ECS Cluster and Service**: We create an ECS cluster and service that launches tasks based on the specified task definition. In this case, it would be a Fargate service which is serverless.

2. **Datadog Integration**: Once the ECS service is up and running, we need to monitor it. We assume that the Datadog agent is set up and running as a sidecar in the ECS task to collect metrics. Datadog's `ServiceDefinitionYaml` resource is used to define the service monitoring configuration.

Let's go through the code to see how this is done:

```typescript
import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';
import * as awsx from '@pulumi/awsx';
import * as datadog from '@pulumi/datadog';

// Create an ECS cluster
const cluster = new aws.ecs.Cluster("my-cluster", {});

// Define the execution role that the ECS tasks will use
const executionRole = new aws.iam.Role("execution-role", {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "ecs-tasks.amazonaws.com" }),
});

// Attach the necessary policies to the execution role
new aws.iam.RolePolicyAttachment("execution-role-policy", {
    role: executionRole,
    policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

// Specify the required task definition configuration
const taskDefinition = new aws.ecs.FargateTaskDefinition("app-task", {
    cpu: "256",
    memory: "512",
    executionRoleArn: executionRole.arn,
    containerDefinitions: pulumi.output([{
        name: "my-app",
        image: "my-app-image",
        portMappings: [{ containerPort: 80 }],
    }, {
        // Configuration for the Datadog agent container
        name: "datadog-agent",
        image: "datadog/agent:latest",
        environment: [
            { name: "DD_API_KEY", value: "your-datadog-api-key" },
            { name: "ECS_FARGATE", value: "true" },
        ],
        // Add more Datadog-specific configurations as needed
    }]).apply(JSON.stringify),
});

// Create a Fargate service using AWSX, a higher level abstraction over AWS resources
const service = new awsx.ecs.FargateService("app-svc", {
    cluster: cluster.arn,
    taskDefinition: taskDefinition.arn,
    desiredCount: 1,
}, { dependsOn: [executionRole] });

// Datadog service definition in YAML format
const datadogServiceDefinitionYaml = `
#ad_identifiers:
#- my-app
instances:
  - url: "http://${service.endpoint}:80/metrics"
tags:
  - "env:my-environment"
logs:
 - type: "docker"
   service: "my-service"
   source: "my-app"
`;

// Create a Datadog service definition resource
const ddServiceDefinition = new datadog.ServiceDefinitionYaml("dd-service-def", {
    serviceDefinition: datadogServiceDefinitionYaml,
});

// Export the ECS service endpoint to access the application
export const serviceEndpoint = service.endpoint;
```

This program accomplishes the following:

- Sets up an ECS Fargate service which has the flexibility of serverless execution.
- Includes a Datadog agent as a sidecar container within your task definition for monitoring.
- Utilizes a `ServiceDefinitionYaml` integration with Datadog to define how the ECS service should be monitored.

You would need to replace placeholders like `"my-app-image"`, `"your-datadog-api-key"`, and `"env:my-environment"` with your application's Docker image, your Datadog API key, and the appropriate tags for your environment, respectively.

This program demonstrates integrating Amazon ECS with Datadog for monitoring using Pulumi. The comments will guide you through each step and resources used to monitor your ECS service with ease.