1. How to save EC2 app logs to S3?

    TypeScript

    To achieve log saving from an EC2 instance to S3, AWS offers services which can be used. Here, we're going to use Amazon CloudWatch Logs agent on the EC2 instance to monitor and ship logs.

    Once logs are in CloudWatch you can then set up an export task, to automatically push logs to your S3 bucket.

    Below is a Pulumi program which does setup that you can use to achieve this goal. It creates an EC2 instance, installs Cloudwatch agent on it, and sets up a CloudWatch log group and an S3 bucket. At the end of the day, the logs will be exported to S3.

    Please note: make sure to replace your-key-name with your actual AWS Key Pair name, your-security-group-id with your existing security group id, and your-ami-Id with an AMI id that you wish to use.

    import * as aws from "@pulumi/aws"; import * as awsx from "@pulumi/awsx"; //Create a new EC2 instance const ec2Instance = new aws.ec2.Instance("myInstance", { instanceType: "t2.medium", keyName: "your-key-name", securityGroups: ["your-security-group-id"], ami: "your-ami-id", userData: `#!/bin/bash echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O python3 ./awslogs-agent-setup.py -n -r ${aws.getRegion().name} -c s3://your_bucket/cloudwatch-logs-agent-config.json` }); // Create a CloudWatch Log Group for the EC2 instance to push logs to const logGroup = new aws.cloudwatch.LogGroup("myLogGroup"); // Create an S3 bucket to export logs const bucket = new aws.s3.Bucket("myBucket"); new aws.cloudwatch.LogDestination("testDestination", { name: "my_log_destination", roleArn: aws.iam.Role.ROLE_Arn, targetArn: bucket.arn }); const trail = new aws.cloudtrail.Trail("myTrail", { eventSelectors: [{ includeManagementEvents: true, readWriteType: "All", dataResources: [{ type: "AWS::S3::Object", values: [bucket.arn] }] }], isMultiRegionTrail: true, s3BucketName: bucket.id, }); new aws.cloudtrail.S3BucketNotification("cloudtrailBucketNotification", { bucket: bucket.bucket, events: ["s3:ObjectCreated:*"], queueArn: aws_sqs_queue.queue.arn, dependsOn: [trail], }); export const publicIp = ec2Instance.publicIp; export const publicHostName = ec2Instance.publicDns; export const bucketName = bucket.id;

    In the program above:

    • An EC2 instance is created, and the AWS CloudWatch Logs agent is installed on it using User Data. The agent configuration is fetched from an S3 bucket.
    • A CloudWatch Log Group is created. This group will receive the logs from the EC2 Instance.
    • An S3 bucket is created. This S3 bucket will store logs exported from CloudWatch.
    • A Log Destination is set, where logs are exported from CloudWatch to the designated S3 bucket.
    • A cloudtrail.Trail resource is also created. This represents a CloudTrail trail - a configuration that enables delivery of events as log files to an Amazon S3 bucket.

    At the end of the script, we're exporting the EC2 public IP and hostname, and the newly created S3 bucket name.

    Please replace your_bucket with the S3 bucket storing your CloudWatch Agent configuration (the one mentioned in cloudwatch-logs-agent-config.json), and your_cluster_name with your actual cluster name.

    Please also make sure to create the relevant IAM roles, policies and permissions so that all services can interact with each other.

    Learn more in AWS Docs for aws.ec2.Instance and CloudWatch Log Group.


GPT4o|Modeexplainbalancedcode