How to save EC2 app logs to S3?TypeScript
To achieve log saving from an EC2 instance to S3, AWS offers services which can be used. Here, we're going to use Amazon CloudWatch Logs agent on the EC2 instance to monitor and ship logs.
Once logs are in CloudWatch you can then set up an export task, to automatically push logs to your S3 bucket.
Below is a Pulumi program which does setup that you can use to achieve this goal. It creates an EC2 instance, installs Cloudwatch agent on it, and sets up a CloudWatch log group and an S3 bucket. At the end of the day, the logs will be exported to S3.
Please note: make sure to replace
your-key-namewith your actual AWS Key Pair name,
your-security-group-idwith your existing security group id, and
your-ami-Idwith an AMI id that you wish to use.
In the program above:
- An EC2 instance is created, and the AWS CloudWatch Logs agent is installed on it using User Data. The agent configuration is fetched from an S3 bucket.
- A CloudWatch Log Group is created. This group will receive the logs from the EC2 Instance.
- An S3 bucket is created. This S3 bucket will store logs exported from CloudWatch.
- A Log Destination is set, where logs are exported from CloudWatch to the designated S3 bucket.
- A cloudtrail.Trail resource is also created. This represents a CloudTrail trail - a configuration that enables delivery of events as log files to an Amazon S3 bucket.
At the end of the script, we're exporting the EC2 public IP and hostname, and the newly created S3 bucket name.
your_bucketwith the S3 bucket storing your CloudWatch Agent configuration (the one mentioned in
your_cluster_namewith your actual cluster name.
Please also make sure to create the relevant IAM roles, policies and permissions so that all services can interact with each other.