5 Step to Push EC2 Logs to CloudWatch with the New CloudWatch Agent

Introduction

In the world of cloud computing, managing logs efficiently is crucial for monitoring and troubleshooting applications. AWS CloudWatch provides a centralized logging service that enables you to collect, monitor, and analyze logs from various AWS services or your custom application. To facilitate log collection from EC2 instances, AWS has introduced the new CloudWatch Agent. In this blog post, we will explore how to push logs from an EC2 instance to CloudWatch using the new CloudWatch Agent.

Step 1: Install the CloudWatch Agent:

1. Connect to your EC2 instance using SSH or any other remote access method.

2. Download the CloudWatch Agent package suitable for your EC2 instance's operating system from the AWS documentation or use the following command:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/<latest-version>/amazon-cloudwatch-agent.deb
# or if using AMI 2 latest directly run
sudo yum install amazon-cloudwatch-agent -y

Replace <latest-version> with the appropriate version number.

1. Install the package using the package manager for your operating system. For example, on Ubuntu, use the following command:

sudo dpkg -i amazon-cloudwatch-agent.deb

Step 2: Create a CloudWatch Agent Configuration file:

1. Create a new configuration file for the CloudWatch Agent with CloudWatch Agent Wizard or You can use any text editor to create a file named cloudwatch-agent.json.

2. Open the cloudwatch-agent.json file and paste the following basic configuration:

{
  "agent": {
    "run_as_user": "root",
    "region": "aws-region"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/syslog",
            "log_group_name": "MyLogGroup",
            "log_stream_name": "MyLogStream",
            "timezone": "LOCAL"
          }
        ]
      }
    }
  }
}

In this example, we are collecting logs from the /var/log/syslog file and sending them to a log group named "MyLogGroup" and a log stream named "MyLogStream." You can customize the file path, log group, and log stream names based on your requirements.

1. Save the cloudwatch-agent.json file at any location.

Step 3: Configure and start the CloudWatch Agent:

1. Run the following command to configure the CloudWatch Agent using the configuration file you created:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:<file_path>

Replace <file-path> with the location of the file (eg. /home/ec2-user/cloudwatch-agent.json)

1. Start the CloudWatch Agent using the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a start

Step 4: Verify the status of the cloud-watch agent service

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

The response should look like this

{
       "status": "running",
       "starttime": "2017-12-12T18:41:18",
       "version": "1.73.4"
}

Step 5: Verify log ingestion in CloudWatch:

1. Go to the AWS Management Console and navigate to the CloudWatch service.

2. Select "Logs" from the sidebar and choose your desired log group.

3. You should see log streams corresponding to the EC2 instance you configured. Click on a log stream to view the log data.

Conclusion: By following these steps, you can easily set up the new CloudWatch Agent on an EC2 instance and start pushing logs to CloudWatch for centralized monitoring and analysis. The CloudWatch Agent provides a flexible and scalable solution for collecting logs, enabling you to gain valuable insights into your applications and infrastructure.

Ref : AWS documentation reference link