Whizlabs Blog

Knowledge Hub for Project Managers & Tech Geeks

AWS Certification : How to setup Cloudwatch Logs for AWS services?

     -     Dec 17th, 2016   -     AWS Certifications   -     1 Comment

We have launched AWS Certified Solutions Architect Associate certification exam which is the basic level of AWS certifications. As part of training AWS certification topics, we have started writing important topics that are useful for preparing for the aws certification exams. In this articles, we are writing about AWS CloudTrail logs, these topics are part of the security in Amazon Web Services (AWS). Here is the snapshot of the exam blueprint.

AWS CloutWatch Logs Exam Info

Exam Objective

This topic addresses the Data Security topic as highlighted in the AWS Blueprint for the exam guide.

cloudwatch

AWS Cloudwatch is a monitoring mechanism provided by Amazon. It has the ability to monitor services as mentioned below

  • Amazon EC2 instances
  • Amazon RDS
  • Amazon Dynamo DB
  • And a host of other custom metrics

Since Cloudwatch fulfills a key security compliance requirement from a monitoring perspective, this is an important security aspect provided by AWS.

What is AWS CloudWatch?

So as mentioned in the Exam objective, AWS Cloudwatch is a monitoring mechanism provided by Amazon. It helps in a lot of aspects, with one of the most common one shown below. The below diagram from AWS documentation shows how Auto scaling is combined with Cloudwatch.

So let’s look at each part in a little bit more detail

  • The Auto Scaling Group consists of EC2 instances which can increase or decrease in number depending on the load which is required.
  • Now one way to trigger the auto scaling group is via a Cloud watch alarm. So let’s say an alarm is created which says that if the EC2 instances in the group reach an upper threshold of 80% in CPU time, then we need to scale or increase the capacity of the EC2 instances to take the additional load. So the 80% CPU usage is a Cloudwatch metric. When this metric is reached an alarm is triggered by Cloudwatch to the Auto scaling group to increase the number of the EC2 instances in the Auto scaling group.

AWS Cloudwatch Diagram

What are AWS CloudWatch logs?

AWS Cloudwatch logs can be used to take logs from EC2 instances and process them accordingly. You can use Cloudwatch logs to achieve the following

  • Monitor and store the logs for your operating system to understand better how your application is performing
  • It can be used to track the number of errors in your application by processing the logs for error messages.
  • It can also send notification if the number of error messages goes beyond a threshold value.

The below diagram from AWS documentation shows how an example of Cloudwatch Logs. So let’s look at each part in a little bit more detail

AWS Cloudwatch Logs Agent

  • We have a web server inside a cluster. This web server is hosted inside a container.
  • We then use custom scripts which are present for Linux which creates custom logs. These custom logs are sent via an agent to the Cloudwatch logs service.
  • The cloud watch logs can process these sent logs from the web server and see any patterns for any errors or any other sort of analysis which is required.

A closer look at AWS CloudWatch

Now let’s have a closer look at AWS Cloudwatch.

Step 1: Log into your AWS Console. You will see Cloud Watch under Management Tools.

cloudwatch in aws console

Step 2: You can then click on Logs to get started with Cloudwatch logs.

aws-article3-4

Step 3: So in order to work with CloudWatch logs, we will need to create an Ec2 instance which will send logs to Cloudwatch. The first step is to create a policy which will be used to allow the EC2 instance to work with Cloudwatch.

Go to Identity and Access Management, click on Policies and click on Create Policy

Create cloudwatch policy in EC2 instance

Step 4: In the screen, choose the option of “Create Your Own Policy”

aws-article3-6

Step 5: Now give the Policy a name and add the following JSON to the policy document

{  
   "Version":"2012-10-17",
   "Statement":[  
      {  
         "Effect":"Allow",
         "Action":[  
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents",
            "logs:DescribeLogStreams"
         ],
         "Resource":[  
            "arn:aws:logs:*:*:*"
         ]
      }
   ]
}

Cloudwatch policy creation steps in aws

Step 6: Once the Policy has been created, it’s now time to create the Role. Go to the Roles section and select the create new role

Cloudwatch policy rules

Step 7: Now give the role a name

Create cloudwatch policy rule name

Step 8: Next select the role type as Amazon EC2

AWS cloudwatch role type

Step 9: Search for the Cloudwatch policy which was created earlier

Search AWS cloudwatch policy

Step 10: Now that the role has been created. Create your EC2 instance, when the instance is created, make sure that the IAM role chosen is the one which is created above

EC2 instance and auto scaling for cloudwatch role

Step 11: Now follow the steps in the this link for the type of EC2 instance you spin up.

An example set of steps is given below for an Ubuntu instance

  1. Log into the Ec2 instance
  2. Do an apt-get update to download the latest packages on the system
  3. Do an apt-get install –y python to install the python software
  4. Run the command curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
  5. Run the command to setup the awslogs – sudo python ./awslogs-agent-setup.py –region us-east-1

You will be asked for the following details during the configuration

  • The below is only required if the role is not configured for the EC2 instance
    • The AWS access key ID.
    • The AWS secret access key
  • Default region name – The default is us-east-1. You can set this to us-east-1, us-west-1, us-west-2, ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, eu-central-1, eu-west-1, or sa-east-1.
  • The default output format
  • Path of log file to upload – The location of the file that contains the log data you want to send.
  • Destination Log Group name – The name for your log group.
  • Destination Log Stream name – By default, this is the name of the host.
  • Timestamp format – Specify the format of the timestamp within the specified log file.
  • Initial position – How data is uploaded. Set this to start_of_file to upload everything in the data file. Set to end_of_file to upload only newly-appended data.

AWS cloudwatch log console view

Once configured all log from the server will then be configured into Cloudwatch logs

AWS Certification Points about CloudWatch Logs

The above explanation is good enough to understand the concepts about cloudwatch and how to configure a cloudwatch for EC2 instance. You have to remember the following important points to answer the questions in AWS certification exam.

  • You can have an indefinite retention period for your logs. This enables one to go through the logs at a future date.
  • The Cloudwatch agent on the EC2 instance can used for log rotation so that logs can be moved off the host and on to a logging service. The raw data in the log files can then be accessed accordingly.
  • Currently the Cloudwatch log agent is supported on Amazon Linux, Ubuntu, CentOS, Red Hat Enterprise Linux, and Windows.

Summary

We have learnt about the cloudwatch in AWS and how to configure for EC2 instances with logs. There are many other features that are useful for managing your business. If you have any questions about the cloudwatch logs, please write it in the comments section.

Practice Questions

It is important to practice more number of questions for preparing for the solutions architect certification exam. We have prepared 300+ high quality questions that covers all the exam objectives and provides explanation for all the option given for the question. This would help you to improve our confidence on the exam before you are taking the real exam.

Technical Support

If you are looking for any technical support like more explanation on each questions or clarifications, we have a dedicated support for Amazon Web Services (AWS) to answer all your queries. You can drop us a mail at info@whizlabs.com with your queries. We would respond to your questions within 12 hours of time.

Good luck for your exam preparation!!


There is 1 comment


  • 8 months ago

    Zubkiewicz   /   Reply

    Good work! Thanks


Your Comment

Your email address will not be published.