Blog DevOps The Complete Guide to Kubernetes Logging

The Complete Guide to Kubernetes Logging

Application logs are your best friend when it comes to troubleshooting problems in your application. If you’re working with containerized workloads and using Kubernetes, it’s especially important for you to have a good sense of how logging works because it can help you set up the logging solutions in the right way & make better use of logging.

This article is a deep dive into Kubernetes logging architecture. This article would be especially informative for those unfamiliar with the complexities of working in Kubernetes.

Kubernetes Logging Architecture

The following points talk about Kubernetes Logging Architecture. Going through them is crucial in order to understand other concepts related to logging in Kubernetes.

  • A container writes messages about the current tasks that the process in the container is working at. In Kubernetes, these messages are taken up by the container runtime engine of the nodes and directed to a JSON file on the nodes.
  • In a cluster, some components run as a container (inside a pod) while others run as systemd process on the node. Both generate logs.
  • Components such as kube-apiserver, kube-scheduler, kube-proxy and etcd run as containers. Their logs can be viewed using the kubectl commands, which in turn returns the logs from the JSON file on the nodes.
  • Components such as kubelet, container runtime engines run as systemd processes on the nodes. Their logs can be viewed using the journalctl methods.
  • The following command can be used to view the logs.
    # In order to see logs saved in JSON file on the nodes - 
    kubectl log <pod-name> -c <container-nam
    
    # In order to see logs generated by the cluster's systemd processes -
    Jounalctl -PID=<process-id>

Demo

While learning Kubernetes, it is good to know the theory and better to see things practically.

Let’s use minikube to practically see the files & try the commands discussed above.

  • Install minikube by following the guide here – Guide to minikube installation
  • Start minikube

    minikube start
  • Start a pod

    kubectl run nginx –image=nginx
  • View logs of the pod

    kubectl logs nginx -c nginx
  • Sample output

    ...
    /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
    /docker-entrypoint.sh: Configuration complete; ready for start up
    2022/01/13 20:12:04 [notice] 1#1: using the "epoll" event method
    2022/01/13 20:12:04 [notice] 1#1: nginx/1.21.5
    2022/01/13 20:12:04 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
    2022/01/13 20:12:04 [notice] 1#1: OS: Linux 4.14.209-160.339.amzn2.x86_64
    2022/01/13 20:12:04 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
    2022/01/13 20:12:04 [notice] 1#1: start worker processes
    2022/01/13 20:12:04 [notice] 1#1: start worker process 30
    2022/01/13 20:12:04 [notice] 1#1: start worker process 31
    2022/01/13 20:12:04 [notice] 1#1: start worker process 32
    ...
  • The logs output by the command is being read from the JSON file on the node.
  • SSH into minikube node

    minikube ssh
  • Fing process id of kubelet

    ps aux | grep kubelet
  • Check logs of the kubelet process
    sudo journalctl _PID=1063
    
  • Sample output
    ...
    Jan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.804293 1063 server.go:440] "Kubelet version" kubeletVersion="v1.21.2"
    Jan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.804959 1063 server.go:851] "Client rotation is on, will bootstrap in background"
    Jan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.807635 1063 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/ku
    belet-client-current.pem".
    Jan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.813472 1063 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/c
    erts/ca.crt
    Jan 13 18:56:51 minikube kubelet[1063]: W0113 18:56:51.888596 1063 sysinfo.go:203] Nodes topology is not available, providing CPU topology
    ...
  • The logs output by the command is from the journalctl of the node.

Deep dive about the container logs

The container logs are saved in a JSON file on the nodes. The container writes these files in a standard format containing –

  • Log: The actual message from the container.
  • Stream: The stream type i.e. standard output (stdout) or standard error (stderr).
  • Time: The timestamp of the log.

Format:

{
   “log” : “<log message from a container>”,
   “stream” : “<stream type>”,
   “time” : ”<timestamp>”
}

Example : 

{
    "log" : "listening on port 8080”,
    "stream" : "stdout",
    "time" : "2021-08-31T16:35:59.5109491Z"
}

Let’s see how we can view these JSON files in the node.

  • SSH into minikube
    minikube ssh
  • Go to the /var/log/pods directory
    cd /var/log/pods
  • Each directory here follows a format of “<namespace>_<pod-name>_<pod-id>”Kubernetes Console View
  • Each file inside a directory represents the container inside the pod, it follows the format of “<container-name>.log”
  • Each .log file here contains multiple logs in the JSON format as discussed above.

Logging challenges

Relying on the kubectl commands or the journalctl command to view the logs is not an efficient solution due to a variety of reasons

  • It becomes difficult to cross analyze logs from different containers and processes with these commands.
  • It’s not possible to view logs from a particular point in time with these commands. These commands output only the latest logs.

Due to these reasons, it is common to deploy a third-party solution that makes the management of logs easier and the viewing of logs simple.

Logging solutions

There is a variety of solutions available for kubernetes logging.

  • An example of an open-source logging solution could be – elastic stack.
  • An example of enterprise-level logging solutions is – logz.io, Splunk, SumoLogic, etc.
  • An example of cloud service-provided solutions is – Cloudwatch by AWS, Stackdriver monitoring by GKE, Monitor by Azure, etc.

Out of these, the elastic stack is the most widely used solution.

Elastic Stack

The elastic stack has the following components-

  • Agent: It is responsible for exporting logs from the Kubernetes cluster to a backend. E.g. Filebeat, Fluentd.
  • Backend: It is responsible for storing the logs sent by the agent. E.g. Elasticsearch.
  • Visualization: It reads logs from the backend and displays them in a human-friendly manner. E.g. Kibana.

Logging approaches

According to the use case and the preferences, one of the following approaches can be used to handle the logs with a third-party tool or custom solution.

  • Using a node logging agentUsing a node logging agentA node logging agent is a process that reads logs from a node and pushes them into a backend. Since each node contains some logs, the node logging agent is deployed on each node. For example- filebeat is a node logging agent which can be configured to push logs into elasticsearch i.e. the backend.
  • Streaming sidecar containerStreaming sidecar containerA streaming sidecar container can be configured to read logs from the other containers running in the pod. The sidecar can then output the logs into stdout/stderr from where Kubernetes will write them into a file. This approach is helpful in case some containers don’t write to a stdout/stderr by default.
  • Sidecar container with a logging agentSidecar container with a logging agentJust like the ‘Streaming’ sidecar container, this approach also involves a sidecar reading the logs of the other containers. However, in this approach, the ‘streaming’ sidecar itself is a logging agent and can push the logs into the backend on its own.
  • Exposing logs directly from the applicationExposing logs directly from the applicationIn case you want to avoid the overhead of running a sidecar or a logging agent, you can configure your application’s code to send the logs to a backend on its own.

Logging cleanup

One important thing to make a note of is that Kubernetes Logging does not do anything for clearing up the past logs. It simply does not take responsibility for cleaning up the logs. Therefore, a script of the process has to be implemented to clear out the old logs so that space does not get filled up. For example, Docker runtime has a configuration file daemon.json to configure log rotation options.

Final words

That’s all. I hope this guide helped you understand the concepts about Kubernetes Logging Architecture. In our next articles we will continue to write more about the Kubernetes topics, please stay tuned for more learning & exploring!

Happy Learning!!

Explore more learning @ whizlabs.com/library

About Shishir Khandelwal

Shishir has the passion and zeal to master his field of cloud & containers. He is a strong advocate of finding smart solutions to complex problems by leveraging the power of cloud & container technology such as Kubernetes and a strong believer in learning by doing because of which he does a lot of POCs and personal projects. He is a certified expert in AWS & Kubernetes.
Spread the love

LEAVE A REPLY

Please enter your comment!
Please enter your name here