{"id":80888,"date":"2022-01-14T19:04:34","date_gmt":"2022-01-15T00:34:34","guid":{"rendered":"https:\/\/www.whizlabs.com\/blog\/?p=80888"},"modified":"2022-01-18T08:58:10","modified_gmt":"2022-01-18T14:28:10","slug":"kubernetes-logging","status":"publish","type":"post","link":"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/","title":{"rendered":"The Complete Guide to Kubernetes Logging"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Application logs are your best friend when it comes to troubleshooting problems in your application. <\/span>If you&#8217;re working with containerized workloads and using Kubernetes, it&#8217;s especially important for you to have a good sense of how logging works because it can help you set up the logging solutions in the right way &amp; make better use of logging. <span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><em><span style=\"font-weight: 400;\">This article is a deep dive into <strong>Kubernetes logging architecture<\/strong>. This article would be especially informative for those unfamiliar with the complexities of working in Kubernetes.<\/span><\/em><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #ea7e02;color:#ea7e02\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #ea7e02;color:#ea7e02\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Kubernetes_Logging_Architecture\" >Kubernetes Logging Architecture<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Demo\" >Demo<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Deep_dive_about_the_container_logs\" >Deep dive about the container logs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Logging_challenges\" >Logging challenges<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Logging_solutions\" >Logging solutions<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Elastic_Stack\" >Elastic Stack<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Logging_approaches\" >Logging approaches<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Logging_cleanup\" >Logging cleanup<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.whizlabs.com\/blog\/kubernetes-logging\/#Final_words\" >Final words<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2 style=\"text-align: center;\"><span class=\"ez-toc-section\" id=\"Kubernetes_Logging_Architecture\"><\/span><span style=\"color: #000080;\">Kubernetes Logging Architecture<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The following points talk about Kubernetes Logging Architecture. Going through them is crucial in order to understand other concepts related to logging in Kubernetes.<\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">A container writes messages about the current tasks that the process in the container is working at. In Kubernetes, these messages are taken up by the container runtime engine of the nodes and directed to a JSON file on the nodes.<\/span><\/li>\n<li><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">In a cluster, some components run as a container (inside a pod) while others run as systemd process on the node. Both generate logs.<\/span><\/span><\/span><\/span><\/li>\n<li>Components such as kube-apiserver, kube-scheduler, kube-proxy and etcd run as containers. Their logs can be viewed using the kubectl commands, which in turn returns the logs from the JSON file on the nodes.<\/li>\n<li>Components such as kubelet, container runtime engines run as systemd processes on the nodes. Their logs can be viewed using the journalctl methods.<\/li>\n<li>The following command can be used to view the logs.\n<pre># In order to see logs saved in JSON file on the nodes - \r\nkubectl log &lt;pod-name&gt; -c &lt;container-nam\r\n\r\n# In order to see logs generated by the cluster's systemd processes -\r\n<span style=\"font-weight: 400;\">Jounalctl -PID=&lt;process-id&gt;<\/span><\/pre>\n<\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Demo\"><\/span>Demo<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>While learning Kubernetes, it is good to know the theory and better to see things practically.<\/p>\n<p>Let&#8217;s use minikube to practically see the files &amp; try the commands discussed above.<\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Install minikube by following the guide here &#8211; <a href=\"https:\/\/minikube.sigs.k8s.io\/docs\/start\/\" target=\"_blank\" rel=\"noopener\">Guide to minikube installation<\/a><\/span><\/li>\n<li><span style=\"font-weight: 400;\">Start minikube<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">minikube start<\/span><\/pre>\n<\/li>\n<li><span style=\"font-weight: 400;\">Start a pod<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">kubectl run nginx \u2013image=nginx<\/span><\/pre>\n<\/li>\n<li><span style=\"font-weight: 400;\">View logs of the pod<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">kubectl logs nginx -c nginx<\/span><\/pre>\n<\/li>\n<li><span style=\"font-weight: 400;\">Sample output<\/span><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/span><\/p>\n<pre class=\"p1\"><span class=\"s1\">...\r\n\/docker-entrypoint.sh: Launching \/docker-entrypoint.d\/30-tune-worker-processes.sh\r\n\/docker-entrypoint.sh: Configuration complete; ready for start up\r\n2022\/01\/13 20:12:04 [notice] 1#1: using the \"epoll\" event method\r\n2022\/01\/13 20:12:04 [notice] 1#1: nginx\/1.21.5\r\n2022\/01\/13 20:12:04 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) \r\n2022\/01\/13 20:12:04 [notice] 1#1: OS: Linux 4.14.209-160.339.amzn2.x86_64\r\n2022\/01\/13 20:12:04 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576\r\n2022\/01\/13 20:12:04 [notice] 1#1: start worker processes\r\n2022\/01\/13 20:12:04 [notice] 1#1: start worker process 30\r\n2022\/01\/13 20:12:04 [notice] 1#1: start worker process 31\r\n2022\/01\/13 20:12:04 [notice] 1#1: start worker process 32\r\n...<\/span><\/pre>\n<\/li>\n<li>The logs output by the command is being read from the JSON file on the node.<\/li>\n<li><span style=\"font-weight: 400;\">SSH into minikube node<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">minikube ssh<\/span><\/pre>\n<\/li>\n<li><span style=\"font-weight: 400;\">Fing process id of kubelet<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">ps aux | grep kubelet<\/span><\/pre>\n<\/li>\n<li><span style=\"font-weight: 400;\">Check logs of the kubelet process<\/span>\n<pre><span style=\"font-weight: 400;\">sudo journalctl _PID=1063\r\n<\/span><\/pre>\n<\/li>\n<li>Sample output\n<pre>...\r\nJan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.804293 1063 server.go:440] \"Kubelet version\" kubeletVersion=\"v1.21.2\"\r\nJan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.804959 1063 server.go:851] \"Client rotation is on, will bootstrap in background\"\r\nJan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.807635 1063 certificate_store.go:130] Loading cert\/key pair from \"\/var\/lib\/kubelet\/pki\/ku\r\nbelet-client-current.pem\".\r\nJan 13 18:56:51 minikube kubelet[1063]: I0113 18:56:51.813472 1063 dynamic_cafile_content.go:167] Starting client-ca-bundle::\/var\/lib\/minikube\/c\r\nerts\/ca.crt\r\nJan 13 18:56:51 minikube kubelet[1063]: W0113 18:56:51.888596 1063 sysinfo.go:203] Nodes topology is not available, providing CPU topology\r\n...<\/pre>\n<\/li>\n<\/ul>\n<ul>\n<li>The logs output by the command is from the journalctl of the node.<\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Deep_dive_about_the_container_logs\"><\/span>Deep dive about the container logs<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The container logs are saved in a JSON file on the nodes. The container writes these files in a standard format containing &#8211;<\/p>\n<ul>\n<li><strong><em>Log<\/em><\/strong>: The actual message from the container.<\/li>\n<li><strong><em>Stream<\/em><\/strong>: The stream type i.e. standard output (stdout) or standard error (stderr).<\/li>\n<li><strong><em>Time<\/em><\/strong>: The timestamp of the log.<\/li>\n<\/ul>\n<p>Format:<\/p>\n<pre><span style=\"font-weight: 400;\">{<\/span>\r\n<span style=\"font-weight: 400;\">   \u201clog\u201d : \u201c&lt;log message from a container&gt;\u201d,<\/span>\r\n   <span style=\"font-weight: 400;\">\u201cstream\u201d : \u201c&lt;stream type&gt;\u201d,<\/span>\r\n   <span style=\"font-weight: 400;\">\u201ctime\u201d : \u201d&lt;timestamp&gt;\u201d<\/span>\r\n<span style=\"font-weight: 400;\">}<\/span><\/pre>\n<p><span style=\"font-weight: 400;\">Example :\u00a0<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">{<\/span>\r\n<span style=\"font-weight: 400;\">    \"log\" : \"listening on port 8080\u201d,<\/span>\r\n    <span style=\"font-weight: 400;\">\"stream\" : \"stdout\",<\/span>\r\n<span style=\"font-weight: 400;\">    \"time\" : \"2021-08-31T16:35:59.5109491Z\"<\/span>\r\n<span style=\"font-weight: 400;\">}<\/span>\r\n<\/pre>\n<p>Let&#8217;s see how we can view these JSON files in the node.<\/p>\n<ul>\n<li>SSH into minikube\n<pre>minikube ssh<\/pre>\n<\/li>\n<li>Go to the \/var\/log\/pods directory\n<pre>cd \/var\/log\/pods<\/pre>\n<\/li>\n<li>Each directory here follows a format of &#8220;&lt;namespace&gt;_&lt;pod-name&gt;_&lt;pod-id&gt;&#8221;<img decoding=\"async\" class=\"aligncenter wp-image-80910 size-large\" title=\"Kubernetes Console View\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-1024x89.png\" alt=\"Kubernetes Console View\" width=\"1024\" height=\"89\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-1024x89.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-300x26.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-768x67.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-1536x134.png 1536w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-2048x178.png 2048w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-640x56.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.24.50-AM-681x59.png 681w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/li>\n<li>Each file inside a directory represents the container inside the pod, it follows the format of &#8220;&lt;container-name&gt;.log&#8221;<br \/>\n<img decoding=\"async\" class=\"aligncenter wp-image-80911 size-large\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-1024x90.png\" alt=\"\" width=\"1024\" height=\"90\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-1024x90.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-300x26.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-768x68.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-1536x136.png 1536w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-2048x181.png 2048w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-640x57.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.19-AM-681x60.png 681w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/li>\n<li>Each .log file here contains multiple logs in the JSON format as discussed above.<img decoding=\"async\" class=\"aligncenter wp-image-80912 size-large\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-1024x135.png\" alt=\"\" width=\"1024\" height=\"135\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-1024x135.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-300x39.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-768x101.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-1536x202.png 1536w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-2048x270.png 2048w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-640x84.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-15-at-2.25.41-AM-681x90.png 681w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Logging_challenges\"><\/span><span style=\"font-weight: 400;\">Logging challenges<\/span><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\">Relying on the <a href=\"https:\/\/www.whizlabs.com\/blog\/kubectl-imperative-commands-kubernetes\/\" target=\"_blank\" rel=\"noopener\">kubectl commands<\/a> or the journalctl command to view the logs is not an efficient solution due to a variety of reasons<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">It becomes difficult to cross analyze logs from different containers and processes with these commands.<\/span><\/span><\/li>\n<li><span style=\"font-weight: 400;\">It\u2019s not possible to view logs from a particular point in time with these commands. These commands output only the latest logs.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Due to these reasons, it is common to deploy a third-party solution that makes the management of logs easier and the viewing of logs simple.<\/span><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Logging_solutions\"><\/span>Logging solutions<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\">There is a variety of solutions available for kubernetes logging.<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">An example of an <em><strong>open-source<\/strong><\/em> logging solution could be &#8211; elastic stack.<\/span><\/span><\/li>\n<li><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">An example of <em><strong>enterprise-level<\/strong><\/em> logging solutions is &#8211; logz.io, Splunk, SumoLogic, etc.<\/span><\/span><\/li>\n<li><span style=\"font-weight: 400;\">An example of <em><strong>cloud service-provided<\/strong><\/em> solutions is &#8211; Cloudwatch by AWS, Stackdriver monitoring by GKE, Monitor by Azure, etc.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Out of these, the elastic stack is the most widely used solution.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Elastic_Stack\"><\/span>Elastic Stack<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The elastic stack has the following components-<\/p>\n<ul>\n<li><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><em><strong>Agent<\/strong><\/em>: It is responsible for exporting logs from the Kubernetes cluster to a backend. E.g. Filebeat, Fluentd.<\/span><\/span><\/li>\n<li><span style=\"font-weight: 400;\"><em><strong>Backend<\/strong><\/em>: It is responsible for storing the logs sent by the agent. <\/span><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">E.g. Elasticsearch.<\/span><\/span><\/li>\n<li><em><strong>Visualization<\/strong><\/em>: It reads logs from the backend and displays them in a human-friendly manner. E.g. Kibana.<\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Logging_approaches\"><\/span><strong>Logging approaches<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\">According to the use case and the preferences, one of the following approaches can be used to handle the logs with a third-party tool or custom solution.<\/span><\/p>\n<ul>\n<li><em><em><strong>Using a node logging agent<\/strong><\/em><\/em><img decoding=\"async\" class=\"aligncenter wp-image-80899 size-large\" title=\"Using a node logging agent\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-1024x576.png\" alt=\"Using a node logging agent\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-1024x576.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-300x169.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-768x432.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-747x420.png 747w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-640x360.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1-681x383.png 681w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/1.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>A node logging agent is a process that reads logs from a node and pushes them into a backend. Since each node contains some logs, the node logging agent is deployed on each node. For example- filebeat is a node logging agent which can be configured to push logs into elasticsearch i.e. the backend.<\/li>\n<li style=\"font-weight: 400;\"><em><em><strong>Streaming sidecar container<\/strong><\/em><\/em><img decoding=\"async\" class=\"aligncenter wp-image-80900 size-large\" title=\"Streaming sidecar container\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-1024x576.png\" alt=\"Streaming sidecar container\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-1024x576.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-300x169.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-768x432.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-747x420.png 747w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-640x360.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2-681x383.png 681w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/2.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>A streaming sidecar container can be configured to read logs from the other containers running in the pod. The sidecar can then output the logs into stdout\/stderr from where Kubernetes will write them into a file. This approach is helpful in case some containers don\u2019t write to a stdout\/stderr by default.<\/li>\n<li><em><em><strong>Sidecar container with a logging agent<\/strong><\/em><\/em><img decoding=\"async\" class=\"aligncenter wp-image-80901 size-large\" title=\"Sidecar container with a logging agent\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-1024x576.png\" alt=\"Sidecar container with a logging agent\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-1024x576.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-300x169.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-768x432.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-747x420.png 747w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-640x360.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3-681x383.png 681w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/3.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>Just like the &#8216;Streaming&#8217; sidecar container, this approach also involves a sidecar reading the logs of the other containers. However, in this approach, the &#8216;streaming&#8217; sidecar itself is a logging agent and can push the logs into the backend on its own.<\/li>\n<li><em><strong>Exposing logs directly from the application<\/strong><\/em><img decoding=\"async\" class=\"aligncenter wp-image-80902 size-large\" title=\"Exposing logs directly from the application\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-1024x576.png\" alt=\"Exposing logs directly from the application\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-1024x576.png 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-300x169.png 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-768x432.png 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-747x420.png 747w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-640x360.png 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4-681x383.png 681w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/4.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>In case you want to avoid the overhead of running a sidecar or a logging agent, you can configure your application\u2019s code to send the logs to a backend on its own.<\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Logging_cleanup\"><\/span>Logging cleanup<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div id=\"faq-question-1636607953571\" class=\"rank-math-list-item\">\n<div class=\"rank-math-answer \">\n<p>One important thing to make a note of is that Kubernetes Logging does not do anything for clearing up the past logs. It simply does not take responsibility for cleaning up the logs. Therefore, a script of the process has to be implemented to clear out the old logs so that space does not get filled up. For example, Docker runtime has a configuration file daemon.json to configure log rotation options.<\/p>\n<\/div>\n<\/div>\n<h3 id=\"faq-question-1636608532265\" class=\"rank-math-list-item\"><span class=\"ez-toc-section\" id=\"Final_words\"><\/span>Final words<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>That&#8217;s all. I hope this guide helped you understand the concepts about Kubernetes Logging Architecture. In our next articles we will continue to write more about the Kubernetes topics, please stay tuned for more learning &amp; exploring!<\/p>\n<p>Happy Learning!!<\/p>\n<p>Explore more learning @ <a href=\"https:\/\/www.whizlabs.com\/library\/\">whizlabs.com\/library<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Application logs are your best friend when it comes to troubleshooting problems in your application. If you&#8217;re working with containerized workloads and using Kubernetes, it&#8217;s especially important for you to have a good sense of how logging works because it can help you set up the logging solutions in the right way &amp; make better use of logging. This article is a deep dive into Kubernetes logging architecture. This article would be especially informative for those unfamiliar with the complexities of working in Kubernetes. Kubernetes Logging Architecture The following points talk about Kubernetes Logging Architecture. Going through them is crucial [&hellip;]<\/p>\n","protected":false},"author":357,"featured_media":80970,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[1862],"tags":[],"class_list":["post-80888","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"uagb_featured_image_src":{"full":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",600,315,false],"thumbnail":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture-150x150.png",150,150,true],"medium":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture-300x158.png",300,158,true],"medium_large":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",600,315,false],"large":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",600,315,false],"1536x1536":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",600,315,false],"2048x2048":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",600,315,false],"profile_24":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",24,13,false],"profile_48":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",48,25,false],"profile_96":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",96,50,false],"profile_150":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",150,79,false],"profile_300":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",300,158,false],"tptn_thumbnail":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture-250x250.png",250,250,true],"web-stories-poster-portrait":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",600,315,false],"web-stories-publisher-logo":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",96,50,false],"web-stories-thumbnail":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/01\/kubernetes-logging-architecture.png",150,79,false]},"uagb_author_info":{"display_name":"Shishir Khandelwal","author_link":"https:\/\/www.whizlabs.com\/blog\/author\/shishir-khandelwal\/"},"uagb_comment_info":1,"uagb_excerpt":"Application logs are your best friend when it comes to troubleshooting problems in your application. If you&#8217;re working with containerized workloads and using Kubernetes, it&#8217;s especially important for you to have a good sense of how logging works because it can help you set up the logging solutions in the right way &amp; make better&hellip;","_links":{"self":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts\/80888","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/users\/357"}],"replies":[{"embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/comments?post=80888"}],"version-history":[{"count":21,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts\/80888\/revisions"}],"predecessor-version":[{"id":80925,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts\/80888\/revisions\/80925"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/media\/80970"}],"wp:attachment":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/media?parent=80888"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/categories?post=80888"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/tags?post=80888"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}