What it’s worthwhile to learn about cluster logging in Kubernetes

Server and utility logging is a vital facility for builders, operators, and safety groups to grasp an utility’s state working of their manufacturing setting.

Logging permits operators to find out if the purposes and the required elements are working easily and detect if one thing uncommon is occurring to allow them to react to the state of affairs.

For builders, logging offers visibility to troubleshoot the code throughout and after improvement. In a manufacturing setting, the developer often depends on a logging facility with out debugging instruments. Coupled with logging from the methods, builders can work hand in hand with operators to successfully troubleshoot points.

The most essential beneficiary of logging services is the safety group, particularly in a cloud-native setting. Having the power to gather info from purposes and system logs allows the safety group to investigate the information from authentication, utility entry to malware actions the place they will reply to them if wanted.

Kubernetes is the main container platform the place an increasing number of purposes get deployed in manufacturing. I imagine that understanding the logging structure of Kubernetes is an important endeavor that each Dev, Ops, and Security group must take significantly.

In this text, I focus on how completely different container logging patterns in Kubernetes work.

System logging and utility logging

Before I dig deeper into the Kubernetes logging structure, I’d wish to discover the completely different logging approaches and the way each functionalities are important options of Kubernetes logging.

There are two varieties of system elements: Those that run in a container and people that don’t. For instance:

• The Kubernetes scheduler and kube-proxy run in a container.
• The kubelet and container runtime don’t run in containers.

Similar to container logs, system container logs get saved within the /var/log listing, and you must rotate them often.

Here I think about container logging. First, I take a look at cluster-level logging and why it’s important for cluster operators. Cluster logs present details about how the cluster performs. Information like why pods acquired evicted or the node dies. Cluster logging can even seize info like cluster and utility entry and the way the applying makes use of compute sources. Overall, a cluster logging facility offers the cluster operators info that’s helpful for cluster operation and safety.

The different solution to seize container logs is thru the applying’s native logging facility. Modern utility design most definitely has a logging mechanism that helps builders troubleshoot utility efficiency points by commonplace out (stdout) and error streams (stderr).

To have an efficient logging facility, Kubernetes implementation requires each app and system logging elements.

3 varieties of Kubernetes container logging

There are three outstanding strategies of cluster-level logging that you simply see in many of the Kubernetes implementations today.

1. Node-level logging agent
2. Sidecar container utility for logging
3. Exposing utility logs on to logging backend

Node degree logging agent

I’d wish to think about the node-level logging agent. You often implement these utilizing a DaemonSet as a deployment technique to deploy a pod (which acts as a logging agent) in all of the Kubernetes nodes. This logging agent then will get configured to learn the logs from all Kubernetes nodes. You often configure the agent to learn the nodes /var/logs listing capturing stdout/stderr streams and ship it to the logging backend storage.

The determine under reveals node-level logging working as an agent in all of the nodes.

To arrange node-level logging utilizing the fluentd method for example, it’s worthwhile to do the next:

1. First, it’s worthwhile to create a ServiceAccount known as fluentdd. This service account will get utilized by the Fluentd Pods to entry the Kubernetes API, and it’s worthwhile to create them within the logging Namespace with the label app: fluentd.

   #fluentd-SA.yaml
   apiVersion: v1
   variety: ServiceAccount
   metadata:
     title: fluentd
     namespace: logging
     labels:
       app: fluentd

   You can view the entire instance on this repo.

2. You then must create a ConfigMap fluentd-configmap. This offers a config file to the fluentd daemonset with all of the required properties.

   #fluentd-daemonset.yaml
   apiVersion: extensions/v1beta1
   variety: DaemonSet
   metadata:
     title: fluentd
     namespace: logging
     labels:
       app: fluentd
       kubernetes.io/cluster-service: "true"
   spec:
     selector:
       matchLabels:
         app: fluentd
         kubernetes.io/cluster-service: "true"
     template:
       metadata:
         labels:
           app: fluentd
           kubernetes.io/cluster-service: "true"
       spec:
         serviceAccount: fluentd
         containers:
         - title: fluentd
           picture: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-elasticsearch7-1.0
           env:
             - title:  FLUENT_ELASTICSEARCH_HOST
               worth: "elasticsearch.logging.svc.cluster.local"
             - title:  FLUENT_ELASTICSEARCH_PORT
               worth: "9200"
             - title: FLUENT_ELASTICSEARCH_SCHEME
               worth: "http"
             - title: FLUENT_ELASTICSEARCH_USER
               worth: "elastic"
             - title: FLUENT_ELASTICSEARCH_PASSWORD
               valueFrom:
                 secretKeyRef:
                   title: efk-pw-elastic
                   key: password
             - title: FLUENT_ELASTICSEARCH_SED_DISABLE
               worth: "true"
           sources:
             limits:
               reminiscence: 512Mi
             requests:
               cpu: 100m
               reminiscence: 200Mi
           volumeMounts:
           - title: varlog
             mountPath: /var/log
           - title: varlibdockercontainers
             mountPath: /var/lib/docker/containers
             learnOnly: true
           - title: fluentconfig
             mountPath: /fluentd/and so on/fluent.conf
             subPath: fluent.conf
         terminationGracePeriodSeconds: 30
         volumes:
         - title: varlog
           hostPath:
             path: /var/log
         - title: varlibdockercontainers
           hostPath:
             path: /var/lib/docker/containers
         - title: fluentconfig
           configMap:
             title: fluentdconf

   You can view the entire instance on this repo.

Now, I take a look at the code on the best way to deploy a fluentd daemonset because the log agent.

#fluentd-daemonset.yaml
apiVersion: extensions/v1beta1
variety: DaemonSet
metadata:
  title: fluentd
  namespace: logging
  labels:
    app: fluentd
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      app: fluentd
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        app: fluentd
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      containers:
      - title: fluentd
        picture: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-elasticsearch7-1.0
        env:
          - title:  FLUENT_ELASTICSEARCH_HOST
            worth: "elasticsearch.logging.svc.cluster.local"
          - title:  FLUENT_ELASTICSEARCH_PORT
            worth: "9200"
          - title: FLUENT_ELASTICSEARCH_SCHEME
            worth: "http"
          - title: FLUENT_ELASTICSEARCH_USER
            worth: "elastic"
          - title: FLUENT_ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                title: efk-pw-elastic
                key: password
          - title: FLUENT_ELASTICSEARCH_SED_DISABLE
            worth: "true"
        sources:
          limits:
            reminiscence: 512Mi
          requests:
            cpu: 100m
            reminiscence: 200Mi
        volumeMounts:
        - title: varlog
          mountPath: /var/log
        - title: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          learnOnly: true
        - title: fluentconfig
          mountPath: /fluentd/and so on/fluent.conf
          subPath: fluent.conf
      terminationGracePeriodSeconds: 30
      volumes:
      - title: varlog
        hostPath:
          path: /var/log
      - title: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - title: fluentconfig
        configMap:
          title: fluentdconf

To put this collectively: 

kubectl apply -f fluentd-SA.yaml 
              -f fluentd-configmap.yaml 
              -f fluentd-daemonset.yaml

Sidecar container utility for logging

The different method is by utilizing a devoted sidecar container with a logging agent. The commonest implementation of the sidecar container is by utilizing Fluentd as a log collector. In the enterprise deployment (the place you will not fear about a little bit compute useful resource overhead), a sidecar container utilizing fluentd (or similar) implementation provides flexibility over cluster-level logging. This is as a result of you’ll be able to tune and configure the collector agent based mostly on the kind of logs, frequency, and different potential tunings it’s worthwhile to seize.

The determine under reveals a sidecar container as a logging agent.

For instance, a pod runs a single container, and the container writes to 2 completely different log information utilizing two completely different codecs. Here’s a configuration file for the pod:

#log-sidecar.yaml
apiVersion: v1
variety: Pod
metadata:
  title: counter
spec:
  containers:
  - title: rely
    picture: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      whereas true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      accomplished
    volumeMounts:
    - title: varlog
      mountPath: /var/log
  - title: count-log
    picture: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
    volumeMounts:
    - title: varlog
      mountPath: /var/log
  volumes:
  - title: varlog
    emptyDir: {}

To put this collectively, you’ll be able to run this pod:

$ kubectl apply -f log-sidecar.yaml

To confirm if the sidecar container works as a logging agent, you are able to do:

$ kubectl logs counter count-log

The anticipated output ought to appear like this:

$ kubectl logs counter count-log-1
Thu 04 Nov 2021 09:23:21 NZDT
Thu 04 Nov 2021 09:23:22 NZDT
Thu 04 Nov 2021 09:23:23 NZDT
Thu 04 Nov 2021 09:23:24 NZDT

Exposing utility logs on to logging backend

The third method, which (for my part) is probably the most versatile logging answer for Kubernetes container and utility logs, is by pushing the logs on to the logging backend answer. Although this sample doesn’t depend on the native Kubernetes functionality, it provides flexibility that almost all enterprises want like:

1. Extend a greater variety of help for community protocols and output codecs.
2. Allows load balancing functionality and enhances efficiency.
3. Configurable to just accept advanced logging necessities by upstream aggregation

Because this third method depends on a non-Kubernetes function by pushing logs immediately from each utility, it’s exterior the Kubernetes scope.

Conclusion

The Kubernetes logging facility is an important part for an enterprise deployment of a Kubernetes cluster. I mentioned three potential patterns which might be accessible to be used. You must discover a appropriate sample on your wants.

As proven, the node-level logging utilizing daemonset is the best deployment sample to make use of, however it additionally has some limitations that may not suit your group’s wants. On the opposite hand, the sidecar sample provides flexibility and customization that lets you customise what kind of logs to seize, supplying you with compute useful resource overhead. Finally, exposing utility logs on to the backend log facility is one other attractive method that enables additional customization.

The selection is yours. You simply want to search out the method that matches your group’s necessities.