Server and utility logging is a vital facility for builders, operators, and safety groups to grasp an utility’s state working of their manufacturing setting.
Logging permits operators to find out if the purposes and the required elements are working easily and detect if one thing uncommon is occurring to allow them to react to the state of affairs.
For builders, logging offers visibility to troubleshoot the code throughout and after improvement. In a manufacturing setting, the developer often depends on a logging facility with out debugging instruments. Coupled with logging from the methods, builders can work hand in hand with operators to successfully troubleshoot points.
The most essential beneficiary of logging services is the safety group, particularly in a cloud-native setting. Having the power to gather info from purposes and system logs allows the safety group to investigate the information from authentication, utility entry to malware actions the place they will reply to them if wanted.
Kubernetes is the main container platform the place an increasing number of purposes get deployed in manufacturing. I imagine that understanding the logging structure of Kubernetes is an important endeavor that each Dev, Ops, and Security group must take significantly.
In this text, I focus on how completely different container logging patterns in Kubernetes work.
System logging and utility logging
Before I dig deeper into the Kubernetes logging structure, I’d wish to discover the completely different logging approaches and the way each functionalities are important options of Kubernetes logging.
There are two varieties of system elements: Those that run in a container and people that don’t. For instance:
- The Kubernetes scheduler and
kube-proxyrun in a container. - The
kubeletand container runtime don’t run in containers.
Similar to container logs, system container logs get saved within the /var/log listing, and you must rotate them often.
Here I think about container logging. First, I take a look at cluster-level logging and why it’s important for cluster operators. Cluster logs present details about how the cluster performs. Information like why pods acquired evicted or the node dies. Cluster logging can even seize info like cluster and utility entry and the way the applying makes use of compute sources. Overall, a cluster logging facility offers the cluster operators info that’s helpful for cluster operation and safety.
The different solution to seize container logs is thru the applying’s native logging facility. Modern utility design most definitely has a logging mechanism that helps builders troubleshoot utility efficiency points by commonplace out (stdout) and error streams (stderr).
To have an efficient logging facility, Kubernetes implementation requires each app and system logging elements.
3 varieties of Kubernetes container logging
There are three outstanding strategies of cluster-level logging that you simply see in many of the Kubernetes implementations today.
- Node-level logging agent
- Sidecar container utility for logging
- Exposing utility logs on to logging backend
Node degree logging agent
I’d wish to think about the node-level logging agent. You often implement these utilizing a DaemonSet as a deployment technique to deploy a pod (which acts as a logging agent) in all of the Kubernetes nodes. This logging agent then will get configured to learn the logs from all Kubernetes nodes. You often configure the agent to learn the nodes /var/logs listing capturing stdout/stderr streams and ship it to the logging backend storage.
The determine under reveals node-level logging working as an agent in all of the nodes.
To arrange node-level logging utilizing the fluentd method for example, it’s worthwhile to do the next:
- First, it’s worthwhile to create a ServiceAccount known as
fluentdd. This service account will get utilized by the Fluentd Pods to entry the Kubernetes API, and it’s worthwhile to create them within the logging Namespace with the labelapp: fluentd.#fluentd-SA.yaml
apiVersion: v1
variety: ServiceAccount
metadata:
title: fluentd
namespace: logging
labels:
app: fluentdYou can view the entire instance on this repo.
- You then must create a ConfigMap
fluentd-configmap. This offers a config file to thefluentd daemonsetwith all of the required properties.#fluentd-daemonset.yaml
apiVersion: extensions/v1beta1
variety: DaemonSet
metadata:
title: fluentd
namespace: logging
labels:
app: fluentd
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
app: fluentd
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
app: fluentd
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
containers:
- title: fluentd
picture: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-elasticsearch7-1.0
env:
- title: FLUENT_ELASTICSEARCH_HOST
worth: "elasticsearch.logging.svc.cluster.local"
- title: FLUENT_ELASTICSEARCH_PORT
worth: "9200"
- title: FLUENT_ELASTICSEARCH_SCHEME
worth: "http"
- title: FLUENT_ELASTICSEARCH_USER
worth: "elastic"
- title: FLUENT_ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
title: efk-pw-elastic
key: password
- title: FLUENT_ELASTICSEARCH_SED_DISABLE
worth: "true"
sources:
limits:
reminiscence: 512Mi
requests:
cpu: 100m
reminiscence: 200Mi
volumeMounts:
- title: varlog
mountPath: /var/log
- title: varlibdockercontainers
mountPath: /var/lib/docker/containers
learnOnly: true
- title: fluentconfig
mountPath: /fluentd/and so on/fluent.conf
subPath: fluent.conf
terminationGracePeriodSeconds: 30
volumes:
- title: varlog
hostPath:
path: /var/log
- title: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- title: fluentconfig
configMap:
title: fluentdconfYou can view the entire instance on this repo.
Now, I take a look at the code on the best way to deploy a fluentd daemonset because the log agent.
#fluentd-daemonset.yaml
apiVersion: extensions/v1beta1
variety: DaemonSet
metadata:
title: fluentd
namespace: logging
labels:
app: fluentd
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
app: fluentd
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
app: fluentd
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
containers:
- title: fluentd
picture: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-elasticsearch7-1.0
env:
- title: FLUENT_ELASTICSEARCH_HOST
worth: "elasticsearch.logging.svc.cluster.local"
- title: FLUENT_ELASTICSEARCH_PORT
worth: "9200"
- title: FLUENT_ELASTICSEARCH_SCHEME
worth: "http"
- title: FLUENT_ELASTICSEARCH_USER
worth: "elastic"
- title: FLUENT_ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
title: efk-pw-elastic
key: password
- title: FLUENT_ELASTICSEARCH_SED_DISABLE
worth: "true"
sources:
limits:
reminiscence: 512Mi
requests:
cpu: 100m
reminiscence: 200Mi
volumeMounts:
- title: varlog
mountPath: /var/log
- title: varlibdockercontainers
mountPath: /var/lib/docker/containers
learnOnly: true
- title: fluentconfig
mountPath: /fluentd/and so on/fluent.conf
subPath: fluent.conf
terminationGracePeriodSeconds: 30
volumes:
- title: varlog
hostPath:
path: /var/log
- title: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- title: fluentconfig
configMap:
title: fluentdconf
To put this collectively:
kubectl apply -f fluentd-SA.yaml
-f fluentd-configmap.yaml
-f fluentd-daemonset.yaml
Sidecar container utility for logging
The different method is by utilizing a devoted sidecar container with a logging agent. The commonest implementation of the sidecar container is by utilizing Fluentd as a log collector. In the enterprise deployment (the place you will not fear about a little bit compute useful resource overhead), a sidecar container utilizing fluentd (or similar) implementation provides flexibility over cluster-level logging. This is as a result of you’ll be able to tune and configure the collector agent based mostly on the kind of logs, frequency, and different potential tunings it’s worthwhile to seize.
The determine under reveals a sidecar container as a logging agent.
For instance, a pod runs a single container, and the container writes to 2 completely different log information utilizing two completely different codecs. Here’s a configuration file for the pod:
#log-sidecar.yaml
apiVersion: v1
variety: Pod
metadata:
title: counter
spec:
containers:
- title: rely
picture: busybox
args:
- /bin/sh
- -c
- >
i=0;
whereas true;
do
echo "$i: $(date)" >> /var/log/1.log;
echo "$(date) INFO $i" >> /var/log/2.log;
i=$((i+1));
sleep 1;
accomplished
volumeMounts:
- title: varlog
mountPath: /var/log
- title: count-log
picture: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
volumeMounts:
- title: varlog
mountPath: /var/log
volumes:
- title: varlog
emptyDir: {}
To put this collectively, you’ll be able to run this pod:
$ kubectl apply -f log-sidecar.yamlTo confirm if the sidecar container works as a logging agent, you are able to do:
$ kubectl logs counter count-logThe anticipated output ought to appear like this:
$ kubectl logs counter count-log-1Thu 04 Nov 2021 09:23:21 NZDT
Thu 04 Nov 2021 09:23:22 NZDT
Thu 04 Nov 2021 09:23:23 NZDT
Thu 04 Nov 2021 09:23:24 NZDT
Exposing utility logs on to logging backend
The third method, which (for my part) is probably the most versatile logging answer for Kubernetes container and utility logs, is by pushing the logs on to the logging backend answer. Although this sample doesn’t depend on the native Kubernetes functionality, it provides flexibility that almost all enterprises want like:
- Extend a greater variety of help for community protocols and output codecs.
- Allows load balancing functionality and enhances efficiency.
- Configurable to just accept advanced logging necessities by upstream aggregation
Because this third method depends on a non-Kubernetes function by pushing logs immediately from each utility, it’s exterior the Kubernetes scope.
Conclusion
The Kubernetes logging facility is an important part for an enterprise deployment of a Kubernetes cluster. I mentioned three potential patterns which might be accessible to be used. You must discover a appropriate sample on your wants.
As proven, the node-level logging utilizing daemonset is the best deployment sample to make use of, however it additionally has some limitations that may not suit your group’s wants. On the opposite hand, the sidecar sample provides flexibility and customization that lets you customise what kind of logs to seize, supplying you with compute useful resource overhead. Finally, exposing utility logs on to the backend log facility is one other attractive method that enables additional customization.
The selection is yours. You simply want to search out the method that matches your group’s necessities.