Science and technology

Start monitoring your Kubernetes cluster with Prometheus and Grafana

In my introductory article about chaos engineering, one of many major issues I coated was the significance of getting the regular state of your working Kubernetes cluster. Before you can begin inflicting chaos, you’ll want to know what the cluster seems to be like in a gentle state.

This article will cowl the way to get those metrics using Prometheus and Grafana. This walkthrough additionally makes use of Pop!_OS 20.04, Helm three, Minikube 1.14.2, and Kubernetes 1.19.

Configure Minikube

Install Minikube in no matter approach is smart on your setting. If you’ve got sufficient sources, I like to recommend giving your digital machine a bit greater than the default reminiscence and CPU energy:

$ minikube config set reminiscence 8192
❗  These adjustments will take impact upon a minikube delete and then a minikube begin
$ minikube config set cpus 6
❗  These adjustments will take impact upon a minikube delete and then a minikube begin

Then begin and test your system’s standing:

$ minikube begin
😄  minikube v1.14.2 on Debian bullseye/sid
🎉  minikube 1.19.zero is on the market! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.19.zero
💡  To disable this discover, run: 'minikube config set WantUpdateNotification false'

✨  Using the docker driver based mostly on consumer configuration
👍  Starting management airplane node minikube in cluster minikube
🔥  Creating docker container (CPUs=6, Memory=8192MB) ...
🐳  Preparing Kubernetes v1.19.zero on Docker 19.03.eight ...
🔎  Verifying Kubernetes elements...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to make use of "minikube" by default
$ minikube standing
minikube
sort: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install Prometheus

Once the cluster is ready up, begin your installations. Install Prometheus first by following the directions beneath.

First, add the repository in Helm:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Then set up your Prometheus Helm chart. You ought to see:

$ helm set up prometheus prometheus-community/prometheus
NAME: prometheus
LAST DEPLOYED: Sun May  9 11:37:19 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server may be accessed through port 80 on the next DNS title from inside your cluster:
prometheus-server.default.svc.cluster.native

Get the Prometheus server URL by working these instructions in the identical shell:

  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath=".items[0].metadata.name")
  kubectl --namespace default port-forward $POD_NAME 9090

You can entry the Prometheus Alertmanager through port 80 on this DNS title from inside your cluster:

prometheus-alertmanager.default.svc.cluster.native

Get the Alertmanager URL by working these instructions in the identical shell:

  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath=".items[0].metadata.name")
  kubectl --namespace default port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been moved to a worldwide property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################

You can entry the Prometheus PushGateway through port 9091 on this DNS title from inside your cluster:

prometheus-pushgateway.default.svc.cluster.native

Get the PushGateway URL by working these instructions in the identical shell:

  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=pushgateway" -o jsonpath=".items[0].metadata.name")
  kubectl --namespace default port-forward $POD_NAME 9091

For extra info on working Prometheus, go to:
https://prometheus.io/

Check to verify your pods are working:

$ kubectl get pods -n default
NAME                                             READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-ccf8f68cd-hcrqr          2/2     Running   zero          3m22s
prometheus-kube-state-metrics-685b975bb7-mhv54   1/1     Running   zero          3m22s
prometheus-node-exporter-mfcwj                   1/1     Running   zero          3m22s
prometheus-pushgateway-74cb65b858-7ffhs          1/1     Running   zero          3m22s
prometheus-server-d9fb67455-2g2jw                2/2     Running   zero          3m22s

Next, expose your port on the Prometheus server pod in an effort to see the Prometheus net interface. To do that, you want the service title and port. You additionally have to provide you with a reputation to open the service utilizing the Minikube service command.

Get the service title for prometheus-server:

$ kubectl get svc -n default
NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                      ClusterIP   10.96.zero.1        <none>        443/TCP        13m
prometheus-alertmanager         ClusterIP   10.106.68.12     <none>        80/TCP         8m22s
prometheus-kube-state-metrics   ClusterIP   10.104.167.239   <none>        8080/TCP       8m22s
prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP       8m22s
prometheus-pushgateway          ClusterIP   10.99.90.233     <none>        9091/TCP       8m22s
prometheus-server               ClusterIP   10.103.195.104   <none>        9090/TCP       8m22s

Expose the service as sort Node-port. Provide a goal port of 9090 and a reputation you need to name the server. The node port is the server listening port. This is an extract of the Helm chart:

    ## Port for Prometheus Service to hear on
    ##
    port: 9090

The command is:

$ kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prom-server
service/prom-server uncovered

Next, you want Minikube to open the service and browser:

jess@Athena:~$ minikube service prom-server
|-----------|-------------|-------------|---------------------------|
| NAMESPACE |    NAME     | TARGET PORT |            URL            |
|-----------|-------------|-------------|---------------------------|
| default   | prom-server |          80 | http://192.168.49.2:32169 |
|-----------|-------------|-------------|---------------------------|
🎉  Opening service default/prom-server in default browser...

Your browser ought to open and present you the Prometheus service.

Congratulations! You now have Prometheus put in in your cluster.

Install Grafana

Next, set up Grafana and configure it to work with Prometheus. Follow the steps beneath to show a service to configure Grafana and gather information from Prometheus to assemble your regular state.

Start with getting your Helm chart:

$ helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories

Search on your chart:

$ helm search repo grafana
NAME                                            CHART VERSION   APP VERSION     DESCRIPTION                                      
bitnami/grafana       5.2.11      7.5.5         Grafana is an open supply, characteristic wealthy metrics...
bitnami/grafana-operator       zero.6.5      three.10.zero   Kubernetes Operator based mostly on the Operator SDK f...
grafana/grafana    6.9.zero                7.5.5           The main software for querying and visualizing t...
secure/grafana    5.5.7         7.1.1           DEPRECATED - The main software for querying and ...

Since secure/grafana is depreciated, set up bitnami/grafana. Then set up your chart:

helm set up grafana bitnami/grafana
NAME: grafana
LAST DEPLOYED: Sun May  9 12:09:53 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be affected person whereas the chart is being deployed **

  1. Get the applying URL by working:

    echo "Browse to http://127.0.0.1:8080"
    kubectl port-forward svc/grafana 8080:3000 &

  2. Get the admin credentials:

    echo "User: admin"
    echo "Password: $(kubectl get secret grafana-admin --namespace default -o jsonpath=".information.GF_SECURITY_ADMIN_PASSWORD" | base64 --decode)"

As you’ll be able to see within the Helm set up output, the goal port for Grafana is 3000, so you’ll use that port for exposing the service to see Grafana’s net frontend. Before exposing the service, verify your providers are working:

$ kubectl get pods -A
NAMESPACE     NAME                                             READY   STATUS    RESTARTS   AGE
default       grafana-6b84bbcd8f-xt6vd                         1/1     Running   zero          4m21s

Expose the service:

$ kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-server
service/grafana-server uncovered

Enable the service to open a browser with a Minikube service:

jess@Athena:~$ minikube service grafana-server
|-----------|----------------|-------------|---------------------------|
| NAMESPACE |      NAME      | TARGET PORT |            URL            |
|-----------|----------------|-------------|---------------------------|
| default   | grafana-server |        3000 | http://192.168.49.2:30549 |
|-----------|----------------|-------------|---------------------------|
🎉  Opening service default/grafana-server in default browser...

You will see the welcome display the place you’ll be able to log in.

Set up credentials to log into Grafana utilizing kubectl. The instructions appeared within the set up’s output; listed below are the instructions in use:

$ echo "User: admin"
User: admin
$ echo "Password: $(kubectl get secret grafana-admin --namespace default -o jsonpath=".information.GF_SECURITY_ADMIN_PASSWORD" | base64 --decode)"
Password: G6U5VeAejt

Log in together with your new credentials, and you will note the Grafana dashboard.

Congratulations! You now have a working Grafana set up in your Minikube cluster with the flexibility to log in. The subsequent step is to configure Grafana to work with Prometheus to assemble information and present your regular state.

Configure Grafana with Prometheus

Now that you would be able to log in to your Grafana occasion, you’ll want to arrange the information assortment and dashboard. Since that is a wholly web-based configuration, I’ll undergo the setup utilizing screenshots. Start by including your Prometheus information assortment. Click the gear icon on the left-hand aspect of the show to open the Configuration settings, then choose Data Source.

On the subsequent display, click on Add information supply.

Select Prometheus.

Because you configured your Prometheus occasion to be uncovered on port 80, use the service title prometheus-server and the server port 80.

Save and check your new information supply by scrolling to the underside of the display and clicking Save and Test. You ought to see a inexperienced banner that claims Data supply is working.

Return to the highest of the web page and click on Dashboards.

Import all three dashboard choices.

Click the magnifying glass icon on the left-hand aspect to verify all three dashboards have been imported.

Now that the whole lot is configured, click on Prometheus 2.zero Stats, and it is best to see one thing much like this.

Congratulations! You have a arrange primary information assortment from Prometheus about your cluster.

Import extra monitoring dashboards

You can import extra detailed dashboards from Grafana Labs’ community dashboards assortment. I picked two of my favorites, Dash-minikube and Kubernetes Cluster Monitoring, for this fast walkthrough.

To import a dashboard, you want its ID from the dashboards assortment. First, click on the plus (+) signal on the left-hand aspect to create a dashboard, then click on Import within the dropdown listing, and enter the ID. For Dash-minikube, it is ID 10219.

Click Load, and enter the information supply on the subsequent display. Since this makes use of Prometheus, enter your Prometheus information supply.

Click Import, and the brand new dashboard will seem.

Now you’ve got a brand new dashboard to maintain observe of your Minikube stats. If you comply with the identical steps utilizing Kubernetes Cluster Monitoring (ID 2115), you will note a extra verbose monitoring dashboard.

Now you’ll be able to preserve observe of your regular state with Grafana and Prometheus information collections and visuals.

Final ideas

With these open supply instruments, you’ll be able to gather your cluster’s regular state and preserve a great pulse on it. This is vital in chaos engineering as a result of it permits you to test the whole lot in a damaging, unstable state and use that information to check your speculation about what may occur to its state throughout an outage.

Most Popular

breakingExpress.com features the latest multimedia technologies, from live video streaming to audio packages to searchable archives of news features and background information. The site is updated continuously throughout the day.

Copyright © 2017 Breaking Express, Green Media Corporation

To Top