In my introductory article about chaos engineering, one of many major issues I coated was the significance of getting the regular state of your working Kubernetes cluster. Before you can begin inflicting chaos, you’ll want to know what the cluster seems to be like in a gentle state.
This article will cowl the way to get those metrics using Prometheus and Grafana. This walkthrough additionally makes use of Pop!_OS 20.04, Helm three, Minikube 1.14.2, and Kubernetes 1.19.
Configure Minikube
Install Minikube in no matter approach is smart on your setting. If you’ve got sufficient sources, I like to recommend giving your digital machine a bit greater than the default reminiscence and CPU energy:
$ minikube config set reminiscence 8192
❗ These adjustments will take impact upon a minikube delete and then a minikube begin
$ minikube config set cpus 6
❗ These adjustments will take impact upon a minikube delete and then a minikube begin
Then begin and test your system’s standing:
$ minikube begin
? minikube v1.14.2 on Debian bullseye/sid
? minikube 1.19.zero is on the market! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.19.zero
? To disable this discover, run: 'minikube config set WantUpdateNotification false'✨ Using the docker driver based mostly on consumer configuration
? Starting management airplane node minikube in cluster minikube
? Creating docker container (CPUs=6, Memory=8192MB) ...
? Preparing Kubernetes v1.19.zero on Docker 19.03.eight ...
? Verifying Kubernetes elements...
? Enabled addons: storage-provisioner, default-storageclass
? Done! kubectl is now configured to make use of "minikube" by default
$ minikube standing
minikube
sort: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
Install Prometheus
Once the cluster is ready up, begin your installations. Install Prometheus first by following the directions beneath.
First, add the repository in Helm:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
Then set up your Prometheus Helm chart. You ought to see:
$ helm set up prometheus prometheus-community/prometheus
NAME: prometheus
LAST DEPLOYED: Sun May 9 11:37:19 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server may be accessed through port 80 on the next DNS title from inside your cluster:
prometheus-server.default.svc.cluster.native
Get the Prometheus server URL by working these instructions in the identical shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath=".items[0].metadata.name")
kubectl --namespace default port-forward $POD_NAME 9090
You can entry the Prometheus Alertmanager through port 80 on this DNS title from inside your cluster:
prometheus-alertmanager.default.svc.cluster.nativeGet the Alertmanager URL by working these instructions in the identical shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath=".items[0].metadata.name")
kubectl --namespace default port-forward $POD_NAME 9093
#################################################################################
###### WARNING: Pod Security Policy has been moved to a worldwide property. #####
###### use .Values.podSecurityPolicy.enabled with pod-based #####
###### annotations #####
###### (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################
You can entry the Prometheus PushGateway through port 9091 on this DNS title from inside your cluster:
prometheus-pushgateway.default.svc.cluster.nativeGet the PushGateway URL by working these instructions in the identical shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=pushgateway" -o jsonpath=".items[0].metadata.name")
kubectl --namespace default port-forward $POD_NAME 9091For extra info on working Prometheus, go to:
https://prometheus.io/
Check to verify your pods are working:
$ kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-ccf8f68cd-hcrqr 2/2 Running zero 3m22s
prometheus-kube-state-metrics-685b975bb7-mhv54 1/1 Running zero 3m22s
prometheus-node-exporter-mfcwj 1/1 Running zero 3m22s
prometheus-pushgateway-74cb65b858-7ffhs 1/1 Running zero 3m22s
prometheus-server-d9fb67455-2g2jw 2/2 Running zero 3m22s
Next, expose your port on the Prometheus server pod in an effort to see the Prometheus net interface. To do that, you want the service title and port. You additionally have to provide you with a reputation to open the service utilizing the Minikube service command.
Get the service title for prometheus-server:
$ kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.zero.1 <none> 443/TCP 13m
prometheus-alertmanager ClusterIP 10.106.68.12 <none> 80/TCP 8m22s
prometheus-kube-state-metrics ClusterIP 10.104.167.239 <none> 8080/TCP 8m22s
prometheus-node-exporter ClusterIP None <none> 9100/TCP 8m22s
prometheus-pushgateway ClusterIP 10.99.90.233 <none> 9091/TCP 8m22s
prometheus-server ClusterIP 10.103.195.104 <none> 9090/TCP 8m22s
Expose the service as sort Node-port. Provide a goal port of 9090 and a reputation you need to name the server. The node port is the server listening port. This is an extract of the Helm chart:
## Port for Prometheus Service to hear on
##
port: 9090
The command is:
$ kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prom-server
service/prom-server uncovered
Next, you want Minikube to open the service and browser:
jess@Athena:~$ minikube service prom-server
|-----------|-------------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------|-------------|---------------------------|
| default | prom-server | 80 | http://192.168.49.2:32169 |
|-----------|-------------|-------------|---------------------------|
? Opening service default/prom-server in default browser...
Your browser ought to open and present you the Prometheus service.
Congratulations! You now have Prometheus put in in your cluster.
Install Grafana
Next, set up Grafana and configure it to work with Prometheus. Follow the steps beneath to show a service to configure Grafana and gather information from Prometheus to assemble your regular state.
Start with getting your Helm chart:
$ helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
Search on your chart:
$ helm search repo grafana
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/grafana 5.2.11 7.5.5 Grafana is an open supply, characteristic wealthy metrics...
bitnami/grafana-operator zero.6.5 three.10.zero Kubernetes Operator based mostly on the Operator SDK f...
grafana/grafana 6.9.zero 7.5.5 The main software for querying and visualizing t...
secure/grafana 5.5.7 7.1.1 DEPRECATED - The main software for querying and ...
Since secure/grafana is depreciated, set up bitnami/grafana. Then set up your chart:
helm set up grafana bitnami/grafana
NAME: grafana
LAST DEPLOYED: Sun May 9 12:09:53 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be affected person whereas the chart is being deployed **
- Get the applying URL by working:
echo "Browse to http://127.0.0.1:8080"
kubectl port-forward svc/grafana 8080:3000 & - Get the admin credentials:
echo "User: admin"
echo "Password: $(kubectl get secret grafana-admin --namespace default -o jsonpath=".information.GF_SECURITY_ADMIN_PASSWORD" | base64 --decode)"
As you’ll be able to see within the Helm set up output, the goal port for Grafana is 3000, so you’ll use that port for exposing the service to see Grafana’s net frontend. Before exposing the service, verify your providers are working:
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default grafana-6b84bbcd8f-xt6vd 1/1 Running zero 4m21s
Expose the service:
$ kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-server
service/grafana-server uncovered
Enable the service to open a browser with a Minikube service:
jess@Athena:~$ minikube service grafana-server
|-----------|----------------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|----------------|-------------|---------------------------|
| default | grafana-server | 3000 | http://192.168.49.2:30549 |
|-----------|----------------|-------------|---------------------------|
? Opening service default/grafana-server in default browser...
You will see the welcome display the place you’ll be able to log in.
Set up credentials to log into Grafana utilizing kubectl. The instructions appeared within the set up’s output; listed below are the instructions in use:
$ echo "User: admin"
User: admin
$ echo "Password: $(kubectl get secret grafana-admin --namespace default -o jsonpath=".information.GF_SECURITY_ADMIN_PASSWORD" | base64 --decode)"
Password: G6U5VeAejt
Log in together with your new credentials, and you will note the Grafana dashboard.
Congratulations! You now have a working Grafana set up in your Minikube cluster with the flexibility to log in. The subsequent step is to configure Grafana to work with Prometheus to assemble information and present your regular state.
Configure Grafana with Prometheus
Now that you would be able to log in to your Grafana occasion, you’ll want to arrange the information assortment and dashboard. Since that is a wholly web-based configuration, I’ll undergo the setup utilizing screenshots. Start by including your Prometheus information assortment. Click the gear icon on the left-hand aspect of the show to open the Configuration settings, then choose Data Source.
On the subsequent display, click on Add information supply.
Select Prometheus.
Because you configured your Prometheus occasion to be uncovered on port 80, use the service title prometheus-server and the server port 80.
Save and check your new information supply by scrolling to the underside of the display and clicking Save and Test. You ought to see a inexperienced banner that claims Data supply is working.
Return to the highest of the web page and click on Dashboards.
Import all three dashboard choices.
Click the magnifying glass icon on the left-hand aspect to verify all three dashboards have been imported.
Now that the whole lot is configured, click on Prometheus 2.zero Stats, and it is best to see one thing much like this.
Congratulations! You have a arrange primary information assortment from Prometheus about your cluster.
Import extra monitoring dashboards
You can import extra detailed dashboards from Grafana Labs’ community dashboards assortment. I picked two of my favorites, Dash-minikube and Kubernetes Cluster Monitoring, for this fast walkthrough.
To import a dashboard, you want its ID from the dashboards assortment. First, click on the plus (+) signal on the left-hand aspect to create a dashboard, then click on Import within the dropdown listing, and enter the ID. For Dash-minikube, it is ID 10219.
Click Load, and enter the information supply on the subsequent display. Since this makes use of Prometheus, enter your Prometheus information supply.
Click Import, and the brand new dashboard will seem.
Now you’ve got a brand new dashboard to maintain observe of your Minikube stats. If you comply with the identical steps utilizing Kubernetes Cluster Monitoring (ID 2115), you will note a extra verbose monitoring dashboard.
Now you’ll be able to preserve observe of your regular state with Grafana and Prometheus information collections and visuals.
Final ideas
With these open supply instruments, you’ll be able to gather your cluster’s regular state and preserve a great pulse on it. This is vital in chaos engineering as a result of it permits you to test the whole lot in a damaging, unstable state and use that information to check your speculation about what may occur to its state throughout an outage.