Introduction:
In previous blogs, we have explored the Azure Container Service and Kubernetes. In this blog, we will explore the monitoring with Kubernetes.
Prerequisite:
- Running Kubernetes Cluster
- Kubectl utility
For reliable applications, it is required to have in place monitoring of the Kubernetes Cluster. It helps to determine availability, scalability, and reliability of the application deployed over Kubernetes Cluster.
Heapster aggregator is used for monitoring and event the logs. Heapster stores the information in storage backend. Currently, it supports Google Cloud Monitoring and InfluxDB as the storage backends. Heapster runs as a pod in the Kubernetes Cluster. It communicates with each node of the Kubernetes Cluster. Kubelet agent is responsible to provide the monitoring information to Heapster. Kubelet itself collects the data from cAdvisor.
cAdvisor:
cAdvisor is an open source container usage and performance analysis agent. In Kubernetes cAdvisor is included into Kubelet binary.cAdvisor auto-discovers all containers. It collects the information of CPU, memory, network and file system usage statistics.
Kubelet:
Kubelet bridge the gap between Kubernetes Master and Kubernetes Node. It manages the Pods and Containers on each machine.
InfluxDB and Grafana are used for storing the data and visualizing it. Google cloud monitoring provides the hosted solution for monitoring Kubernetes Cluster. Heapster can be set up to send the metrics to Google Cloud monitoring.
Let’s check the Monitoring Kubernetes Cluster created with Minikube:
Kubernetes Cluster created locally by minikube supports add-ons.
| $ minikube addons list – addon-manager: enabled – dashboard: enabled – kube-dns: enabled – default-storageclass: enabled – heapster: disabled – ingress: disabled – registry: disabled – registry-creds: disabled |
Enable the addon:
| $ minikube addons enable heapster |
To open the web interface:
| $ minikube addons open heapster |
The result will be displayed on the grafana.
In Minikube add-ons are helping for monitoring but it’s also possible to add heapster as Kubernetes deployment. This will be the manual installation of heapster, grafana and influxdb.
Following is the heapster.yaml :
| apiVersion: v1 kind: ServiceAccount metadata: name: heapster namespace: kube-system — apiVersion: extensions/v1beta1 kind: Deployment metadata: name: heapster namespace: kube-system spec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: heapster spec: serviceAccountName: heapster containers: – name: heapster image: gcr.io/google_containers/heapster-amd64:v1.4.0 imagePullPolicy: IfNotPresent command: – /heapster – –source=kubernetes:https://kubernetes.default – –sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086 — apiVersion: v1 kind: Service metadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: ‘true’ kubernetes.io/name: Heapster name: heapster namespace: kube-system spec: ports: – port: 80 targetPort: 8082 selector: k8s-app: heapster |
You can get the latest version of the heapster at https://github.com/kubernetes/heapster/ .
Using Kubectl:
| $ kubectl create -f heapster.yaml |
For grafana, use grafana.yaml:
| apiVersion: extensions/v1beta1 kind: Deployment metadata: name: monitoring-grafana namespace: kube-system spec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: grafana spec: containers: – name: grafana image: gcr.io/google_containers/heapster-grafana-amd64:v4.4.3 ports: – containerPort: 3000 protocol: TCP volumeMounts: – mountPath: /etc/ssl/certs name: ca-certificates readOnly: true – mountPath: /var name: grafana-storage env: – name: INFLUXDB_HOST value: monitoring-influxdb – name: GF_SERVER_HTTP_PORT value: “3000” # The following env variables are required to make Grafana accessible via # the kubernetes api-server proxy. On production clusters, we recommend # removing these env variables, setup auth for grafana, and expose the grafana # service using a LoadBalancer or a public IP. – name: GF_AUTH_BASIC_ENABLED value: “false” – name: GF_AUTH_ANONYMOUS_ENABLED value: “true” – name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin – name: GF_SERVER_ROOT_URL # If you’re only using the API Server proxy, set this value instead: # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy value: / volumes: – name: ca-certificates hostPath: path: /etc/ssl/certs – name: grafana-storage emptyDir: {} — apiVersion: v1 kind: Service metadata: labels: # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: ‘true’ kubernetes.io/name: monitoring-grafana name: monitoring-grafana namespace: kube-system spec: # In a production setup, we recommend accessing Grafana through an external Loadbalancer # or through a public IP. # type: LoadBalancer # You could also use NodePort to expose the service at a randomly-generated port # type: NodePort ports: – port: 80 targetPort: 3000 selector: k8s-app: grafana |
You can get the latest version of grafana.yaml at https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/influxdb/grafana.yaml .
Using kubectl :
| $ kubectl create -f grafana.yaml |
If influxdb is the storage backend, then use following YAML for deploying influxdb in Kubernetes Cluster:
| apiVersion: extensions/v1beta1 kind: Deployment metadata: name: monitoring-influxdb namespace: kube-system spec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: influxdb spec: containers: – name: influxdb image: gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3 volumeMounts: – mountPath: /data name: influxdb-storage volumes: – name: influxdb-storage emptyDir: {} — apiVersion: v1 kind: Service metadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: ‘true’ kubernetes.io/name: monitoring-influxdb name: monitoring-influxdb namespace: kube-system spec: ports: – port: 8086 targetPort: 8086 selector: k8s-app: influxdb |
You can get the latest version of influxdb.yaml at https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/influxdb/influxdb.yaml
Using Kubectl:
| $ kubectl create -f influxdb.yaml |
To access grafana dashboard with the manual setup, describe the grafana service and check endpoint of the service.
To describe the service using Kubectl use following command:
| $ kubectl describe svc monitoring-grafana –namespace kube-system Name: monitoring-grafana Namespace: kube-system Labels: addonmanager.kubernetes.io/mode=Reconcile kubernetes.io/minikube-addons=heapster kubernetes.io/minikube-addons-endpoint=heapster kubernetes.io/name=monitoring-grafana Annotations: kubectl.kubernetes.io/last-applied-configuration={“apiVersion”:”v1″,”kind”:”Service”,”metadata”:{“annotations”:{},”labels”:{“addonmanager.kubernetes.io/mode”:”Reconcile”,”kubernetes.io/minikube-addons… Selector: addonmanager.kubernetes.io/mode=Reconcile,name=influxGrafana Type: NodePort IP: 10.0.0.62 Port: <unset> 80/TCP NodePort: <unset> 30943/TCP Endpoints: 172.17.0.9:3000 Session Affinity: None Events: <none> |
Prometheus and Data Dog are also good tools for monitoring the Kubernetes Cluster. If you need the quick solution with Prometheus operator then https://github.com/camilb/prometheus-kubernetes is one of the good repo for monitoring with Kubernetes.
Visit our other blogs here.
