By Jakir Patel Azure Kubernetes Monitoring September 11, 2018

Introduction:

In previous blogs, we have explored the Azure Container Service and Kubernetes. In this blog, we will explore the monitoring with Kubernetes.

Prerequisite:

Running Kubernetes Cluster
Kubectl utility

For reliable applications, it is required to have in place monitoring of the Kubernetes Cluster. It helps to determine availability, scalability, and reliability of the application deployed over Kubernetes Cluster.

Heapster aggregator is used for monitoring and event the logs. Heapster stores the information in storage backend. Currently, it supports Google Cloud Monitoring and InfluxDB as the storage backends. Heapster runs as a pod in the Kubernetes Cluster. It communicates with each node of the Kubernetes Cluster. Kubelet agent is responsible to provide the monitoring information to Heapster. Kubelet itself collects the data from cAdvisor.

cAdvisor:

cAdvisor is an open source container usage and performance analysis agent. In Kubernetes cAdvisor is included into Kubelet binary.cAdvisor auto-discovers all containers. It collects the information of CPU, memory, network and file system usage statistics.

Kubelet:

Kubelet bridge the gap between Kubernetes Master and Kubernetes Node. It manages the Pods and Containers on each machine.

InfluxDB and Grafana are used for storing the data and visualizing it. Google cloud monitoring provides the hosted solution for monitoring Kubernetes Cluster. Heapster can be set up to send the metrics to Google Cloud monitoring.

Let’s check the Monitoring Kubernetes Cluster created with Minikube:

Kubernetes Cluster created locally by minikube supports add-ons.

$ minikube addons list
– addon-manager: enabled
– dashboard: enabled
– kube-dns: enabled
– default-storageclass: enabled
– heapster: disabled
– ingress: disabled
– registry: disabled
– registry-creds: disabled

Enable the addon:

$ minikube addons enable heapster

To open the web interface:

$ minikube addons open heapster

The result will be displayed on the grafana.

In Minikube add-ons are helping for monitoring but it’s also possible to add heapster as Kubernetes deployment. This will be the manual installation of heapster, grafana and influxdb.

Following is the heapster.yaml :

apiVersion: v1
kind: ServiceAccount
metadata:
name: heapster
namespace: kube-system
—
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster
namespace: kube-system
spec:
replicas: 1
template:
   metadata:
     labels:
       task: monitoring
       k8s-app: heapster
   spec:
     serviceAccountName: heapster
     containers:
     – name: heapster
       image: gcr.io/google_containers/heapster-amd64:v1.4.0
       imagePullPolicy: IfNotPresent
       command:
       – /heapster
       – –source=kubernetes:https://kubernetes.default
       – –sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
—
apiVersion: v1
kind: Service
metadata:
labels:
   task: monitoring
   # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
   # If you are NOT using this as an addon, you should comment out this line.
   kubernetes.io/cluster-service: ‘true’
   kubernetes.io/name: Heapster
name: heapster
namespace: kube-system
spec:
ports:
– port: 80
   targetPort: 8082
selector:
   k8s-app: heapster

You can get the latest version of the heapster at https://github.com/kubernetes/heapster/ .

Using Kubectl:

$ kubectl create -f heapster.yaml

For grafana, use grafana.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: monitoring-grafana
namespace: kube-system
spec:
replicas: 1
template:
   metadata:
     labels:
       task: monitoring
       k8s-app: grafana
   spec:
     containers:
     – name: grafana
       image: gcr.io/google_containers/heapster-grafana-amd64:v4.4.3
       ports:
       – containerPort: 3000
         protocol: TCP
       volumeMounts:
       – mountPath: /etc/ssl/certs
         name: ca-certificates
         readOnly: true
       – mountPath: /var
         name: grafana-storage
       env:
       – name: INFLUXDB_HOST
         value: monitoring-influxdb
       – name: GF_SERVER_HTTP_PORT
         value: “3000”
         # The following env variables are required to make Grafana accessible via
         # the kubernetes api-server proxy. On production clusters, we recommend
         # removing these env variables, setup auth for grafana, and expose the grafana
         # service using a LoadBalancer or a public IP.
       – name: GF_AUTH_BASIC_ENABLED
         value: “false”
       – name: GF_AUTH_ANONYMOUS_ENABLED
         value: “true”
       – name: GF_AUTH_ANONYMOUS_ORG_ROLE
         value: Admin
       – name: GF_SERVER_ROOT_URL
         # If you’re only using the API Server proxy, set this value instead:
         # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
         value: /
     volumes:
     – name: ca-certificates
       hostPath:
         path: /etc/ssl/certs
     – name: grafana-storage
       emptyDir: {}
—
apiVersion: v1
kind: Service
metadata:
labels:
   # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
   # If you are NOT using this as an addon, you should comment out this line.
   kubernetes.io/cluster-service: ‘true’
   kubernetes.io/name: monitoring-grafana
name: monitoring-grafana
namespace: kube-system
spec:
# In a production setup, we recommend accessing Grafana through an external Loadbalancer
# or through a public IP.
# type: LoadBalancer
# You could also use NodePort to expose the service at a randomly-generated port
# type: NodePort
ports:
– port: 80
   targetPort: 3000
selector:
   k8s-app: grafana

You can get the latest version of grafana.yaml at https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/influxdb/grafana.yaml .

Using kubectl :

$ kubectl create -f grafana.yaml

If influxdb is the storage backend, then use following YAML for deploying influxdb in Kubernetes Cluster:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: monitoring-influxdb
namespace: kube-system
spec:
replicas: 1
template:
   metadata:
     labels:
       task: monitoring
       k8s-app: influxdb
   spec:
     containers:
     – name: influxdb
       image: gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3
       volumeMounts:
       – mountPath: /data
         name: influxdb-storage
     volumes:
     – name: influxdb-storage
       emptyDir: {}
—
apiVersion: v1
kind: Service
metadata:
labels:
   task: monitoring
   # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
   # If you are NOT using this as an addon, you should comment out this line.
   kubernetes.io/cluster-service: ‘true’
   kubernetes.io/name: monitoring-influxdb
name: monitoring-influxdb
namespace: kube-system
spec:
ports:
– port: 8086
   targetPort: 8086
selector:
   k8s-app: influxdb

You can get the latest version of influxdb.yaml at https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/influxdb/influxdb.yaml

Using Kubectl:

$ kubectl create -f influxdb.yaml

To access grafana dashboard with the manual setup, describe the grafana service and check endpoint of the service.

To describe the service using Kubectl use following command:

$ kubectl describe svc monitoring-grafana –namespace kube-system
Name: monitoring-grafana
Namespace: kube-system
Labels: addonmanager.kubernetes.io/mode=Reconcile
kubernetes.io/minikube-addons=heapster
kubernetes.io/minikube-addons-endpoint=heapster
kubernetes.io/name=monitoring-grafana
Annotations: kubectl.kubernetes.io/last-applied-configuration={“apiVersion”:”v1″,”kind”:”Service”,”metadata”:{“annotations”:{},”labels”:{“addonmanager.kubernetes.io/mode”:”Reconcile”,”kubernetes.io/minikube-addons…
Selector: addonmanager.kubernetes.io/mode=Reconcile,name=influxGrafana
Type: NodePort
IP: 10.0.0.62
Port: <unset> 80/TCP
NodePort: <unset> 30943/TCP
Endpoints: 172.17.0.9:3000
Session Affinity: None
Events: <none>

Prometheus and Data Dog are also good tools for monitoring the Kubernetes Cluster. If you need the quick solution with Prometheus operator then https://github.com/camilb/prometheus-kubernetes is one of the good repo for monitoring with Kubernetes.

Visit our other blogs here.