Install ETCD on Kubernetes with High Availability

In this multi-part series, we will cover how to install, maintain and monitor an Etcd cluster on Kubernetes with High Availability. We will be installing Etcd v3.5.4 on Amazon Elastic Kubernetes Service (EKS), but similar setup should work for other Kubernetes clusters as well.

Prerequisites

  1. Install Etcd on Kubernetes using Helm

We will be using Etcd helm package to install Etcd on Kubernetes. We can directly execute helm commands or for simplicity, we can create a Makefile which acts as a wrapper. First, create a Makefile as shown below.

NAMESPACE=coderiseio-etcd
#install
.PHONY: install-etcd
install-etcd: create-etcd-ns create-etcd-secret setup-etcd upgrade-etcd
 
.PHONY: create-etcd-ns
create-etcd-namespace:
    kubectl create namespace $(NAMESPACE)
 
.PHONY: create-etcd-secret
install-etcd-secret:
    @kubectl create secret generic etcd-root-password --from-literal=etcd-root-password=$(ETCD_ROOT_PASSWORD) -n $(NAMESPACE)

.PHONY: setup-etcd
setup-etcd:
    helm install etcd charts/etcd/ -n $(NAMESPACE)
 
.PHONY: upgrade-etcd
upgrade-etcd:
    helm upgrade etcd charts/etcd/ -n $(NAMESPACE)
 
.PHONY: init-etcd
init-etcd:
    @kubectl exec etcd-0 -n $(NAMESPACE) -- bash -c "etcdctl role add root && etcdctl user add root:$(ETCD_ROOT_PASSWORD) && etcdctl user  grant-role  root root && etcdctl auth enable"
 
#uninstall
 
.PHONY: uninstall-etcd
uninstall-etcd:
    @echo -n "Uninstall etcd and delete $(NAMESPACE) namespace. Are you sure? [y/N] " && read ans && [ ${ans:-N} = y ]
    helm uninstall etcd -n $(NAMESPACE)
    kubectl delete namespace $(NAMESPACE)

Installation

To install Etcd, run the following command:

make install-etcd ETCD_ROOT_PASSWORD=<password>

The command will create a namespace, secret, run helm to install etcd and finally upgrade etcd. An upgrade step is required to switch the initial cluster state attribute value from new to existing as described below.

In this paragraph, we will discuss about the initial cluster state attribute that plays a key role in Etcd installation. When Etcd cluster is provisioned, it sets the –initial-cluster-state attribute to new. This configuration works well for new clusters but pods won’t be able to join existing clusters upon restart. To fix this issue, we will need to update the flag from “new to existing” using the make install-etcd command.

The –initial-cluster-state attribute is defined in the statefulset i.e. 

InitialClusterState Initial cluster state. Allowed values: ‘new’ or ‘existing’. If this values is not set, the default values below are set:
~ ‘new’: when installing the chart (‘helm install …’)
~ ‘existing’: when upgrading the chart (‘helm upgrade …’)

Configuration

To configure Etcd, run the following command:

make init-etcd ETCD_ROOT_PASSWORD=<password>

The command will add a role, user and assign role to that user. It will also enable authentication.

Uninstallation

To uninstall Etcd, run the following command:

make uninstall-etcd

The command will uninstall Etcd and delete the namespace.

Conclusion

In conclusion, we covered how to install Etcd on Kubernetes in high availability mode. Since, we have an Etcd cluster running now, in the next blog, we will cover how to assign Etcd pods to specific nodes using node affinities.

Leave a Reply

Your email address will not be published. Required fields are marked *