Skip to main content

etcd Dashboard

Select Home screen > cluster project (CP) > Cluster > etcd Dashboard

Note

Requirements

  • The WhaTap Kubernetes agent 1.7.16 or later is required.

  • If it is a Kubernetes cluster project (CP), you can view the corresponding menu.

etcd is a core component in the Kubernetes environment that stores the data related to the cluster status. WhaTap provides etcd Dashboard to monitor the performance and status of etcd. This increases the stability of the cluster and enables a quick response when a problem occurs.

Basic screen guide

etcd dashboard

etcd Dashboard provides key etcd metrics collected over the last one hour at an interval of 1 minute. The key metrics you can see on the dashboard are as follows. For the metrics details, see the metric guide at the bottom.

  • The resources used by etcd, number of Pod restarts, and log details are provided.

  • The commits and number of applies in etcd are provided.

  • It provides the existence status of etcd leaders and the history of leader changes.

Activation of the etcd monitoring

Activate the agent by modifying the downloaded YAML file and registering the mTLS certificate when installing the WhaTap Kubernetes agent.

Checking the certificate and creating the secret

Checking the etcd certificate

An mTLS certificate is required for etcd. The etcd certificates are typically stored in the /etc/kubernetes/pki/etcd path on the control plane node. The file owner is the root account and the required certificates are as follows:

  • ca.crt

  • server.crt

  • server.key

On the machine where the kubectl command is executed, check if the certificate exists using the following command.

sudo ls /etc/kubernetes/pki/etcd

Creating the secret

If a certificate exists, create a secret with the following command.

sudo kubectl create secret generic whatap-etcd-cert -n whatap-monitoring --from-file=ca.crt=/etc/kubernetes/pki/etcd/ca.crt --from-file=server.crt=/etc/kubernetes/pki/etcd/server.crt --from-file=server.key=/etc/kubernetes/pki/etcd/server.key

When a secret is created successfully, the following message appears.

secret/whatap-etcd-cert created
In case the root account is unavailable

If you cannot use the kubectl command from the root account, see the following instructions.

  1. After copying the certificate, go to the accessible path.

  2. Change the owner of the duplicated certificate file to the current owner.

In case there is no namespace

If you create a secret before applying the Kubernetes agent YAML, the secret creation may fail because the whatap-monitoring namespace does not exist. After creating the namespace, create a secret again with the following command.

kubectl create namespace whatap-monitoring

Modifying the WhaTap Kubernetes agent YAML

Modify the WhaTap Kubernetes agent YAML file to add container settings and then attach the secret.

Configuring the container

  1. Add the etcd-related settings to the args section in the whatap-control-plane-helper container of Deployment.

    -collect_etcd_monitoring_enabled=true
    -etcd_hosts=user_etcd_ip, user_etcd_ip

    See the following example:

    Example
    -etcd_hosts=192.168.1.45,192.168.1.46,192.168.1.47
  2. Add the following to the environment variables (env) of the whatap-master-agent container of Deployment.

    - name: "collect_etcd_monitoring_enabled"
    value: "true"

Connecting the secret

  1. Add a secret to the volumes section in whatap-master-agent of Deployment.

    - name: "whatap-etcd-cert"
    secret:
    secretName: "whatap-etcd-cert"
  2. Mount the secret into the whatap-control-plane-helper container.

    volumeMounts:
    - name: "whatap-etcd-cert"
    readOnly: true
    mountPath: "/etc/kubernetes/pki/etcd"

Full YAML example

Below is an example of the YAML file with all settings applied. See the highlighted area.

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "whatap-master-agent"
namespace: "whatap-monitoring"
spec:
replicas: 1
selector:
matchLabels:
name: "whatap-master-agent"
template:
metadata:
labels:
name: "whatap-master-agent"
spec:
containers:
- command:
- "/bin/entrypoint.sh"
env:
- name: "WHATAP_LICENSE"
value: "x208gnc8i1kko-z2p19shk7npc4t-z788sgbj3n123a"
- name: "WHATAP_HOST"
value: "59.13.101.109"
- name: "WHATAP_PORT"
value: "61574"
- name: "WHATP_MEM_LIMIT"
valueFrom:
resourceFieldRef:
containerName: "whatap-master-agent"
resource: "limits.memory"
- name: "collect_control_plane_monitoring_enabled"
value: "true"
- name: "collect_etcd_monitoring_enabled"
value: "true"
image: "whatap/kube_mon"
name: "whatap-master-agent"
ports:
- containerPort: 6600
resources:
limits:
cpu: "200m"
memory: "350Mi"
requests:
cpu: "100m"
memory: "300Mi"
volumeMounts:
- mountPath: "/bin/entrypoint.sh"
name: "start-script-volume"
readOnly: true
subPath: "entrypoint.sh"
- mountPath: "/whatap_conf"
name: "whatap-config-volume"
- args:
- "-collect_control_plane_monitoring_enabled=true"
- "-collect_etcd_monitoring_enabled=true"
- "-etcd_hosts=192.168.1.45,192.168.1.46,192.168.1.47"
command:
- "/data/agent/master/whatap_control_plane_helper"
image: "whatap/kube_mon"
imagePullPolicy: "Always"
name: "whatap-control-plane-helper"
ports:
- containerPort: 9496
volumeMounts:
- name: "whatap-etcd-cert"
readOnly: true
mountPath: "/etc/kubernetes/pki/etcd"
serviceAccount: "whatap"
volumes:
- configMap:
defaultMode: 448
name: "master-start-script"
name: "start-script-volume"
- emptyDir: {}
name: "whatap-config-volume"
- name: "whatap-etcd-cert"
secret:
secretName: "whatap-etcd-cert"

etcd metrics

TypeDescription
Number of PodsNumber of the current etcd Pods
etcd restart count trendetcd Pod restart count sum trend
Reader change trendetcd Pod reader change count trend
Reader exists or notWhether the etcd Pod as the reader role exists or not
CPU usage sumetcd Pod CPU usage sum
CPU usageetcd Pod CPU usage trend
Memory usage sumetcd Pod memory usage sum
Memory usageetcd Pod memory usage trend
Commit trendNumber of etcd change commits
Apply trendNumber of applies after a commit
etcd logLogs generated from the etcd Pod