Kubernetes metrics
Some metrics collected by Kubernetes are the same as those of server and application monitoring.
Container(container) metric
The container category collects all custom labels set on the container's pods as tags.
- Target: Cluster project, Namespace project
- Collection interval: 5 seconds
- Statistical data: 5 minutes
Tags
| Tag name | Description | Remarks |
|---|---|---|
| agentOid | Node agent's unique ID | Unique value |
| agentPcode | pcode | Unique value |
| command | Execution command | - |
| containerId | Container ID | Unique value |
| containerKey | Container key | - |
| created | Time stamp generated by the container | - |
| image | Container image name | - |
| imageHash | Image hash value | - |
| imageId | Image ID | - |
| k8s-app | Value for the pod's label k8s-app | - |
| microOid | Unique ID of the WhaTap APM agent installed in the container | - |
| name | Container name | - |
| namespace | Namespace to which the container belongs | - |
| namespaceHash | Hash value of the namespace to which the container belongs | - |
| okind | Unique ID of OKIND specified in the WhaTap APM agent installed in the container | - |
| okindName | Name of OKIND specified in the WhaTap APM agent installed in the container | - |
| oname | Name of the WhaTap APM agent installed in the container | - |
| onode | Unique ID of the node agent on which the container is running | - |
| onodeName | Node name on which the container is running | - |
| podHash | Hash value of the container's Pod | - |
| podName | Container's Pod name | - |
| replicaSetHash | Hash value of the container's replica set | - |
| replicaSetName | Name of the container's replica set | - |
| whatap_project | Name of the WhaTab project to which the container belongs | - |
Fields
| Field name | Unit | Shortname, Name, Description |
|---|---|---|
| blkio_rbps | Byte | IoReadBytes |
| Container Block I/O Read Byte | ||
| Sum of bytes read per second across all block devices in the container | ||
| blkio_riops | Count | IoReadIops |
| Container Block I/O Read IOPS | ||
| Sum of counts read per second across all block devices in the container | ||
| blkio_wbps | Byte | IoWriteBytes |
| Container Block I/O Write Byte | ||
| Sum of bytes written per second across all block devices in the container | ||
| blkio_wiops | Count | IoWriteIops |
| Container Block I/O Write IOPS | ||
| Sum of counts written per second across all block devices in the container | ||
| cpu_per_quota | Percent | CpuByLimit |
| Container CPU Usage by Limit (%) | ||
| Container CPU utilization by limit | ||
| cpu_quota | Millicore | CpuLimit |
| Container CPU Limit (core) | ||
| Container CPU Limit Quota If the limit is not set, the total CPU cores of the node where the container is running appears in millicores. | ||
| cpu_quota_percent | Percent | CpuLimitByNode |
| Container CPU Limit by Node (%) | ||
| Container CPU Limit Quota against Node CPU If the limit is not set, the total CPU cores of the node where the container is running appears in percentage. | ||
| cpu_sys | Percent | CpuSysByNode |
| Container CPU Sys Usage by Node (%) | ||
| Container CPU System Utilization against Node CPU | ||
| cpu_throttledperiods | Count | CpuThrottledCnt |
| Container CPU Throttling Count | ||
| Container CPU Throttled Count | ||
| cpu_throttledtime | Nanosecond | CpuThrottledTime |
| Container CPU Throttling Time | ||
| Container CPU Throttled Time | ||
| cpu_total | Percent | CpuByNode |
| Container CPU Usage by Node (%) | ||
| Container CPU Utilization against Node CPU | ||
| cpu_total_milli | Millicore | CpuTotUsage |
| Container CPU Usage (millicore) | ||
| Container CPU Usage | ||
| cpu_user | Percent | CpuUserByNode |
| Container CPU User Usage by Node (%) | ||
| Container CPU User Utilization against Node CPU | ||
| cpu_request | Millicore | CpuRequest |
| Container CPU Request (core) | ||
| Container CPU Request | ||
| cpu_per_request | Percent | CpuByRequest |
| Container CPU Usage by Request (%) | ||
| Utilization against Container CPU Request = cpu_total_milli/cpu_request * 100 | ||
| mem_failcnt | Count | MemFailCnt |
| Container Memory Failure Count | ||
| Container Memory Limit reached Count | ||
| mem_limit | Byte | MemLimit |
| Container Memory Limit (byte) | ||
| Container Memory Limit Size | ||
| mem_maxusage | Byte | MemMaxUsage |
| Container Memory Max Usage (byte) | ||
| Recorded Value for Container Memory Maximum Usage | ||
| mem_percent | Percent | MemWsByLimit |
| Container Memory Working Set by Limit (%) | ||
| Working Set Usage based on Container Memory Limit = mem_usage/mem_limit * 100 | ||
| mem_totalcache | Byte | MemTotCache |
| Container Memory Total Cache (byte) | ||
| Container's Total Cache Size | ||
| mem_totalpgfault | Count | MemTotPageFaultCnt |
| Container Memory Total Page Fault Count | ||
| Container's Page Fault Count | ||
| mem_totalrss | Byte | MemTotRss |
| Container Memory Total RSS (byte) | ||
| Container's Total RSS Memory Size | ||
| mem_totalrss_percent | Percent | MemTotRssByLimit |
| Container Memory Total RSS By Limit (%) | ||
| Container's Total RSS Memory Utilization | ||
| mem_totalunevictable | Byte | MemTotUnevictable |
| Container Memory Total Unevictable (byte) | ||
| Container's Total Unevictable Memory Size | ||
| mem_usage | Byte | MemUsage |
| Container Memory Usage (byte) | ||
| Container Memory Usage | ||
| mem_working_set | Byte | MemWs |
| Container Memory Working Set (byte) | ||
| Container memory working set = mem_usage - inactive file | ||
| mem_working_set_percent | Percent | MemWsByLimit |
| Container Memory Working Set by Limit (%) | ||
| Working Set Usage based on Container Memory Limit = mem_usage/mem_limit * 100 | ||
| mem_request | Byte | MemRequest |
| Container Memory Request (byte) | ||
| Container Memory Request Size | ||
| mem_per_request | Percent | MemWsByRequest |
| Container Memory Working Set by Request (%) | ||
| Working Set Usage based on Container Memory Request = mem_working_set / mem_request * 100 | ||
| network_rbps | Byte | NetRxBytes |
| Container Network Receive Byte | ||
| Sum of bytes read per second across all block devices in the container | ||
| network_rdropped | Byte | NetRxDropped |
| Container Network Receive Dropped | ||
| Container Network Receive Dropped Count | ||
| network_rerror | Byte | NetRxError |
| Container Network Receive Error | ||
| Container Network Receive Error Count | ||
| network_riops | Byte | NetRxIops |
| Container Network Receive IOPS | ||
| Container Network Receive Error Count | ||
| network_wbps | Byte | NetTxByes |
| Container Network Transmit Byte | ||
| Container Network Transmit Data Size | ||
| network_wdropped | Count | NetTxDropped |
| Container Network Transmit Dropped | ||
| Container Network Transmit Dropped Count | ||
| network_werror | Count | NetTxError |
| Container Network Transmit Error | ||
| Container Network Transmit Error Count | ||
| network_wiops | Count | NetTxIops |
| Container Network Transmit IOPS | ||
| Container Network Transmit Error Count | ||
| node_cpu | Percent | ConNodeCpu |
| Container Work Node CPU Usage (%) | ||
| CPU Usage of the Node where the container is running | ||
| node_mem | Percent | ConNodeMem |
| Container Work Node Memory Usage (%) | ||
| Memory Usage of the Node where the container is running | ||
| phase | String | Pod lifecycle ① PENDING ② RUNNING ③ SUCCEEDED ④ FAILED ⑤ UNKNOWN |
| restart_count | Positive number | ConRestartCnt |
| Container Restart Count | ||
| Container Restart Count | ||
| state | Positive number | ConState |
| Container Current State | ||
| Container State Code ① RUNNING = 114 ② PAUSE = 112 ③ RESTARTING = 101 ④ OOMKILLED = 111M ⑤ DEAD = 100 ⑥ WAITING = 119 | ||
| status | String | ConStatus |
| Container Current Status | ||
| Container State Information ① running: Displays the uptime information ② waiting/terminated: Displays the reason of the state |
Kubernetes node (kube_node) metric
The kube_node category collects all custom labels set on the node as tags.
- Target: Cluster project, Namespace project
- Collection interval: 5 seconds
- Statistical data: 5 minutes, 1 hour
Tags
| Tag name | Description | Remarks |
|---|---|---|
| nodeName | Node name | - |
Fields
| Field name | Unit | Description | Remarks |
|---|---|---|---|
| allocatable_cpu | Millicore | CPU size that can be assigned to node | - |
| allocatable_memory | Byte | Memory size that can be assigned to node | - |
| allocatable_pods | Positive number | Number of Pods that can be assigned to node | - |
| limit_cpu | Millicore | Sum of node CPU limits | - |
| limit_memory | Byte | Sum of node memory limits | - |
| pods | Positive number | Total number of node Pods | - |
| request_cpu | Millicore | Sum of node CPU requests | - |
| request_memory | Byte | Sum of node memory requests | - |
Kubernetes event (kube_event) metric
The kube_event category collects cluster-wide data for cluster projects, and collects data only for events that occurred in the namespace for namespace projects.
- Target: Cluster project, Namespace project
- Collection interval: 5 seconds
- Statistical data: 5 minutes, 1 hour
Tags
| Tag name | Description | Remarks |
|---|---|---|
| field_path | Field Path | - |
| kind | Type | Object type on which the event occurred |
| name | Object name | Kubernetes object name on which the event occurred |
| namespace | Namespace name | Namespace on which the event occurred |
| reason | Event occurrence cause | - |
| type | Event type | Warning or normal |
| uid | UID | Object on which the event occurred |
Fields
| Field name | Unit | Description | Remarks |
|---|---|---|---|
| action | String | Action name | - |
| count | Count | Event occurrence count | - |
| event_time | Positive number | Time stamp for the first event | - |
| first_timestamp | Positive number | First event occurrence time | - |
| last_timestamp | Positive number | Last event occurrence time | - |
| message | String | Event Message | - |
| reasonFiled | String | Event reason | - |
| reporting_component | String | Component that reports the current event | - |
| reporting_instance | String | Instance that reports the current event | - |
| series_last_observed_time | Positive number | Series last observed time | - |
Kubernetes Cluster (kube_stat) metric
The kube_stat category collects all the clusters for the cluster project, and the namespace projects collects objects associated with the namespace.
- Target: Cluster project, Namespace project
- Collection interval: 5 seconds
- Statistical data: 5 minutes, 1 hour
Tags
| Tag name | Description | Remarks |
|---|---|---|
| name | kube_stat | Fixed value |
Fields
| Field name | Unit | Description | Remarks |
|---|---|---|---|
| alloctable_cpu | Millicore | Number of cluster cores | Cluster project only |
| alloctable_ephemeral-storage | Byte | Ephemeral storage that can be allocated to all clusters | Cluster project only |
| alloctable_hugepages-1gi | Byte | Hugepages-1Gi that can be allocated to all clusters | Cluster project only |
| alloctable_hugepages-2mi | Byte | Hugepages-2Gi that can be allocated to all clusters | Cluster project only |
| alloctable_memory | Byte | Memory that can be allocated to all clusters | Cluster project only |
| alloctable_pods | Positive number | Number of pods that can be allocated | - |
| available_pod | Positive number | Number of pods whose phase is in Running state | - |
| desired_pod | Positive number | Sum of the number of pods deployed without metadata.ownerReferences and the number of desired pods defined in Kubernetes objects (ReplicaSet, Daemonset, StatefulSet) | - |
Same as the number of pods retrieved by kubectl get pods -A | - | ||
| nodes | Positive number | Number of nodes | - |
| pod_phase_Pending | Positive number | Number of pending pods | - |
| pod_phase_Running | Positive number | Number of running pods | - |
| running_containers | Positive number | Number of running containers | - |
| stopped_containers | Positive number | Number of stopped containers | - |
| total_available_cpu | Positive number | Total allocatable CPU | - |
| total_available_memory | Positive number | Total sum of allocatable memory | - |
| total_limit_cpu | Millicore | Total sum of limit CPU | - |
| total_limit_memory | Byte | Total sum of limit memory | - |
| total_request_cpu | Millicore | Total sum of request CPU | - |
| total_request_memory | Byte | Total sum of request memory | - |
| unavailable_pod | Positive number | Number of pods whose phase is not in Running state (Pending, Failed, Succedded) | - |
| waiting_containers | Positive number | Waiting container count | - |
Pod (kube_pod) metric
The kube_pod category collects all custom labels set on the Pod as tags.
- Target: Master (cluster) project, Namespace project
- Collection interval: 5 seconds
- Statistical data: 5 minutes
Tags
| Tag name | Description | Remarks |
|---|---|---|
| agentOid | Node agent's unique ID | Unique value |
| agentPcode | pcode | Unique value |
| command | Execution command | - |
| containerIds | Container ID that belongs to the pod | - |
| containerIdsCount | Number of containerIds | - |
| containerKeys | Hash value for the container ID that belongs to the pod | - |
| containerKeysCount | Number of containerKeys | - |
| DaemonSet | DaemonSet name of the pod | - |
| Deployment | Deployment | - |
| k8s-app | Value for the pod's label k8s-app | - |
| microOid | ID of the agent running on the applications inside the pod's container. | - |
| microOids | Multiple IDs of the agents running on applications inside multiple containers in the pod | - |
| microOidsCount | Number of microOids | - |
| name | Pod Name | - |
| onames | Name of the agent running on the applications inside the pod's container. | - |
| onamesCount | Number of onames | - |
| podName | Pod Name | - |
| namespace | Namespace to which the Pod belongs | - |
| namespaceHash | Hash value of the namespace to which the Pod belongs | - |
| replicaSetHash | Hash value of ReplicaSet of the Pod | - |
| replicaSetName | ReplicaSet name of the Pod | - |
| whatap_project | Name of the WhaTab project to which the Pod belongs | - |
Fields
| Field name | Unit | (Shortname, Name, Description) |
|---|---|---|
| blkio_rbps | Byte | IoReadBytes |
| Pod Block I/O Read Byte | ||
| Sum of bytes read per second across all block devices in the Pod | ||
| blkio_riops | Count | IoReadIops |
| Pod Block I/O Read IOPS | ||
| Sum of cases read per second across all block devices in the Pod | ||
| blkio_wbps | Byte | IoWriteBytes |
| Pod Block I/O Write Byte | ||
| Sum of bytes written per second across all block devices in the Pod | ||
| blkio_wiops | Count | IoWriteIops |
| Pod Block I/O Write IOPS | ||
| Sum of cases written per second across all block devices in the Pod | ||
| cpu_per_limit | Percent | CpuByLimit |
| Pod CPU Usage by Limit (%) | ||
| Container CPU utilization by limit | ||
| cpu_per_request | Percent | CpuByRequest |
| Pod CPU Usage by Limit (%) | ||
| Container CPU utilization by limit | ||
| cpu_quota_percent | Percent | CpuLimitByNode |
| Pod CPU Limit by Node (%) | ||
| Pod CPU limit quota against node CPU If the limit is not set, the total CPU cores of the node where the Pod is running appears in percentage. | ||
| cpu_sys | Percent | CpuSysByNode |
| Pod CPU Sys Usage by Node (%) | ||
| Pod CPU System Utilization against Node CPU | ||
| cpu_throttledperiods | Count | CpuThrottledCnt |
| Pod CPU Throttling Count | ||
| Pod CPU Throttled Count | ||
| cpu_throttledtime | Nanosecond | CpuThrottledTime |
| Pod CPU Throttling Time | ||
| Pod CPU Throttled Time | ||
| cpu_total | Percent | CpuByNode |
| Pod CPU Usage by Node (%) | ||
| Pod CPU Utilization against Node CPU | ||
| cpu_total_milli | Millicore | CpuTotUsage |
| Pod CPU Usage (millicore) | ||
| Pod CPU Utilization | ||
| cpu_user | Percent | CpuUserByNode |
| Pod CPU User Usage by Node (%) | ||
| Pod CPU User Utilization against Node CPU | ||
| cpu_request | Millicore | CpuRequest |
| Pod CPU Request (core) | ||
| Pod CPU Request | ||
| cpu_per_request | Percent | CpuByRequest |
| Pod CPU Usage by Request (%) | ||
| Utilization against Pod CPU Request = cpu_total_milli/cpu_request * 100 | ||
| mem_totalcache | Byte | MemTotCache |
| Pod Memory Total Cache (byte) | ||
| Total Pod Cache Size | ||
| mem_totalpgfault | Count | MemTotPageFaultCnt |
| Pod Memory Total Page Fault Count | ||
| Pod's Page Fault Count | ||
| mem_totalrss | Byte | MemTotRss |
| Pod Memory Total RSS (byte) | ||
| Pod's Total RSS Memory Size | ||
| mem_totalrss_percent | Percent | MemTotRssByLimit |
| Pod Memory Total RSS by Limit (%) | ||
| Pod's Total RSS Memory Utilization | ||
| mem_totalunevictable | Byte | MemTotUnevictable |
| Pod Memory Total Unevictable (byte) | ||
| Pod's Total Unevictable Memory Size | ||
| mem_usage | Byte | MemUsage |
| Pod Memory Usage (byte) | ||
| Pod Memory Usage | ||
| mem_working_set | Byte | MemWs |
| Pod Memory Working Set (byte) | ||
| Pod Memory working set = mem_usage - inactive file | ||
| memory_request | Byte | MemRequest |
| Pod Memory Request (byte) | ||
| Pod memory requests | ||
| memory_limit | Byte | MemLimit |
| Pod Memory Limit (byte) | ||
| Pod memory limit quota | ||
| memory_per_request | Percent | MemByRequest |
| Pod Memory by Request (%) | ||
| Usage based on Pod memory requests | ||
| memory_per_limit | Percent | MemByLimit |
| Pod Memory by Limit (%) | ||
| Usage based on Pod memory limit | ||
| network_rbps | Byte | NetRxBytes |
| Pod Network Receive Byte | ||
| Sum of bytes read per second across all block devices in the Pod | ||
| network_rdropped | Byte | NetRxDropped |
| Pod Network Receive Dropped | ||
| Pod Network Receive Dropped Count | ||
| network_rerror | Byte | NetRxError |
| Pod Network Receive Error | ||
| Pod Network Receive Error Count | ||
| network_riops | Byte | NetRxIops |
| Pod Network Receive IOPS | ||
| Pod Network Receive Count | ||
| network_wbps | Byte | NetTxByes |
| Pod Network Transmit Byte | ||
| Pod Network Transmit Data Size | ||
| network_wdropped | Count | NetTxDropped |
| Pod Network Transmit Dropped | ||
| Pod Network Transmit Dropped Count | ||
| network_werror | Count | NetTxError |
| Pod Network Transmit Error | ||
| Pod Network Transmit Error Count | ||
| network_wiops | Count | NetTxIops |
| Pod Network Transmit IOPS | ||
| Pod Network Transmit Count | ||
| phase | String | Phase |
| Pod Current Phase | ||
| Pod lifecycle ① PENDING ② RUNNING ③ SUCCEEDED ④ FAILED ⑤ UNKNOWN |
The following fields are reserved for internal use.
| Field name | Description | Remarks |
|---|---|---|
| kube_sless_normal | Number of Kubernetes informative events | - |
| kube_sless_warning | Number of Kubernetes warning events | - |
| micro_sful_critical | Number of APM events that are critical | - |
| micro_sful_info | APM informative event count | - |
| micro_sful_warning | APM warning event count | - |
| micro_sless_critical | Number of APM events that are not critical | - |
| micro_sless_info | Number of APM events that are not informative | - |
| micro_sless_warning | Number of APM events that are not for warning | - |
| sful_critical | Number of events that are critical in the metric | - |
| sful_info | Number of events that are informative in the metric | - |
| sful_warning | Number of events that are for warning in the metric | - |
| sless_critical | Number of events that are not critical in the metric | - |
| sless_info | Number of events that are not informative in the metric | - |
| sless_warning | Number of events that are not for warning in the metric | - |
Kubernetes Pod Statistics (kube_pod_stat) metric
The kube_pod_stat category cluster project collects data for all clusters, and the namespace project collects data only for pods that belong to the namespace.
- Target: Cluster project, Namespace project
- Collection interval: 5 seconds
- Statistical data: 5 minutes, 1 hour
Tags
| Tag name | Description | Remarks |
|---|---|---|
| kind | Type | The cluster project has cluster as the fixed value, and the namespace projects collect only for Deployment or ReplicaSet. |
| name | Kubernetes resource name | The cluster project has no name value and the namespace projects have the name for Deployment or ReplicaSet. |
Fields
| Field name | Unit | Description | Remarks |
|---|---|---|---|
| available_pod | Positive number | Number of pods whose phase is in Running state | - |
| desired_pod | Positive number | Sum of the number of pods deployed without metadata.ownerReferences and the number of desired pods defined in Kubernetes objects (ReplicaSet, Daemonset, StatefulSet) | - |
Same as the number of pods retrieved by kubectl get pods -A | - | ||
| limit_cpu | Millicore | CPU Limit Usage | - |
| limit_memory | Byte | Memory Limit Usage | - |
| request_cpu | Millicore | CPU Request Usage | - |
| request_memory | Byte | Memory Request Usage | - |
| running_container | Positive number | Running Container Count | - |
| stopped_container | Positive number | Stopped Container Count | - |
| waiting_container | Positive number | Waiting container count | - |
Kubernetes Horizontal Pod Autoscaler (HPA) (kube_hpa_stat) metric
Metric collection starts only when HPA is added to the ClusterRole used by WhaTap.
- Target: Cluster project
- Collection interval: 5 seconds
- Statistical data: 5 minutes, 1 hour
Tags
| Tag name | Description | Remarks |
|---|---|---|
| name | HPA name | - |
Fields
| Field name | Unit | Description | Remarks |
|---|---|---|---|
| currentReplicas | Positive number | Current Replica Count | - |
| desiredReplicas | Positive number | Desired Replica Count | - |
| lastScaleTime | Positive number | Last scaled TimeStamp | - |
| maxReplicas | Positive number | Maximum Replica Count | - |
| minReplicas | Positive number | Minimum Replica Count | - |