Kubernetes metrics

This page lists the metrics collected by Kubernetes monitoring, organized by category. Each category includes its Tags (dimensions) and Fields (measurements).

• Some metrics are also used in server monitoring and application monitoring.
• To search or chart metrics directly, see Metrics Search and Metrics Chart.

Container(container) metrics

The container category collects all custom labels set on the container's pods as tags.

• Target: Cluster project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes

Tags

Tag	Type	Unit	Description
agentOid	-	-	Node agent ID (unique)
agentPcode	-	-	Project code (unique)
command	-	-	Execution command
containerId	-	-	Container ID (unique)
containerKey	-	-	Container key
created	-	-	Time stamp generated by the container
image	-	-	Container image name
imageHash	-	-	Image hash value
imageId	-	-	Image ID
k8s-app	-	-	Value for the pod's label k8s-app
microOid	-	-	Unique ID of the WhaTap APM agent installed in the container
name	-	-	Container name
namespace	-	-	Namespace to which the container belongs
namespaceHash	-	-	Hash value of the namespace to which the container belongs
okind	-	-	Unique ID of OKIND specified in the WhaTap APM agent installed in the container
okindName	-	-	Name of OKIND specified in the WhaTap APM agent installed in the container
oname	-	-	Name of the WhaTap APM agent installed in the container
onode	-	-	Unique ID of the node agent on which the container is running
onodeName	-	-	Node name on which the container is running
podHash	-	-	Hash value of the container's Pod
podName	-	-	Container's Pod name
replicaSetHash	-	-	Hash value of the container's replica set
replicaSetName	-	-	Name of the container's replica set
whatap_project	-	-	Name of the WhaTap project to which the container belongs

Fields

Field	Type	Unit	Shortname/Name/Description
blkio_rbps	-	byte	IoReadBytes
			Container Block I/O Read Byte
			Sum of bytes read per second across all block devices in the container
blkio_riops	-	count	IoReadIops
			Container Block I/O Read IOPS
			Sum of counts read per second across all block devices in the container
blkio_wbps	-	byte	IoWriteBytes
			Container Block I/O Write Byte
			Sum of bytes written per second across all block devices in the container
blkio_wiops	-	count	IoWriteIops
			Container Block I/O Write IOPS
			Sum of counts written per second across all block devices in the container
cpu_per_quota	-	percent	CpuByLimit
			Container CPU usage by limit (%)
			Container CPU utilization by limit
cpu_quota	-	millicores	CpuLimit
			Container CPU Limit (core)
			Container CPU Limit Quota If the limit is not set, the total CPU cores of the node where the container is running appears in millicores
cpu_quota_percent	-	percent	CpuLimitByNode
			Container CPU Limit by Node(%)
			Container CPU Limit quota relative to node CPU - If Limit is not set, the total CPU cores of the node where the container is running are displayed as a percentage
cpu_sys	-	percent	CpuSysByNode
			Container CPU Sys Usage by Node(%)
			Container CPU System Utilization against Node CPU
cpu_throttledperiods	-	count	CpuThrottledCnt
			Container CPU Throttling Count
			Container CPU Throttled Count
cpu_throttledtime	-	nanosecond(ns)	CpuThrottledTime
			Container CPU Throttling Time
			Container CPU Throttled Time
cpu_total	-	percent	CpuByNode
			Container CPU Usage by Node(%)
			Container CPU Utilization against Node CPU
cpu_total_milli	-	millicores	CpuTotUsage
			Container CPU Usage (millicore)
			Container CPU Usage
cpu_user	-	percent	CpuUserByNode
			Container CPU User Usage by Node(%)
			Container CPU User Utilization against Node CPU
cpu_request	-	millicores	CpuRequest
			Container CPU Request (core)
			Container CPU Request
cpu_per_request	-	percent	CpuByRequest
			Container CPU Usage by Request(%)
			Utilization against Container CPU Request = cpu_total_milli / cpu_request * 100
mem_failcnt	-	count	MemFailCnt
			Container Memory Failure Count
			Container Memory Limit reached Count
mem_limit	-	byte	MemLimit
			Container Memory Limit (byte)
			Container Memory Limit Size
mem_maxusage	-	byte	MemMaxUsage
			Container Memory Max Usage (byte)
			Recorded maximum container memory usage See the guide below for details.
mem_percent	-	percent	MemWsByLimit
			Container Memory Working Set by Limit(%)
			Working Set Usage based on Container Memory Limit = mem_usage / mem_limit * 100
mem_totalcache	-	byte	MemTotCache
			Container Memory Total Cache (byte)
			Container's Total Cache Size
mem_totalpgfault	-	count	MemTotPageFaultCnt
			Container Memory Total Page Fault Count
			Container's Page Fault Count
mem_totalrss	-	byte	MemTotRss
			Container Memory Total RSS (byte)
			Container's Total RSS Memory Size
mem_totalrss_percent	-	percent	MemTotRssByLimit
			Container Memory Total RSS By Limit (%)
			Container's Total RSS Memory Utilization
mem_totalunevictable	-	byte	MemTotUnevictable
			Container Memory Total Unevictable (byte)
			Container's Total Unevictable Memory Size
mem_usage	-	byte	MemUsage
			Container Memory Usage (byte)
			Container Memory Usage
mem_working_set	-	byte	MemWs
			Container Memory Working Set (byte)
			Container memory working set = mem_usage - inactive file
mem_working_set_percent	-	percent	MemWsByLimit
			Container Memory Working Set by Limit (%)
			Working Set Usage based on Container Memory Limit = mem_usage / mem_limit * 100
mem_request	-	byte	MemRequest
			Container Memory Request (byte)
			Container Memory Request Size
mem_per_request	-	percent	MemWsByRequest
			Container Memory Working Set by Request (%)
			Working Set Usage based on Container Memory Request = mem_working_set / mem_request * 100
network_rbps	-	byte	NetRxBytes
			Container Network Receive Byte
			Sum of bytes read per second across all block devices in the container
network_rdropped	-	byte	NetRxDropped
			Container Network Receive Dropped
			Container Network Receive Dropped Count
network_rerror	-	byte	NetRxError
			Container Network Receive Error
			Container Network Receive Error Count
network_riops	-	byte	NetRxIops
			Container Network Receive IOPS
			Container Network Receive Error Count
network_wbps	-	byte	NetTxByes
			Container Network Transmit Byte
			Container Network Transmit Data Size
network_wdropped	-	count	NetTxDropped
			Container Network Transmit Dropped
			Container Network Transmit Dropped Count
network_werror	-	count	NetTxError
			Container Network Transmit Error
			Container Network Transmit Error Count
network_wiops	-	count	NetTxIops
			Container Network Transmit IOPS
			Container Network Transmit Error Count
node_cpu	-	percent	ConNodeCpu
			Container Work Node CPU Usage (%)
			CPU Usage of the Node where the container is running
node_mem	-	percent	ConNodeMem
			Container Work Node Memory Usage (%)
			Memory Usage of the Node where the container is running
phase	string	-	Pod lifecycle ① PENDING ② RUNNING ③ SUCCEEDED ④ FAILED ⑤ UNKNOWN
restart_count	integer	-	ConRestartCnt
			Container Restart Count
			Number of container restarts
state	integer	-	ConState
			Container Current State
			Container State Code ① RUNNING = 114 ② PAUSE = 112 ③ RESTARTING = 101 ④ OOMKILLED = 111 ⑤ DEAD = 100 ⑥ WAITING = 119
status	string	-	ConStatus
			Container Current Status
			Container State Information ① running: Displays the uptime information ② waiting/terminated: Displays the reason of the state

Note

mem_maxusage indicates the maximum memory usage recorded while the container was running. However, if the Linux kernel version is lower than 5.19, the raw data for this metric may not be supported. In this case, the value may be displayed as 0. Update the Linux kernel version to 5.19 or later to collect this metric.

Kubernetes node (kube_node) metrics

The kube_node category collects all custom labels set on the node as tags.

• Target: Cluster project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes, 1 hour

Tags

Tag	Type	Unit	Description
nodeName	-	-	Node name

The following tag has been added.

Tag	Type	Unit	Description
infrastructureType	string	-	Node infrastructure type - baremetal, vm, unknown

Fields

Field	Type	Unit	Description
allocatable_cpu	-	millicores	CPU size that can be assigned to node
allocatable_memory	-	byte	Memory size that can be assigned to node
allocatable_pods	integer	-	Number of Pods that can be assigned to node
limit_cpu	-	millicores	Sum of node CPU limits
limit_memory	-	byte	Sum of node memory limits
pods	integer	-	Total number of node Pods
request_cpu	-	millicores	Sum of node CPU requests
request_memory	-	byte	Sum of node memory requests

The GPU quantity of a node is collected as parallel list fields by device type. *_types and *_counts correspond 1:1 by index, and hold nvidia.com/gpu (whole-GPU) and all MIG profiles (nvidia.com/mig-*) as individual entries.

Field	Type	Unit	Description
gpu_capacity_types	list	-	List of device types of the node GPU capacity
gpu_capacity_counts	list	count	Quantity per type above for capacity (1:1 index correspondence)
gpu_allocatable_types	list	-	List of device types of the node GPU allocatable
gpu_allocatable_counts	list	count	Quantity per type above for allocatable (1:1 index correspondence)

Note

Starting from v1.9.14, the whole-GPU-only scalar fields capacity_gpu and allocatable_gpu have been removed. This prevents the problem where, in MIG environments, the scalar fields hold only nvidia.com/gpu and omit MIG slices, making the aggregation a false total. The whole-GPU quantity is also checked from the nvidia.com/gpu entry in the list, and the total and per-type aggregations are the sum of *_counts.

Category	Removed scalar field	Replacement list field
kube_node	capacity_gpu, allocatable_gpu	gpu_capacity_types/gpu_capacity_counts, gpu_allocatable_types/gpu_allocatable_counts

Kubernetes node performance (kube_node_perf) metrics

The kube_node_perf category collects node-level performance metrics by summing up the resource usage and allocation of all Pods/Containers scheduled on the node.

• Target: Cluster project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes, 1 hour

Tags

Tag	Type	Unit	Description
onode	-	-	Node hash value
onodeName	-	-	Node name
agentPcode	-	-	Agent project code
agentOid	-	-	Agent unique ID

The following tag has been added.

Tag	Type	Unit	Description
cgroup_version	-	-	cgroup mount mode of the node host - unified (cgroup v2), hybrid, legacy (cgroup v1), unknown (inaccessible)

The /sys/fs/cgroup mount mode of the node host is detected directly and collected as a tag. You can aggregate the cgroup version status per node on the server.

The node virtualization determination tags have been added.

Tag	Type	Unit	Description
virtualization	string	-	Whether the node is virtualized - baremetal, vm, unknown
hypervisor	string	-	Hypervisor vendor name - - for bare metal or when it cannot be determined

The vendor is first cross-checked against the DMI information of the node host (/sys/class/dmi/id/sys_vendor, product_name). If no vendor matches, only whether it is a virtual machine is determined by the hypervisor flag in /proc/cpuinfo. Non-x86 nodes without DMI information are collected as unknown.

Vendors such as QEMU/KVM, VMware, Hyper-V, Xen, VirtualBox, Parallels, Amazon EC2, Google GCE, OpenStack, Nutanix AHV, oVirt/RHV, and Alibaba Cloud are recognized. The value does not change while the node is running, so it is read once and cached.

Constraint: Bare metal provided by a cloud provider, such as an Amazon EC2 .metal instance, is not distinguishable from a general instance by DMI information alone and is determined as vm.

Collected on node agent 1.9.21 or later.

Fields

Field	Type	Unit	Description
cpuRequest	long	millicores	Total CPU Request sum of all containers in the node
cpuLimit	long	millicores	Total CPU Limit sum of all containers in the node
cpuTotal	long	millicores	Total CPU usage sum of all containers in the node
cpuPerRequest	float	percent	Actual CPU utilization relative to node CPU Request
cpuPerLimit	float	percent	Actual CPU utilization relative to node CPU Limit
memoryRequest	long	byte	Total Memory Request sum of all containers in the node
memoryLimit	long	byte	Total Memory Limit sum of all containers in the node
memTotal	long	byte	Total Memory usage sum of all containers in the node
memoryPerRequest	float	percent	Actual Memory utilization relative to node Memory Request
memoryPerLimit	float	percent	Actual Memory utilization relative to node Memory Limit

The following fields have been added.

Field	Type	Unit	Description
conntrack_count	long	count	Current number of conntrack entries
conntrack_max	long	count	Upper limit of conntrack entries (nf_conntrack_max)
conntrack_pct	float	percent	Usage against the upper limit

• The number of entries is collected from the /proc/1/net/stat/nf_conntrack path to read the value based on the host network namespace.
• Nodes where the nf_conntrack module is not loaded do not send the fields. This behavior distinguishes the value 0 from non-collection.
• Collected on node agent 1.9.18 or later. You can turn the collection off with the collect_conntrack_enabled option.

Kubernetes event (kube_event) metrics

The kube_event category collects cluster-wide data for cluster projects, and collects data only for events that occurred in the namespace for namespace projects.

• Target: Cluster project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes, 1 hour

Tags

Tag	Type	Unit	Description
field_path	-	-	Field Path
kind	-	-	Object type on which the event occurred
name	-	-	Kubernetes object name on which the event occurred
namespace	-	-	Namespace on which the event occurred
reason	-	-	Event occurrence cause
type	-	-	Event type - Warning or Normal
uid	-	-	UID - Object where an event occurred

Fields

Field	Type	Unit	Description
action	string	-	Action name
count	-	count	Event occurrence count
event_time	integer	-	Time stamp for the first event
first_timestamp	integer	-	First event occurrence time
last_timestamp	integer	-	Last event occurrence time
message	string	-	Event Message
reasonFiled	string	-	Event reason
reporting_component	string	-	Component that reports the current event
reporting_instance	string	-	Instance that reports the current event
series_last_observed_time	integer	-	Series last observed time

Kubernetes Cluster (kube_stat) metrics

The kube_stat category collects all the clusters for the cluster project, and the namespace projects collects objects associated with the namespace.

• Target: Cluster project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes, 1 hour

Tags

Tag	Type	Unit	Description
name	-	-	kube_stat (fixed value)

The following tag has been added.

Tag	Type	Unit	Description
namespace	-	-	Name of the namespace being collected

Fields

Field	Type	Unit	Description
alloctable_cpu	-	millicores	Number of all cores in the cluster (cluster project only)
alloctable_ephemeral-storage	-	byte	Cluster-wide allocatable ephemeral storage (cluster project only)
alloctable_hugepages-1gi	-	byte	Cluster-wide allocatable hugepages-1Gi (cluster project only)
alloctable_hugepages-2mi	-	byte	Cluster-wide allocatable hugepages-2Mi (cluster project only)
alloctable_memory	-	byte	Total memory allocatable in the cluster (cluster project only)
alloctable_pods	integer	-	Number of pods that can be allocated
available_pod	integer	-	Number of pods whose phase is in Running state
desired_pod	integer	-	Sum of the number of pods deployed without metadata.ownerReferences and the number of desired pods defined in Kubernetes objects (ReplicaSet, Daemonset, StatefulSet)
			Same as the number of pods retrieved by kubectl get pods -A
nodes	integer	-	Number of nodes
pod_phase_Pending	integer	-	Number of pending pods
pod_phase_Running	integer	-	Number of running pods
running_containers	integer	-	Number of running containers
stopped_containers	integer	-	Number of stopped containers
total_available_cpu	integer	-	Total allocatable CPU
total_available_memory	integer	-	Total sum of allocatable memory
total_limit_cpu	-	millicores	Total sum of limit CPU
total_limit_memory	-	byte	Total sum of limit memory
total_request_cpu	-	millicores	Total sum of request CPU
total_request_memory	-	byte	Total sum of request memory
unavailable_pod	integer	-	Number of pods whose phase is not in Running state (Pending, Failed, Succedded)
waiting_containers	integer	-	Waiting container count

The upper limits and consumption of the ResourceQuota configured for a namespace are collected. The following fields have been added.

Field	Type	Unit	Description
quota_count	long	count	Number of ResourceQuotas configured for the namespace
quota_request_cpu_hard	long	millicores	Upper limit of the CPU Request quota
quota_request_cpu_used	long	millicores	Consumption of the CPU Request quota
quota_request_memory_hard	long	byte	Upper limit of the Memory Request quota
quota_request_memory_used	long	byte	Consumption of the Memory Request quota
quota_limit_cpu_hard	long	millicores	Upper limit of the CPU Limit quota
quota_limit_cpu_used	long	millicores	Consumption of the CPU Limit quota
quota_limit_memory_hard	long	byte	Upper limit of the Memory Limit quota
quota_limit_memory_used	long	byte	Consumption of the Memory Limit quota
quota_pods_hard	long	count	Upper limit of the Pod count quota
quota_pods_used	long	count	Consumption of the Pod count quota

• The status.hard and status.used values calculated by Kubernetes are used as they are.
• When a namespace has several ResourceQuotas with different scopes, the values are summed per item.
• A namespace with no quota configured sends only quota_count as 0 and does not send the remaining fields. This behavior distinguishes no quota from an upper limit of 0.
• Collected on master agent 1.9.17 or later.

Permission: The list permission for the core API resourcequotas is required in the ClusterRole. Without the permission, only the ResourceQuota collection is skipped and the rest of the kube_stat collection continues.

Pod (kube_pod) metrics

The kube_pod category collects all custom labels set on the Pod as tags.

• Target: Master (cluster) project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes

Tags

Tag	Type	Unit	Description
agentOid	-	-	Node agent ID (unique)
agentPcode	-	-	Project code (unique)
command	-	-	Execution command
containerIds	-	-	Container ID that belongs to the Pod
containerIdsCount	-	-	Number of containerIds
containerKeys	-	-	Hash value for the container ID that belongs to the pod
containerKeysCount	-	-	Number of containerKeys
DaemonSet	-	-	DaemonSet name of the pod
Deployment	-	-	Deployment
k8s-app	-	-	Value for the pod's label k8s-app
microOid	-	-	ID of the agent running on the applications inside the Pod's container.
microOids	-	-	Multiple IDs of the agents running on applications inside multiple containers in the pod
microOidsCount	-	-	Number of microOids
name	-	-	Pod Name
onames	-	-	Name of the agent running on the applications inside the Pod's container.
onamesCount	-	-	Number of onames
podName	-	-	Pod Name
namespace	-	-	Namespace to which the Pod belongs
namespaceHash	-	-	Hash value of the namespace to which the Pod belongs
replicaSetHash	-	-	Hash value of ReplicaSet of the Pod
replicaSetName	-	-	ReplicaSet name of the Pod
whatap_project	-	-	Name of the WhaTap project to which the Pod belongs

Fields

Field	Type	Unit	Shortname, Name, Description
blkio_rbps	-	byte	IoReadBytes
			Pod Block I/O Read Byte
			Sum of bytes read per second across all block devices in the Pod
blkio_riops	-	count	IoReadIops
			Pod Block I/O Read IOPS
			Sum of cases read per second across all block devices in the Pod
blkio_wbps	-	byte	IoWriteBytes
			Pod Block I/O Write Byte
			Sum of bytes written per second across all block devices in the Pod
blkio_wiops	-	count	IoWriteIops
			Pod Block I/O Write IOPS
			Sum of cases written per second across all block devices in the Pod
cpu_per_limit	-	percent	CpuByLimit
			Pod CPU Usage by Limit (%)
			Container CPU utilization by limit
cpu_per_request	-	percent	CpuByRequest
			Pod CPU Usage by Request (%)
			Total CPU utilization based on the CPU requests
cpu_quota_percent	-	percent	CpuLimitByNode
			Pod CPU Limit by Node (%)
			Pod CPU limit quota against the node - If the limit is not set, the total CPU cores of the node where the Pod is running appears in percentage.
cpu_sys	-	percent	CpuSysByNode
			Pod CPU Sys Usage by Node (%)
			Pod CPU System Utilization against Node CPU
cpu_throttledperiods	-	count	CpuThrottledCnt
			Pod CPU Throttling Count
			Pod CPU Throttled Count
cpu_throttledtime	-	nanosecond(ns)	CpuThrottledTime
			Pod CPU Throttling Time
			Pod CPU Throttled Time
cpu_total	-	percent	CpuByNode
			Pod CPU Usage by Node (%)
			Pod CPU Utilization against Node CPU
cpu_total_milli	-	millicores	CpuTotUsage
			Pod CPU Usage (millicore)
			Pod CPU usage
cpu_user	-	percent	CpuUserByNode
			Pod CPU User Usage by Node (%)
			Pod CPU User Utilization against Node CPU
cpu_request	-	millicores	CpuRequest
			Pod CPU Request (core)
			Pod CPU Request
cpu_per_request	-	percent	CpuByRequest
			Pod CPU Usage by Request (%)
			Utilization against Pod CPU Request = cpu_total_milli/cpu_request * 100
mem_totalcache	-	byte	MemTotCache
			Pod Memory Total Cache (byte)
			Total Pod Cache Size
mem_totalpgfault	-	count	MemTotPageFaultCnt
			Pod Memory Total Page Fault Count
			Pod's Page Fault Count
mem_totalrss	-	byte	MemTotRss
			Pod Memory Total RSS (byte)
			Pod's Total RSS Memory Size
mem_totalrss_percent	-	percent	MemTotRssByLimit
			Pod Memory Total RSS by Limit (%)
			Pod's Total RSS Memory Utilization
mem_totalunevictable	-	byte	MemTotUnevictable
			Pod Memory Total Unevictable (byte)
			Pod's Total Unevictable Memory Size
mem_usage	-	byte	MemUsage
			Pod Memory Usage (byte)
			Pod Memory Usage
mem_working_set	-	byte	MemWs
			Pod Memory Working Set (byte)
			Pod Memory working set = mem_usage - inactive file
memory_request	-	byte	MemRequest
			Pod Memory Request (byte)
			Pod memory requests
memory_limit	-	byte	MemLimit
			Pod Memory Limit (byte)
			Pod memory limit quota
memory_per_request	-	percent	MemByRequest
			Pod Memory Working Set by Request (%)
			Working Set usage based on the Pod memory request
memory_per_limit	-	percent	MemByLimit
			Pod Memory Working Set by Limit (%)
			Working Set usage based on the Pod memory limit
network_rbps	-	byte	NetRxBytes
			Pod Network Receive Byte
			Sum of bytes read per second across all block devices in the Pod
network_rdropped	-	byte	NetRxDropped
			Pod Network Receive Dropped
			Pod Network Receive Dropped Count
network_rerror	-	byte	NetRxError
			Pod Network Receive Error
			Pod's network receive error count
network_riops	-	byte	NetRxIops
			Pod Network Receive IOPS
			Pod Network Receive Count
network_wbps	-	byte	NetTxByes
			Pod Network Transmit Byte
			Pod Network Transmit Data Size
network_wdropped	-	count	NetTxDropped
			Pod Network Transmit Dropped
			Pod Network Transmit Dropped Count
network_werror	-	count	NetTxError
			Pod Network Transmit Error
			Pod Network Transmit Error Count
network_wiops	-	count	NetTxIops
			Pod Network Transmit IOPS
			Pod Network Transmit Count
phase	string	-	Phase
			Pod Current Phase
			Pod lifecycle ① PENDING ② RUNNING ③ SUCCEEDED ④ FAILED ⑤ UNKNOWN

The following fields are reserved for internal use.

Field	Type	Unit	Description
kube_sless_normal	-	-	Number of Kubernetes informative events
kube_sless_warning	-	-	Number of Kubernetes warning events
micro_sful_critical	-	-	Number of APM events that are critical
micro_sful_info	-	-	APM informative event count
micro_sful_warning	-	-	APM warning event count
micro_sless_critical	-	-	Number of APM events that are not critical
micro_sless_info	-	-	Number of APM events that are not informative
micro_sless_warning	-	-	Number of APM events that are not for warning
sful_critical	-	-	Number of events that are critical in the metric
sful_info	-	-	Number of events that are informative in the metric
sful_warning	-	-	Number of events that are for warning in the metric
sless_critical	-	-	Number of events that are not critical in the metric
sless_info	-	-	Number of events that are not informative in the metric
sless_warning	-	-	Number of events that are not for warning in the metric

The GPU Request and Limit quantities of a Pod are collected as parallel list fields by device type. *_types and *_counts correspond 1:1 by index, and hold nvidia.com/gpu (whole-GPU) and all MIG profiles (nvidia.com/mig-*) as individual entries.

Field	Type	Unit	Description
gpu_request_types	list	-	List of device types of the Pod GPU Request
gpu_request_counts	list	count	Quantity per type above for Request (1:1 index correspondence)
gpu_limit_types	list	-	List of device types of the Pod GPU Limit
gpu_limit_counts	list	count	Quantity per type above for Limit (1:1 index correspondence)

Note

Starting from v1.9.14, the whole-GPU-only scalar fields gpu_request and gpu_limit have been removed. This prevents the problem where, in MIG environments, the scalar fields hold only nvidia.com/gpu and omit MIG slices, making the aggregation a false total. The whole-GPU quantity is also checked from the nvidia.com/gpu entry in the list, and the total and per-type aggregations are the sum of *_counts.

Kubernetes Pod Statistics (kube_pod_stat) metrics

The kube_pod_stat category cluster project collects data for all clusters, and the namespace project collects data only for pods that belong to the namespace.

• Target: Cluster project, Namespace project
• Collection interval: 5 seconds
• Statistical data: 5 minutes, 1 hour

Tags

Tag	Type	Unit	Description
kind	-	-	Type - A cluster project has its fixed value, and a namespace project collects only the deployment or ReplicaSet.
name	-	-	Kubernetes resource name - A cluster project has no name value and a namespace project has the name for Deployment or ReplicaSet.

Fields

Field	Type	Unit	Description
available_pod	integer	-	Number of pods whose phase is in Running state
desired_pod	integer	-	Sum of the number of pods deployed without metadata.ownerReferences and the number of desired pods defined in Kubernetes objects (ReplicaSet, Daemonset, StatefulSet)
			Same as the number of pods retrieved by kubectl get pods -A
limit_cpu	-	millicores	CPU Limit Usage
limit_memory	-	byte	Memory Limit Usage
request_cpu	-	millicores	CPU Request Usage
request_memory	-	byte	Memory Request Usage
running_container	integer	-	Running Container Count
stopped_container	integer	-	Stopped Container Count
waiting_container	integer	-	Waiting container count

Kubernetes Pod volume (kube_pod_volume) metrics

The kube_pod_volume category collects the capacity and i-node usage of volumes mounted on Pods running on a node. Since the directory name of an app Pod's PVC volume is the PV name, you can join it with the PVC's spec.volumeName using the volumeName tag.

Tags

Tag	Type	Unit	Description
hostDevice	-	-	Host device where the volume is located
hostFileSystem	-	-	File system of the volume
hostMountPath	-	-	Volume mount path on the host
volumeName	-	-	Volume name - For a PVC volume, the PV name (e.g. pvc-...). Can be joined with the PVC's spec.volumeName
volumePlugin	-	-	Volume plugin - e.g. kubernetes.io~csi, kubernetes.io~nfs, volume-subpaths

Fields

Field	Type	Unit	Description
capacity	long	byte	Total volume capacity
available	long	byte	Available volume capacity
availablePercent	float	percent	Available volume capacity ratio
usedPercent	float	percent	Volume usage
free	long	byte	Free volume capacity
freePercent	float	percent	Free volume capacity ratio
totalInode	long	count	Total number of volume i-nodes
usedInode	long	count	Number of i-nodes in use in the volume
usedInodePercent	float	percent	Volume i-node usage

emptyDir volume collection has been added. A default (disk) medium emptyDir is a directory of the node root file system rather than a separate mount, so it was not caught by the df-based collection. The node agent queries the kubelet volume directory directly and sends the data to the same kube_pod_volume category.

• Query path: {HOST_PREFIX}{kubelet_root_dir}/pods/{podUid}/volumes/kubernetes.io~empty-dir/{volumeName}
• Collection interval: 60 seconds. Because querying directories is expensive, it is separated from the 5-second collection interval.
• The volumePlugin tag value of an emptyDir volume is kubernetes.io~empty-dir.

The following tags have been added.

Tag	Type	Unit	Description
volumeType	string	-	Volume type - EmptyDir for an emptyDir volume
emptyDirMedium	string	-	Storage medium of the emptyDir - Memory or Disk. Based on emptyDir.medium of the Pod spec; unspecified is Disk
emptyDirSpecConfirmed	string	-	Whether the Pod spec was checked - true is a value confirmed by the spec, false is a value estimated from the kubelet path and the file system type

The following fields have been added.

Field	Type	Unit	Description
usedSpace	long	byte	Total volume usage
fileCount	long	count	Number of files in the volume
sizeLimit	long	byte	emptyDir.sizeLimit of the Pod spec - Not sent when it is not set
truncated	integer	-	Whether the collection limit was exceeded - 1 means the value is smaller than the actual one because the limit was hit

capacity, available, and usedPercent are sent only when the volume capacity can be determined.

• When sizeLimit exists, that value is used as the volume capacity.
• When there is no sizeLimit and the medium is Memory, the total tmpfs size created by kubelet is used as the capacity. The default total tmpfs size is half of the node memory.
• A Disk medium without sizeLimit has no upper limit, so no capacity is created. In this case, compare it with the Pod's container.ephemeral_storage_limit.

All of the tags and fields above are sent on node agent 1.9.23 or later. For the collection activation and cost limit options, see Configuring the options for the Kubernetes agent.

Kubernetes Horizontal Pod Autoscaler (HPA) (kube_hpa_stat) metrics

Metric collection starts only when HPA is added to the ClusterRole used by WhaTap.

• Target: Cluster project
• Collection interval: 5 seconds
• Statistical data: 5 minutes, 1 hour

Tags

Tag	Type	Unit	Description
name	-	-	HPA name

The following tags have been added.

Tag	Type	Unit	Description
scaleTargetKind	string	-	Kind of the scale target object (spec.scaleTargetRef.kind)
scaleTargetName	string	-	Name of the scale target object (spec.scaleTargetRef.name)
scaleTargetApiVersion	string	-	API version of the scale target object
creationTimeMillis	long	-	HPA creation time (epoch ms) - Not sent when there is no value

The scale target of an HPA is in spec.scaleTargetRef. The owner_kind, owner_name, and owner_id tags based on ownerReferences are mostly empty for an HPA, so check the scale target with the scaleTarget* tags. The existing tags are kept for backward compatibility.

Collected on master agent 1.9.24 or later.

Fields

Field	Type	Unit	Description
currentReplicas	integer	count	Current Replica Count
desiredReplicas	integer	count	Desired Replica Count
lastScaleTime	integer	count	Last scaled TimeStamp
maxReplicas	integer	count	Maximum Replica Count
minReplicas	integer	count	Minimum Replica Count

Kubernetes HPA list (kube_hpa) metrics

The kube_hpa category collects HPA list data equivalent to the result of kubectl get hpa -o wide. It is a separate category from kube_hpa_stat, which is a time series metric.

• Target: Cluster project
• Collection interval: 30 seconds
• Required version: Master agent 1.9.24 or later

Tags

Tag	Type	Unit	Description
hpaUid	string	-	HPA UID (unique value)
hpaName	string	-	HPA name
namespace	string	-	Namespace the HPA belongs to
creationTimeMillis	long	-	HPA creation time (epoch ms)
scaleTargetKind	string	-	Kind of the scale target object
scaleTargetName	string	-	Name of the scale target object
scaleTargetApiVersion	string	-	API version of the scale target object

Fields

Field	Type	Unit	Description
minReplicas	integer	count	Minimum replica count
maxReplicas	integer	count	Maximum replica count
currentReplicas	integer	count	Current replica count
desiredReplicas	integer	count	Desired replica count
targetMetricType	list	-	List of metric types used as the scaling criteria
targetValue	list	-	List of target values per metric
currentValue	list	-	List of current values per metric
conditionAbleToScale	string	-	Status of the AbleToScale condition
conditionScalingActive	string	-	Status of the ScalingActive condition
conditionScalingLimited	string	-	Status of the ScalingLimited condition

targetMetricType, targetValue, and currentValue are parallel lists where the same index corresponds to one metric. An HPA can have CPU, memory, and custom metrics at the same time, so it cannot be expressed with a single value.

• The values are strings in the same notation as kubectl (for example, 50%, 100m).
• When the current value cannot be obtained because metrics-server is absent, it is collected as <unknown>.
• Besides Resource metrics, Pods, External, and Object metrics are stored with the pods:, external:, and object: prefixes respectively.
• autoscaling/v2 is used first, and clusters that do not support it fall back to autoscaling/v1.

The horizontalpodautoscalers permission is already included in the ClusterRole, so no additional permission setting is required.

Process (kube_process) metrics

Note

Kubernetes agent 1.7.12 or later is required. For more information about agent updates, see the following.

Kubernetes-related processes that exist in the node are collected during monitoring.

• Target: Cluster project, Namespace project

• Collection interval: 5 seconds

• Statistical data: 5 minutes

Tags

Tag	Type	Unit	Description
ppid	string	-	Parent process ID /proc/[pid]/status::PPid
pid	string	-	Process ID /proc/[pid]/status::Pid
cmd1	string	-	Command name /proc/[pid]/status::Name
cmd2	string	-	Command line (all commands and arguments) /proc/[pid]/cmdline
user	string	-	User ID or user name /proc/[pid]/status::Uid
onodeName	string	-	Node name of the process Container system's environment variable (NODE_IP)
createTime	TimeStamp	-	Process start time Field calculated through /proc/uptime

Fields

Field	Type	Unit	Description
cpu	float	percent(%)	CPU utilization - Field calculated through /proc/[pid]/stat
memory	float	percent(%)	Memory utilization - Field calculated through /proc/[pid]/statm
rss	long	byte	Actual memory usage (Resident Set Size) - VmRSS of /proc/[pid]/status
uid	string	-	User ID or name - Uid of /proc/[pid]/status
state	string	-	Process state - State of /proc/[pid]/status
sharedMemory	long	byte	Shared memory size - Field calculated through /proc/[pid]/statm
openFileDescriptors	integer	-	Number of file descriptors that the process has open - Field calculated through /proc/[pid]/fd
vmSize	long	byte	Virtual memory size - VmSize of /proc/[pid]/status
threads	integer	-	Number of threads created by processes - Threads of /proc/[pid]/status

Linux process status in the Kubernetes environment

On Linux, the State field in the /proc/[pid]/status file displays the current state of the process. The meanings of each status are as follows:

Code	Description
R (Running)	The process is running or ready to run.
S (Sleeping)	Interruptible sleep state, waiting for an event.
D (Disk Sleep)	Non-interruptible sleep state, waiting for an I/O operation.
R (Zombie)	The process has been terminated, but the parent process has not yet collected its termination status.
T (Stopped)	The process is stopped by a job control signal (such as SIGSTOP) or debugger.
t (Tracing stop)	Tracing stopped - State being traced by the debugger (indicated by a lowercase t)
X (Dead)	Dead state - The process is dead (commonly unseen)
x (Dead)	Dead state - The kernel thread is dead (commonly unseen)
K (WakeKill)	Forcibly terminated - It ignores any wakeup signal and is immediately dead.
W (Waking)	Waking up - State of being woken up after receiving a wake-up signal
I (Idle)	Kernel thread is idle (usually invisible to user space processes).

Note

Because Kubernetes manages the resources of containers and nodes efficiently, many processes running in containers are actually in waiting state. As a result, most processes may be in Sleeping state.

Agent status (agent_status_summary) metrics

This category collects metrics related to agent status every 10 seconds.

Fields

Field	Type	Unit	Description
inActTime	-	millisecond(ms)	Amount of time the agent remains inactive
isActive	Boolean	-	Whether the current agent is active or not (true / false)
isRestart	Boolean	-	Whether the agent was restarted (true / false)
lastActTime	-	millisecond(ms)	Last time when the agent was activated - 0: Disabled
oid	-	-	Unique IDs for each agent in the project
oType	-	-	Agent type - 1: Application agent - 2: See subType
startTime	-	millisecond(ms)	Timestamp indicating the time when the agent was started
subType	-	-	Agent type -9: Node agent - 10: Master agent

Ingress (kube_ingress) metrics

Note

Kubernetes agent 1.7.13 or later is required. For more information about agent updates, see the following.

It is collected when monitoring metadata and the related information for Ingress resources.

• Target: Cluster project, Namespace project

• Collection interval: 30 seconds

• Statistical data: 5 minutes

Tags

Tag	Type	Unit	Description
ingressUid	string	-	Unique ID of the Ingress resource
ingressName	string	-	Name of the Ingress resource
ingressNamespace	string	-	Namespace of the Ingress resource
creationTimeMillis	long	millisecond(ms)	Created time of the Ingress resource
ingressClassName	string	-	Name of the Ingress class
ingressLoadBalancerIps	List	-	IP of the Ingress load balancer

Fields

Field	Type	Unit	Description
host	List	-	Host name that the Ingress resource listens to (if *, it applies to all hosts)
path	List	-	Request path under a specific host
backendServiceName	List	-	Name of the service passed to the backend
backendServicePort	List	-	Port number passed to the backend
backendServiceUid	List	-	URL of the service passed to the backend
pathType	List	-	Path matching method (e.g. Prefix, Exact)

Kubernetes cron job (kube_cronjob) metrics

It collects metadata and status information for deployment resources by using the tags.

Tags

Tag	Type	Unit	Description
cronJobName	string	-	CronJob name (metadata.name)
namespace	string	-	CronJob namespace (metadata.namespace)
cronJobUid	string	-	CronJob UID (metadata.uid)

Fields

Field	Type	Unit	Description
lastScheduleTime	string	-	Last time when the CronJob was scheduled (status.lastScheduleTime)
lastSuccessfulTime	string	-	Last time when the CronJob succeeded (status.lastSuccessfulTime)
cronJobSpecSchedule	string	-	Scheduling cycle of the CronJob (spec.schedule, e.g.: "0 */5 * * *" )
successfulJobsHistoryLimit	integer	count	Maximum number of successful jobs to keep (spec.successfulJobsHistoryLimit)
failedJobsHistoryLimit	integer	count	Maximum number of failed jobs to keep (spec.failedJobsHistoryLimit)
concurrencyPolicy	string	-	Concurrency policy (Allow, Forbid, Replace) (spec.concurrencyPolicy)
startingDeadlineSeconds	integer	second	Maximum delay time when the job can be run after its scheduled time (spec.startingDeadlineSeconds)

Kubernetes deployment (kube_deployment) metrics

It collects metadata and status information for deployment resources by using the tags.

Tags

Tag	Type	Unit	Description
deployName	string	-	Deployment name (metadata.name)
namespace	string	-	Deployment namespace (metadata.namespace)
deployUid	string	-	Deployment UID (metadata.uid )

Fields

Field	Type	Unit	Description
deployReadyReplicas	integer	count	Number of Pods that are in ready status (status.readyReplicas)
deployTotalReplicas	integer	count	Total number of set Pods (spec.replicas)
deployUpdatedReplicas	integer	count	Number of Pods updated to the latest version (status.updatedReplicas)
deployAvailableReplicas	integer	count	Number of Pods in available state (status.availableReplicas)
deployUnavailableReplicas	integer	count	Number of Pods in unavailable state (status.unavailableReplicas)
deployCreationTime	integer	millisecond(ms)	Deployment creation time (Unix Epoch milliseconds, metadata.creationTimestamp)
deploySelector	string	-	Selector label of the deployment (spec.selector.matchLabels)
deployStrategy	string	-	Deployment strategy (spec.strategy.type, e.g. RollingUpdate, Recreate)

Kubernetes daemonset (kube_daemonset) metrics

It collects metadata and status information for daemonset resources by using the tags.

Tags

Tag	Type	Unit	Description
daemonSetName	string	-	DaemonSet name (metadata.name)
namespace	string	-	DaemonSet namespace (metadata.namespace)
daemonSetUid	string	-	DaemonSet UID (metadata.uid)
creationTime	integer	millisecond(ms)	DaemonSet creation time (metadata.creationTimestamp)
age	string	-	Elapsed time since DaemonSet creation

Fields

Field	Type	Unit	Description
currentNumberScheduled	integer	count	Number of scheduled Pods (status.currentNumberScheduled)
desiredNumberScheduled	integer	count	Number of desired Pods (status.desiredNumberScheduled)
numberReady	integer	count	Number of Pods in ready state (status.numberReady)
numberAvailable	integer	count	Number of available Pods (status.numberAvailable)
numberMisscheduled	integer	count	Number of wrongly scheduled Pods (status.numberMisscheduled)
updatedNumberScheduled	integer	count	Number of updated Pods (status.updatedNumberScheduled)
daemonSetSelector	string	-	Selector label (spec.selector.matchLabels)

Kubernetes replica set (kube_replicaset) metrics

It collects metadata and status information for ReplicaSet resources with the tags.

Tags

Tag	Type	Unit	Description
replicaSetName	string	-	ReplicaSet name (metadata.name)
namespace	string	-	ReplicaSet namespace (metadata.namespace)
replicaSetUid	string	-	ReplicaSet UID (metadata.uid)
ownerKind	string	-	Type of the upper layer object (e.g. Deployment) (metadata.ownerReferences.kind)
ownerName	string	-	Name of the upper layer object (metadata.ownerReferences.name)
ownerUid	string	-	UID of the upper layer object (metadata.ownerReferences.uid)
creationTime	integer	millisecond(ms)	ReplicaSet creation time (metadata.creationTimestamp)
age	string	-	Elapsed time since creation of ReplicaSet

Fields

Field	Type	Unit	Description
replicaSetReplicas	integer	count	Number of set Pods (spec.replicas)
replicaSetReadyReplicas	integer	count	Number of Pods in ready state (status.readyReplicas)
replicaSetAvailableReplicas	integer	count	Number of available Pods (status.availableReplicas)
replicaSetSelector	string	-	Selector label (spec.selector.matchLabels)

Kubernetes stateful set (kube_statefulSet) metrics

It collects metadata and status information for Stateful resources with the tags.

Tags

Tag	Type	Unit	Description
statefulSetName	string	-	Name of StatefulSet (metadata.name)
namespace	string	-	Namespace to which the StatefulSet belongs (metadata.namespace)
statefulSetUid	string	-	UIO of StatefulSet (metadata.uid)
creationTime	integer	millisecond(ms)	StatefulSet creation time (metadata.creationTimestamp)
age	string	-	Elapsed time since StatefulSet creation

Fields

Field	Type	Unit	Description
replicas	integer	count	Number of set Pods (spec.replicas)
readyReplicas	integer	count	Number of Pods in ready state (status.readyReplicas)
currentReplicas	integer	count	Number of created Pods (status.currentReplicas)
updatedReplicas	integer	count	Number of updated Pods (status.updatedReplicas)
statefulSetSelector	string	-	Selector label (spec.selector.matchLabels)

Kubernetes job (kube_job) metrics

It collects metadata and status information for deployment resources by using the tags.

Tags

Tag	Type	Unit	Description
jobName	string	-	Job name (metadata.name)
namespace	string	-	Namespace of the Job (metadata.namespace)
jobUid	string	-	Job UID (metadata.uid)
ownerKind	string	-	Parent object kind (e.g. CronJob)
ownerName	string	-	Parent object name
ownerUid	string	-	Parent object UID

Fields

Field	Type	Unit	Description
succeededPods	integer	count	Number of successful jobs (status.succeeded)
failedPods	integer	count	Number of failed Pods (status.failed)
completionTime	string	-	Job completion time (status.completionTime.toString())
parallelism	integer	count	Number of Pods set to run concurrently (spec.parallelism)
completions	integer	count	Number of Pod executions required for a job to be completed (spec.completions)
backoffLimit	integer	count	Maximum number of possible retries upon failure (spec.backoffLimit)
activeDeadlineSeconds	integer	second(sec)	Maximum time for a job to run (spec.activeDeadlineSeconds)

Kubernetes NetworkPolicy (kube_network_policy) metrics

The kube_network_policy category collects the NetworkPolicy objects of the cluster on an event basis. Instead of periodic collection, a snapshot of that point in time is sent whenever an object is added, changed, or deleted.

• Target: Cluster project, namespace project
• Collection method: Event based

Tags

Tag	Type	Unit	Description
networkPolicyUid	string	-	NetworkPolicy UID (unique value)
networkPolicyName	string	-	NetworkPolicy name
namespace	string	-	Namespace the NetworkPolicy belongs to
resourceVersion	string	-	Object revision (metadata.resourceVersion)

Fields

Field	Type	Unit	Description
generation	long	-	Object generation (metadata.generation)
creationTimeMillis	long	-	Object creation time (epoch ms)
policyTypes	list	-	List of policy types - Ingress, Egress. An empty list means Ingress, the API default
selectedPodUids	list	-	List of Pod UIDs selected by the policy
podSelector	map	-	Pod selector of the policy target (matchLabels, matchExpressions)
ingressRules	list	-	List of Ingress rules - Includes from and ports per rule
egressRules	list	-	List of Egress rules - Includes to and ports per rule
specSchemaVersion	integer	-	Schema version of the spec fields
specHash	long	-	Hash of the normalized spec - Used to determine whether the content changed

The rules are sent in a nested structure to preserve the meaning of the Kubernetes API as it is.

• namespaceSelector and podSelector within one peer are in an AND relationship.
• The peer list and the rule list are in an OR relationship.
• An empty from or to in a rule means all peers, and empty ports means all ports.
• The ipBlock of a peer contains cidr and except.
• A port contains protocol, portType, portNumber or portName, and endPort.

The networkpolicies view permission is required in the ClusterRole. You can turn the collection off with the collect_network_policy_enabled option.