VMware collection items

The server monitoring agent collects the status of ESXi hosts and virtual machines together through the VMware API. The resource usage seen from inside a virtual machine alone makes it difficult to identify the cause of performance degradation. A broken fan on the host, or virtual machines packed onto one host so that vCPUs wait for each other's turn, hardly shows up in the guest metrics. This document describes the items collected through the VMware integration, for operators who run virtualization-based infrastructure and need to trace causes down to the host level.

The VMware data integration can collect data by connecting directly to an ESXi host, even without vCenter. However, the vm_vcenter_inventory category is collected only when the agent is connected to vCenter.

The supported agent versions are Linux and Windows 2.9.23 or later. Fields marked New in 2.9.23 in the Remarks column of each table were added in 2.9.23. Fields without a mark have been provided since earlier versions.

infra_vm_vhost_hw

A category that collects the hardware status of an ESXi host by part. One host becomes one row, and it provides the status per part, such as power, cooling, memory, and processor, along with the highest temperature. Use it to check hardware failures before they lead to performance degradation or an outage. This category was added in 2.9.23, so every field in the table is new.

• Collection target: VMware ESXi

Status values are standardized into the following four. When a part has multiple sensors, the most severe status is displayed as the representative value, and a part with no sensor data is displayed as unknown.

Item	Description
green	Normal
yellow	Warning
red	Failure
unknown	The status cannot be determined

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each hardware status came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
hardwareHealth	string	-	Overall status representing the most severe state among all collectable hardware statuses
powerSupplyStatus	string	-	Status of the power supply units
powerSupplyRedundancyStatus	string	-	Redundancy status of the power supply units. Used to check whether redundancy is degraded by a partial failure
networkDeviceStatus	string	-	Hardware status of the physical network devices and network adapters. Distinct from the network link up and down state
batteryStatus	string	-	Status of the internal battery or the hardware management controller battery. Includes CMOS batteries and RAID cache batteries
processorsStatus	string	-	Hardware status of the physical processors and CPU sockets. A physical hardware status unrelated to CPU usage
memoryModulesStatus	string	-	Hardware status of the physical memory modules. Includes DIMM errors and module failure or warning states
coolingUnitStatus	string	-	Status of the cooling units such as fans and blowers
coolingUnitRedundancyStatus	string	-	Redundancy status of the cooling units. Used to check whether cooling is maintained even when some fans fail
temperatureStatus	string	-	Overall status of the temperature sensors such as the CPU, system board, memory, and air intake
temperatureMax	-	°C	Highest temperature among the sensor readings at collection time. Calculated by applying the unit modifier provided by VMware to the raw sensor reading. Empty when no temperature sensor is available
hardwareHealthDataAvailable	boolean	-	Whether the host reported at least one hardware status or sensor data
hardwareHealthSupportStatus	string	-	Whether hardware status collection is supported. supported means sensor data is provided, unsupported means the API is not supported or the sensors are not exposed
hardwareHealthUnavailableReason	string	-	Reason why hardware status data cannot be collected. Records causes such as an unsupported Hardware Health API, unexposed sensors, or a collection error. Empty on normal collection
temperatureMaxSensor	string	-	Name of the temperature sensor where temperatureMax was measured. Identifies which of the CPU, system board, memory, or air intake sensors actually recorded the highest temperature
temperatureUnit	string	-	Unit applied to temperatureMax. Usually in the Degrees C form, and the expression can vary depending on the sensor provider

infra_vm_vhost_resource_capacity

A category that collects the physical resources of an ESXi host together with the resources allocated to virtual machines. One host becomes one row, and it provides the ratio of allocation to physical capacity (overcommit). Use it to check whether too many virtual machines are placed on a specific host. This category was added in 2.9.23, so every field in the table is new.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each resource metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
cpuCores	-	count	Total number of physical CPU cores on the host. The basis for allocatable virtual machine resources and CPU capacity planning
totalMemory	-	byte	Total memory capacity installed on the host. The basis for the capacity available for virtual machine placement and memory resource planning
activeVCpu	-	count	Sum of the vCPUs allocated to powered-on virtual machines
allocatedVCpu	-	count	Sum of the vCPUs configured for all virtual machines placed on the host. Includes powered-off and suspended virtual machines
activeOverCommitCore	-	-	activeVCpu divided by the number of physical CPU cores. Expressed as a multiple showing how many vCPUs are allocated per physical core on average, based on running virtual machines
allocatedOverCommitCore	-	-	allocatedVCpu divided by the number of physical CPU cores. The vCPU overcommit multiple on a configuration basis, including powered-off and suspended virtual machines
activeMemory	-	byte	Sum of the memory configured for powered-on virtual machines
allocatedMemory	-	byte	Sum of the memory configured for all virtual machines placed on the host. Includes powered-off and suspended virtual machines
activeOverCommitMemory	-	-	activeMemory divided by the physical memory capacity. Expressed as a multiple showing how much memory is allocated relative to the physical memory, based on running virtual machines
allocatedOverCommitMemory	-	-	allocatedMemory divided by the physical memory capacity. The memory overcommit multiple on a configuration basis, including powered-off and suspended virtual machines
vmGuestCount	-	count	Number of virtual machine objects connected to the host. Used to identify the placement and density per host
poweredOffVMCount	-	count	Number of powered-off virtual machines. Included in allocatedVCpu but not in activeVCpu
suspendedVMCount	-	count	Number of suspended virtual machines. Because vCPUs are configured, they are included in allocatedVCpu, but they are not included in activeVCpu since they are not running
unknownPowerStateVMCount	-	count	Number of virtual machines whose power state cannot be determined or is not identified as one of poweredOn, poweredOff, or suspended. Can occur due to vCenter response delays, permissions, or a temporary state mismatch
vcpuSkippedVMCount	-	count	Number of virtual machines excluded from the vCPU sum because the vCPU setting could not be read. Can be caused by a failure to access the configuration, insufficient permissions, an abnormal configuration, or a temporary vCenter response error
memorySkippedVMCount	-	count	Number of virtual machines excluded from the memory sum because the memory setting could not be read. The causes are the same as for vcpuSkippedVMCount

The overcommit ratio is displayed to two decimal places. A value greater than 1 means more resources are allocated to virtual machines than the physical resources. Since a certain level of overcommit is common in virtualized environments, it is better to look at the deviation between hosts in the same cluster and the increasing trend over time rather than the absolute value.

Adjust whether to collect with the following option.

whatap.conf

vm.vhost.perf.resource_capacity.enabled=true    # Whether to collect (Default: true)

infra_vm_vhost_cpu

A category that aggregates the CPU usage and wait time of an ESXi host. One host becomes one row, and it provides the CPU usage and contention of all virtual machines on the host.

• Collection target: VMware ESXi

Field	Type	Unit	Description	Remarks
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each CPU metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
usage	-	percent	Ratio of the host's total CPU resources actually used. The basic metric for checking the CPU load level of the host
usageMhz	-	MHz	Amount of CPU resources actually in use on the host. Check it together with usage when comparing hosts with different CPU capacities
totalCapacity	-	MHz	Total CPU capacity the host can provide. The reference value for interpreting usage, demand, and reserved capacity
demand	-	MHz	Amount of CPU resources the virtual machines of the host actually demanded
readiness	-	percent	Ratio of the time a virtual machine was ready to run but was not immediately given a physical CPU. The key metric for judging CPU contention on the host
ready	-	ms/s	Time spent waiting for a physical CPU while ready to run
latency	-	percent	Level of delay between requesting CPU resources and actually running
costop	-	ms/s	Time a multi-vCPU virtual machine was stopped while waiting to be scheduled simultaneously
wait	-	ms/s	Time the CPU spent waiting without performing actual computation. Includes idle time, I/O waits, and swap waits
swapwait	-	ms/s	Time CPU execution was delayed by memory swap processing
idle	-	ms/s	Time the CPU spent idle
reservedCapacity	-	MHz	CPU capacity secured by reservations. When it is large relative to the total capacity, the actually available CPU headroom decreases
demandPct	-	percent	demand divided by totalCapacity	New in 2.9.23
costopPct	-	percent	costop converted to a ratio (1,000 ms/s = 100%)	New in 2.9.23

Do not judge CPU contention from a single metric. When usage is high and readiness and ready are high together, CPU contention is likely. Conversely, when usage is low but readiness is high, also check for over-allocated vCPUs, resource limits, and the scheduling policy. When demand stays higher than usageMhz, the virtual machines may not be getting as much CPU as they demand.

wait includes idle time, I/O waits, and swap waits, so a high value does not by itself mean the CPU is insufficient. Check it separately against idle and swapwait. swapwait is displayed as a CPU metric, but the cause can be insufficient memory or memory contention, so when it increases, also check the memory usage, ballooning, and swap activity of the host and the virtual machines.

Metrics in ms/s are converted to per-second values so that they can be compared on the same basis even when the collection interval changes.

demandPct and costopPct are the sum over all virtual machine vCPUs on the host, so they can exceed 100%, and no separate upper limit is applied. If costopPct stays high, the number of vCPUs allocated to the virtual machines may be too large for the host capacity.

infra_vm_vhost_mem

A category that aggregates the memory usage and reclamation activity of an ESXi host. One host becomes one row, and it provides everything from the actual occupancy and working set to reclamation metrics such as ballooning, compression, and swapping.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each memory metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
consumed	-	byte	Physical memory the host is actually consuming. Includes the memory occupied by virtual machines, VMkernel, and the ESXi system area
active	-	byte	Memory the virtual machines on the host have actively used recently. Close to the effective working set
activeConsumedRatio	-	percent	active divided by consumed. The share of the consumed memory that has actually been actively used
granted	-	byte	Memory the host granted to the virtual machines
latency	-	percent	Level of delay occurring during memory access or allocation
vmmemctl	-	byte	Memory the balloon driver reclaimed from the virtual machines
swapused	-	byte	Memory that was pushed out to swap on the host
swapInRate	-	bytes/s	Rate at which memory is read back from the swap area
swapOutRate	-	bytes/s	Rate at which memory pages are pushed out to the swap area
compressed	-	byte	Memory compressed through memory compression
compressionRate	-	bytes/s	Rate at which memory compression occurs
shared	-	byte	Memory shared between virtual machines or between memory pages
overhead	-	byte	Memory overhead ESXi uses additionally to run the virtual machines
sysUsage	-	byte	Memory in use by the ESXi system area
heapFree	-	byte	Free memory available in the ESXi heap area

When consumed is high but active is low, many virtual machines have a large allocation and occupancy but little actual usage. activeConsumedRatio shows this relationship directly.

Do not judge a memory shortage from a single metric. As memory pressure grows, ESXi escalates reclamation through ballooning, then compression, then swapping. When compressed and compressionRate rise after vmmemctl has increased, and swapused and swapOutRate also appear, memory contention has progressed considerably. When latency is high at the same time, virtual machine performance is likely affected.

A high swapInRate means the pages a virtual machine needs are in swap and are being read back, which can lead to performance delays.

shared, overhead, sysUsage, and heapFree do not indicate a problem on their own. shared is a supplementary metric for the memory savings effect, and overhead grows with the number of virtual machines and the complexity of their virtual device configurations. If sysUsage or heapFree changes abnormally, check the ESXi system state separately.

infra_vm_vhost_disk

A category that aggregates the disk I/O of an ESXi host. One host becomes one row, and it provides the throughput summed across all disks connected to the host along with the highest latency.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each disk metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
usage	-	bytes/s	Total disk I/O throughput on the host. The sum of reads and writes
read	-	bytes/s	Disk read throughput on the host
write	-	bytes/s	Disk write throughput on the host
maxTotalLatency	-	ms	Highest disk I/O latency among the disks and devices connected to the host

When usage stays high, a large amount of disk I/O is occurring on the virtual machines of that host. Check it together with maxTotalLatency to distinguish a simple throughput increase from a storage bottleneck.

When read is high and the latency increases together, check for read I/O contention, cache misses, and backend storage response delays. When write is high and the latency increases, check for write I/O contention, storage controller delays, and RAID, cache, or synchronization delays.

maxTotalLatency rises when a delay occurs on a specific disk, datastore, LUN, or storage path. Even when the overall throughput of the host is not high, a high value is worth checking for I/O delays on a particular storage path or virtual machine.

infra_vm_vhost_disk_perf

Performance per disk device connected to an ESXi host. One disk becomes one row, and it provides the throughput, the request counts, and the latency per layer. Use it to narrow down which disk is the bottleneck.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each disk metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
diskName	string	-	Name of the disk or device the metrics were collected from. The basis for distinguishing individual disks, LUNs, local disks, and storage devices connected to the host
usage	-	bytes/s	Total I/O throughput of the disk. The sum of read and write
read	-	bytes/s	Read throughput of the disk
write	-	bytes/s	Write throughput of the disk
numberTotal	-	counts/s	Total number of I/O requests. The sum of numberRead and numberWrite
numberRead	-	counts/s	Number of read I/O requests
numberWrite	-	counts/s	Number of write I/O requests
totalLatency	-	ms	Average latency for processing all I/O requests on the disk. The representative latency metric, including the effects of kernel, device, and queue latency
totalReadLatency	-	ms	Average latency for processing read I/O requests
totalWriteLatency	-	ms	Average latency for processing write I/O requests
deviceLatency	-	ms	Average latency for processing I/O on the physical disk or backend storage device
queueLatency	-	ms	Average time an I/O request waited in the queue before being processed
kernelLatency	-	ms	Average latency incurred while the ESXi VMkernel layer processed the I/O
maxQueueDepth	-	count	Maximum queue depth available or observed on the disk
busResets	-	counts/s	Number of bus resets that occurred on the disk or storage bus

Looking at the throughput (usage, read, write) together with the request counts (numberTotal, numberRead, numberWrite) gives a sense of the I/O block size. When the request count is high but the throughput is low, there are many small-block I/Os.

Splitting the latency metrics narrows down where the delay originates.

High metric	Where to look
deviceLatency	The backend storage layer such as the storage device, LUN, array, SAN, or NAS
kernelLatency	Host-internal I/O processing, the driver, the HBA or storage adapter, and the VMkernel storage stack
queueLatency	Requests are queuing because they are not processed fast enough. Check it with maxQueueDepth and the request counts

When busResets keeps occurring with a non-zero value, also check the storage path stability, the HBA and controller logs, and the ESXi events.

The values summed across the whole host are available in infra_vm_vhost_disk.

infra_vm_vhost_storageadapter_perf

Performance per storage adapter (HBA) of an ESXi host. One adapter becomes one row, and it provides the summed performance of the disks and LUNs connected to that adapter. Use it to see which HBA carries the load.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each adapter metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
memberCount	-	count	Number of canonical disk or LUN instances included in the adapter aggregation. A larger value means the performance of more disks and LUNs is summed into one adapter
memberPathNames	string	-	List of the disk and LUN paths mapped to the adapter. Used to check which paths the adapter values were aggregated from
adapterName	string	-	Name of the storage adapter the values were aggregated for. Usually an HBA or storage controller name such as vmhba0
total	-	bytes/s	Total I/O throughput of the adapter. The sum of read and write
read	-	bytes/s	Data throughput read through the adapter. The sum of the read throughput of the underlying disks and LUNs
write	-	bytes/s	Data throughput written through the adapter. The sum of the write throughput of the underlying disks and LUNs
numberTotalAveraged	-	counts/s	Total average number of I/Os on the adapter. The sum of numberReadAveraged and numberWriteAveraged
numberReadAveraged	-	counts/s	Average number of read I/Os on the adapter
numberWriteAveraged	-	counts/s	Average number of write I/Os on the adapter
commandsAveraged	-	counts/s	Total number of I/O commands issued on the disks and LUNs connected to the adapter
totalReadLatency	-	ms	Average read I/O latency of the adapter. The read latency of the underlying disks and LUNs, weighted by the read request count
totalWriteLatency	-	ms	Average write I/O latency of the adapter. The write latency of the underlying disks and LUNs, weighted by the write request count

The values in this category are aggregated from the disks and LUNs under the adapter. Throughput and request counts are simple sums, while latency is an average weighted by the request count. A large memberCount means that many disks and LUNs are folded into one value, so a single slow disk may not stand out in the adapter average.

When the adapter latency looks high, check memberPathNames to see which paths were aggregated, then narrow down per path in infra_vm_vhost_storagepath_perf and per disk in infra_vm_vhost_disk_perf.

infra_vm_vhost_storagepath_perf

Performance per storage path of an ESXi host. One path becomes one row, and it provides the per-path throughput and latency along with the adapter, target, and LUN information. Use it to find a single slow path in a multipath configuration.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each path metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
pathName	string	-	Display name of the storage path. A path name combining the adapter, target, LUN, and canonical disk information
canonicalDiskName	string	-	Canonical name of the SCSI LUN as recognized by the host (for example, naa.*, t10.*, mpx.*). The basis for mapping to the performance counter instance
adapterName	string	-	Name of the storage adapter the path is connected to. Usually an HBA or storage controller name such as vmhba0
targetName	string	-	Name combining the adapter and the target (for example, vmhba0:T0). Used to identify the target path under an adapter
targetId	string	-	Target number the path is connected to in the ESXi SCSI topology. Used to distinguish multiple targets on the same adapter
lunId	string	-	LUN number the path is connected to
adapterKey	string	-	Key that identifies the storage adapter inside vSphere. Used for API mapping and long-term data consistency checks
total	-	bytes/s	Total I/O throughput of the path. The sum of read and write
read	-	bytes/s	Data throughput read through the path
write	-	bytes/s	Data throughput written through the path
numberTotalAveraged	-	counts/s	Total average number of I/Os on the path. The sum of numberReadAveraged and numberWriteAveraged
numberReadAveraged	-	counts/s	Average number of read I/Os on the path
numberWriteAveraged	-	counts/s	Average number of write I/Os on the path
commandsAveraged	-	counts/s	Total number of I/O commands issued on the path. The storage command load including reads and writes
totalReadLatency	-	ms	Average read I/O latency of the path
totalWriteLatency	-	ms	Average write I/O latency of the path

There are several path identification fields because each narrows the problem at a different level. adapterName is per HBA, targetId is per target within the same adapter, and lunId is per LUN. When one LUN has multiple paths, comparing which path alone is slow separates a path failure from a storage device problem.

When totalReadLatency or totalWriteLatency stays high, check the HBA, target, LUN, and SAN segment of that path. If it is high across every path using the same canonicalDiskName, the problem is more likely on the backend storage than on the path.

infra_vm_vhost_datastore_perf

Performance and capacity per datastore accessed by an ESXi host. One datastore becomes one row, and it provides the capacity usage along with the I/O throughput and latency.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each datastore metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
datastoreName	string	-	Name of the datastore the performance and capacity data was collected from. The display name vSphere recognizes, such as VMFS, NFS, or vSAN
datastoreInstanceId	string	-	Datastore identifier mapped to the performance counter instance. A value such as the VMFS volume UUID extracted from the datastore URL, used to link data reliably where names can change or be duplicated
datastoreType	string	-	Type of the datastore (for example, VMFS, NFS, vSAN). Used to distinguish the storage backend
accessible	boolean	-	Whether the datastore is accessible from the host
datastoreUrl	string	-	Internal vSphere URL of the datastore. Includes the VMFS volume UUID or the access path
total	-	bytes/s	Total I/O throughput of the datastore. The sum of read and write
read	-	bytes/s	Data throughput read from the datastore
write	-	bytes/s	Data throughput written to the datastore
numberTotalAveraged	-	counts/s	Total average number of I/Os on the datastore. The sum of numberReadAveraged and numberWriteAveraged
numberReadAveraged	-	counts/s	Average number of read I/Os on the datastore
numberWriteAveraged	-	counts/s	Average number of write I/Os on the datastore
totalReadLatency	-	ms	Average read I/O latency of the datastore
totalWriteLatency	-	ms	Average write I/O latency of the datastore
capacity	-	byte	Total capacity of the datastore
allocated	-	byte	Allocated space. The sum of used and uncommitted
used	-	byte	Actual usage. The total capacity minus the free space
uncommitted	-	byte	Space that is committed but not yet actually used in a thin provisioning environment, and can become actual usage later
free	-	byte	Free space of the datastore
usedPct	-	percent	used divided by capacity

The capacity fields relate as follows.

• capacity = used + free
• allocated = used + uncommitted

When allocated exceeds capacity, thin provisioning has promised more space than physically exists, an overcommit. Since uncommitted turns into actual usage as virtual machine disks grow, check the relationship between allocated and capacity rather than looking at usedPct alone.

When accessible is false, check the connection between the host and the datastore, the storage path, the SAN or NAS network, the permissions, and whether the datastore has failed.

When the latency stays high, check for storage response delays, path problems, backend storage load, and storage network delays. When several hosts share the same datastore, comparing whether the other hosts see the same latency separates a host problem from a backend problem.

infra_vm_vhost_network

A category that aggregates the network usage of an ESXi host. One host becomes one row, and it provides the sum across all physical network interfaces of the host.

• Collection target: VMware ESXi

Field	Type	Unit	Description	Remarks
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each network metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
usage	-	bytes/s	Total network usage of the host. The sum of received and transmitted traffic
bytesRx	-	bytes/s	Data throughput the host received over the network. Includes virtual machines, VMkernel, and the management network
bytesTx	-	bytes/s	Data throughput the host transmitted over the network. Includes virtual machines, VMkernel, and the management network
packetsTotal	-	counts/s	Sum of received packets (packetsRx) and transmitted packets (packetsTx)
packetsRx	-	counts/s	Number of network packets the host received
packetsTx	-	counts/s	Number of network packets the host transmitted
errorsRx	-	counts/s	Number of errors that occurred during reception
errorsTx	-	counts/s	Number of errors that occurred during transmission
droppedRx	-	counts/s	Number of packets dropped during reception
droppedTx	-	counts/s	Number of packets dropped during transmission
errorsRxTx	-	counts/s	Sum of receive errors (errorsRx) and transmit errors (errorsTx)	New in 2.9.23
droppedRxTx	-	counts/s	Sum of receive drops (droppedRx) and transmit drops (droppedTx)	New in 2.9.23

When errorsRx keeps increasing, suspect problems with the physical NIC, cable, switch port, driver, or MTU settings. When errorsTx increases, check the physical NIC, switch port, link status, driver, and network congestion.

droppedRx can be caused by an insufficient receive queue, network flooding, a NIC or driver problem, or vSwitch processing delays. Use droppedTx to check for an insufficient transmit queue, network congestion, or processing delays on the physical NIC or vSwitch.

Use errorsRxTx and droppedRxTx when you want to check only the host-level totals without separating receive and transmit.

infra_vm_vhost_nic_perf

Usage per physical network interface of an ESXi host. One interface becomes one row. Use it to check which interface produces errors or drops.

• Collection target: VMware ESXi

Field	Type	Unit	Description	Remarks
vhostName	string	-	Name of the host being collected. The basis for identifying which ESXi host each NIC metric came from
uuid	string	-	Hardware UUID of the host. Supplementary basis for identifying the same host even when its name or IP changes
nicName	string	-	Name of the network instance the metrics were collected from. Usually a physical NIC name such as vmnic0 or vmnic1, though another name can appear depending on the environment
usage	-	bytes/s	Total network throughput of the NIC. The sum of reception and transmission
bytesRx	-	bytes/s	Data throughput received through the NIC
bytesTx	-	bytes/s	Data throughput transmitted through the NIC
packetsTotal	-	counts/s	Sum of received packets (packetsRx) and transmitted packets (packetsTx)
packetsRx	-	counts/s	Number of network packets the NIC received
packetsTx	-	counts/s	Number of network packets the NIC transmitted
errorsRx	-	counts/s	Number of errors that occurred during reception
errorsTx	-	counts/s	Number of errors that occurred during transmission
droppedRx	-	counts/s	Number of packets dropped during reception
droppedTx	-	counts/s	Number of packets dropped during transmission
errorsRxTx	-	counts/s	Sum of receive errors (errorsRx) and transmit errors (errorsTx)	New in 2.9.23
droppedRxTx	-	counts/s	Sum of receive drops (droppedRx) and transmit drops (droppedTx)	New in 2.9.23

When errors or drops increase on only one NIC, first check that NIC's cable, switch port, driver, and MTU settings. The totals summed across the whole host are available in infra_vm_vhost_network.

Use errorsRxTx and droppedRxTx when you want to check only the per-interface totals without separating receive and transmit.

infra_vm_vhost_vmguest_perf

Performance metrics of the virtual machines running on an ESXi host. One powered-on virtual machine becomes one row, and it provides the CPU, memory, disk, and network usage of the virtual machine as seen from the host.

• Collection target: VMware ESXi

Field	Type	Unit	Description	Remarks
vHostName	string	-	Name of the host on which the virtual machine is running
uuid	string	-	Hardware UUID of the ESXi host on which the virtual machine is running. Supplementary basis for identifying the same host even when its name or IP changes
vmName	string	-	Name of the virtual machine the performance data was collected from. The name shown in the vCenter inventory
vmInstanceUuid	string	-	UUID that identifies the virtual machine instance within vCenter. The main identifier for tracking the same virtual machine regardless of a name change or vMotion
cpuUsage	-	percent	CPU usage of the virtual machine. The vSphere cpu.usage.average value converted to a percentage
cpuUsagemhz	-	MHz	CPU resources the virtual machine is actually using. Check it with the usage rate to see the actual consumption
cpuCostop	-	ms	Time the vCPUs of an SMP virtual machine waited because they could not be scheduled simultaneously
cpuCostopPct	-	percent	cpuCostop divided by the sampling interval and converted to a ratio. Matches %CSTP as shown per VM group in esxtop	New in 2.9.23
cpuCostopPctPerVcpu	-	percent	cpuCostopPct divided by the number of vCPUs, giving the average ratio per vCPU. Always between 0 and 100	New in 2.9.23
cpuReadiness	-	ms	Time the virtual machine waited without being scheduled after it was ready to use the CPU, because of insufficient physical CPU resources
cpuReadinessPct	-	percent	cpuReadiness divided by the sampling interval and converted to a ratio. Matches %RDY as shown per VM group in esxtop	New in 2.9.23
cpuReadinessPerVcpu	-	percent	Average waiting ratio per vCPU. Provided directly by the vSphere cpu.readiness.average counter and always between 0 and 100	New in 2.9.23
cpuMaxlimited	-	percent	Ratio of the CPU time that was available but restricted by a CPU limit or a similar setting
memConsumed	-	MB	Memory the virtual machine actually consumed from the host memory
memVmmemctl	-	KB	Virtual machine memory reclaimed by the balloon driver (vmmemctl)
memActive	-	KB	Memory judged to have been actively used recently. Used to estimate the effective working set
memGranted	-	KB	Memory the host granted to the virtual machine
memSwapped	-	KB	Amount of the virtual machine memory the host swapped out
memSwapoutRate	-	KB/s	Rate at which the virtual machine memory is swapped out to the host swap
memCompressed	-	KB	Amount of the virtual machine memory the host is keeping compressed
memCompressionRate	-	KB/s	Rate at which the virtual machine memory is compressed
diskUsage	-	KB/s	Total disk usage from the guest perspective. The sum of reads and writes
networkUsage	-	KB/s	Total network usage from the guest perspective. The sum of reception and transmission

cpuCostopPct and cpuReadinessPct grow with the number of vCPUs on a multi-vCPU virtual machine, so they can exceed 100%. When judging thresholds, take the vCPU count into account, or use cpuCostopPctPerVcpu and cpuReadinessPerVcpu, which are converted to a per-vCPU basis. The per-vCPU values are always between 0 and 100, so virtual machines with different vCPU counts can be compared on the same basis.

The judgment criteria in the official VMware guide are as follows.

Metric	Normal	Caution	Critical
cpuCostopPctPerVcpu	3% or less	3–10%	10% or more
cpuReadinessPerVcpu	5% or less	5–10%	10% or more

When cpuCostopPctPerVcpu is at the critical level, consider reducing the vCPUs allocated to the virtual machine or moving it to another host. When cpuReadinessPerVcpu is at the critical level, consider insufficient host CPU resources or relocating the virtual machine.

When memVmmemctl, memSwapped, and memCompressed increase, suspect memory pressure on the host. Compression usually occurs before swapping, so when memSwapped also increases after memCompressed has risen, the memory shortage is worsening.

Counters that vCenter does not provide are collected as empty values.

vm_vcenter_inventory

Configuration information of the vCenter server itself. It provides the number of managed objects, the number of active alarms, and the certificate and license expiration information. Use it to check items that need advance preparation, such as vCenter certificate expiration. This category was added in 2.9.23, so every field in the table is new.

• Collection target: VMware vCenter

Field	Type	Unit	Description
vCenterHostname	string	-	Host name of the vCenter server. The representative name that identifies the vCenter
fqdn	string	-	FQDN of the vCenter server. Collected only when it is registered in DNS
ip	string	-	Representative IP address of the vCenter server
allIPs	string	-	List of all IP addresses that can reach the vCenter server
connectionAddress	string	-	Address (IP or host name) the agent used to connect to the vCenter API
uuid	string	-	Unique identifier of the vCenter server. Identifies the same instance regardless of a server migration or a name change
overallHealth	string	-	Overall status of the entire vCenter inventory. Displays the highest severity among green, yellow, and red
redAlarmCount	string	count	Number of currently active critical (red) alarms
yellowAlarmCount	string	count	Number of currently active warning (yellow) alarms
alarmCount	string	count	Number of all currently active alarms
hostCount	string	count	Number of ESXi hosts managed by vCenter
vmCount	string	count	Number of virtual machines managed by vCenter
datacenterCount	string	count	Number of registered data centers
clusterCount	string	count	Number of registered clusters
datastoreCount	string	count	Number of registered datastores
networkCount	string	count	Number of registered networks
resourcePoolCount	string	count	Number of registered resource pools
folderCount	string	count	Number of folders in the vCenter inventory
certificateDaysRemaining	string	count	Days remaining until the vCenter HTTPS certificate expires
certificateExpiration	string	-	Expiration date and time of the vCenter HTTPS certificate
certificateValidFrom	string	-	Date and time from which the vCenter HTTPS certificate is valid
certificateSubject	string	-	Subject (DN) information of the vCenter HTTPS server certificate
certificateIssuer	string	-	Issuer (CA) information that issued the vCenter HTTPS server certificate
certificateThumbprint	string	-	SHA-256 thumbprint of the vCenter HTTPS certificate
licenseExpiration	string	-	Expiration date and time of the vCenter license. Can be empty for a perpetual license
licenseEdition	string	-	Edition of the applied vCenter license
licenseName	string	-	Name of the applied vCenter license
licenseTotal	string	count	Total quantity provided by the license
licenseUsed	string	count	License quantity currently in use
version	string	-	vCenter server version
build	string	-	vCenter server build number
fullName	string	-	Full product name and version string provided by VMware
apiVersion	string	-	vSphere Web Services API version
deploymentType	string	-	vCenter deployment type (for example, Embedded PSC)
timeZone	string	-	Time zone configured on the vCenter appliance
agentName	string	-	Host name of the server where the agent collecting the VMware information is installed

Some items are provided only by the appliance API of vSphere 7 or later. In environments where that API is not available, the values are empty or displayed as substitute values.

Adjust whether to collect with the following option.

whatap.conf

vm.inventory.vcenter.enabled=true    # Whether to collect (Default: true)

vm_vhost_inventory

Configuration information of ESXi hosts. One host becomes one row, and it is sent only when a value changes. It provides everything from the vCenter, data center, and cluster the host belongs to, through the CPU, memory, GPU, and network configuration, to the operational status.

• Collection target: VMware ESXi

Field	Type	Unit	Description
vCenterServerIp	string	-	IP address of the vCenter server that collected this ESXi host information. In environments that collect from multiple vCenters, it identifies which vCenter the host belongs to
dataCenterName	string	-	Name of the data center that contains the host in the vCenter inventory structure
clusterName	string	-	Name of the cluster the host belongs to. Empty or None for a standalone host or a host that is not in a cluster
folder	string	-	Host folder path where the host is located in the vSphere inventory. Displayed based on the host folder under the data center or a management folder created by the user. Interpret it separately from the actual resource configuration location, such as the cluster, resource pool, or datastore
vHostName	string	-	ESXi host name or connection address identified in vCenter. Usually in the form of a host IP or FQDN
esxiHostNameFqdn	string	-	Fully qualified domain name of the ESXi host. Displayed according to the DNS or host name settings
esxiHostNameShort	string	-	Short host name of the ESXi host. The name with the domain removed from the FQDN
esxiHostNameDomain	string	-	Domain area configured in the ESXi host name. Empty when no domain is configured
esxiManagementIp	string	-	Management IP address of the ESXi host. The representative management IP used for vCenter connections, the management network, and API communication
esxiVmIps	string	-	IP information of the VMkernel interfaces configured on the host. Used for management, vMotion, vSAN, iSCSI/NFS, and other network purposes
manufacturer	string	-	Host hardware manufacturer (for example, HP, Dell, Lenovo)
model	string	-	Host hardware model name. A server model or workstation model is displayed
hostType	string	-	Configuration type of the host. cluster when it belongs to a cluster, standalone when it is a standalone host
serialNumber	string	-	Serial number of the host equipment. Used as identifying information for asset management, maintenance contracts, and vendor support requests
uuid	string	-	Unique identifier of the host system. Serves as supplementary identification even when names are duplicated or the IP changes
cpuModel	string	-	CPU model name installed on the host (for example, Intel Xeon, Intel Core, AMD EPYC)
cpuSockets	string	-	Number of physical CPU sockets. Used for licensing, capacity planning, and checking the physical CPU configuration
cpuCores	string	-	Total number of physical CPU cores on the host. The basis for allocatable virtual machine resources and CPU capacity planning
cpuThreads	string	-	Number of logical CPU threads including hyper-threading. The logical CPU unit that can actually be scheduled for virtual machines
hyperThreadingEnabled	string	-	Whether hyper-threading is enabled on the host. Used to interpret the difference between the cpuCores and cpuThreads values
cpuTotalMhz	string	MHz	Total processing capacity of all CPUs on the host. Calculated from the number of CPU cores and the clock information
totalMemoryGb	string	GiB	Total memory capacity installed on the host. The basis for the capacity available for virtual machine placement and memory resource planning
gpuCount	string	count	Number of physical GPU devices identified on the host. Calculated from the VGA, 3D Controller, and Display Controller class devices among the PCI device information
gpuDevices	string	-	List of GPU devices identified on the host. Displays the manufacturer, model name, and PCI device identifier (for example, NVIDIA A100, NVIDIA T4, NVIDIA L40)
gpuProfiles	string	-	List of vGPU profile or shared passthrough GPU type candidates available on the host. The value can be empty depending on the NVIDIA vGPU Manager, the ESXi and vCenter versions, the license, and how the API exposes them
vmGuestCount	string	-	Number of virtual machine objects connected to the host. Used to identify the virtual machine placement and density per host
datastoreCount	string	-	Number of datastores accessible from the host. Used to identify the number of storage connections and the configuration scale available to the host
dataStores	string	-	List of datastores accessible from the host. The storage resources where virtual machine disks can be stored
totalCapacityGb	string	GiB	Total capacity of the datastores accessible from the host. Shared datastores can be included, so do not treat it as the capacity of the host alone
networkCount	string	-	Number of networks or port groups connected to the host
networks	string	-	List of networks or port groups connected to the host. Used to check the network connection range available to virtual machines
nicCount	string	-	Number of physical NICs configured on the host. The basis for checking the scale of physical connections for the management, virtual machine, vMotion, and storage networks
nics	string	-	List of physical NICs configured on the host
vSwitchCount	string	-	Number of virtual switches configured on the host. Used to identify whether standard or distributed switches are used and the network configuration scale
vSwitchs	string	-	List of vSwitches configured on the host
networkInfo	string	-	vSwitch or network configuration information on the host
vMotionEnabled	string	-	Whether vMotion is enabled on the host. Used to determine whether virtual machines can be migrated without downtime
haEnabled	string	-	Whether HA is enabled in the cluster the host belongs to. None or false for a standalone host or when cluster HA is disabled
productVersion	string	-	ESXi product version. Used to determine the patch level, feature support, and compatibility
buildNumber	string	-	ESXi build number. The basis for checking the detailed patch version, responding to security vulnerabilities, and reviewing compatibility
license	string	-	Type of VMware license applied to the host. Used to determine the available feature range and operational constraints
bootTime	string	-	Time the host was last booted. Used to check long-term operation and the recovery point after an incident
upTime	string	-	Time the host has been running since the last boot. Used to identify servers that have not been restarted for a long time
timezone	string	-	Time zone information based on the host or vCenter. The reference time for interpreting event times and performance data times
connectionState	string	-	Host connection status as seen by vCenter. Displayed as connected, disconnected, notResponding, and so on, and directly related to whether collection is possible
powerState	string	-	Power state of the host. poweredOn is normally the operational state
overallStatus	string	-	Overall host status determined by vCenter. Displayed as green, yellow, red, gray, and so on, and used to quickly determine whether there is a warning or failure
maintenanceMode	string	-	Whether the host is in maintenance mode. When true, the host may be excluded from operation for virtual machine migration, patching, or other work

You can join the virtual machine list with the host list in this table using vHostKey in vm_vmguest_inventory.

vm_vmguest_inventory

Configuration information of virtual machines. One virtual machine becomes one row, and it is sent only when a value changes.

• Collection target: VMware ESXi

Field	Type	Unit	Description	Remarks
vHostKey	-	-	Identification key of the ESXi host on which the virtual machine is running. The same value as the key in vm_vhost_inventory	New in 2.9.23
vmName	string	-	Display name of the virtual machine managed in vCenter or ESXi. The representative name operators use to identify it in the vSphere environment
guestHostname	string	-	Host name recognized inside the guest operating system. Collected through VMware Tools, so the value can be empty depending on the Tools status or the guest settings
vCenterServerInfo	string	-	Information about the vCenter server that collected this virtual machine information. In environments that collect from multiple vCenters, it identifies which vCenter the virtual machine belongs to
datacenter	string	-	Name of the vCenter data center that contains the virtual machine
folder	string	-	VM folder path where the virtual machine is located in the vSphere inventory. A logical folder location from the VMs and Templates perspective, separate from the execution or resource allocation location such as the host, cluster, or resource pool
cluster	string	-	Name of the ESXi cluster the virtual machine belongs to. The value can be empty in a standalone host environment or when it is not in a cluster
vHostInfo	string	-	Information about the ESXi host where the virtual machine is running or registered. The display format can vary between the host name, IP, and FQDN depending on the collection source
resourcePool	string	-	Name of the resource pool the virtual machine belongs to. Used to identify the resource policy, allocation structure, and service-level groups
ipAddress	string	-	Representative IP address identified in the guest operating system. Depending on the VMware Tools status, network settings, and multi-NIC configuration, the value can be empty or show only the representative IP
guestOS	string	-	Guest operating system type configured for the virtual machine. Based on the OS type selected in vSphere, so it can differ from the actual OS
guestFullName	string	-	Detailed name of the guest operating system. Identified through VMware Tools or the virtual machine settings
guestId	string	-	Guest operating system identifier used by vSphere (for example, rhel8_64Guest, windows9Server64Guest)
vcpu	string	count	Number of virtual CPUs allocated to the virtual machine
sockets	string	count	Number of virtual CPU sockets configured for the virtual machine. Used to check the vCPU configuration and how the guest operating system recognizes the CPUs
coresPerSocket	string	count	Number of cores per socket configured for the virtual machine. Used to check the CPU topology recognized by the guest operating system and the licensing policy
totalMemory	string	MiB	Total memory capacity allocated to the virtual machine. The basis for judging the resource allocation scale and whether memory needs to be expanded
vGpuCount	string	count	Number of vGPUs allocated to the virtual machine. Calculated from the vGPU devices connected as Virtual PCI Passthrough devices. 0 for a virtual machine with no vGPU allocated
vGpuProfiles	string	-	vGPU profile information allocated to the virtual machine (for example, grid_a100-1-5c, grid_t4-2q). The value can be empty or displayed differently depending on the vCenter and ESXi versions, the NVIDIA vGPU Manager configuration, and how the API exposes it
provisionedSpace	string	GiB	Total storage capacity provisioned for the virtual machine. Can be larger than the actual usage in a thin provisioning environment
usedSpace	string	GiB	Storage capacity actually in use by the virtual machine. Used for datastore usage analysis and capacity management
datastore	string	-	List of datastores in use by the virtual machine. Used to check the storage where the disk files and configuration files are located
network	string	-	Network or port group information connected to the virtual machine. Used to check the network connection configuration and to distinguish the service and management networks
vmPathName	string	-	Datastore path where the virtual machine configuration file is located. Usually in the [datastore] path/vm.vmx form
vmToolsVersion	string	-	VMware Tools version installed on the virtual machine. Related to whether guest information collection, IP identification, and heartbeat status checks are possible
template	string	-	Whether the object is a VM template. When true, it is a deployment template rather than a running virtual machine
firmware	string	-	Firmware type of the virtual machine. Displayed as bios or efi and used to check the boot method, secure boot, and compatibility
version	string	-	Virtual hardware version of the virtual machine. Used to determine ESXi and vCenter compatibility, supported features, and whether migration is possible
cpuHotAddEnabled	string	-	Whether adding vCPUs while the virtual machine is running is enabled. When true, CPUs can be added without powering off the virtual machine
cpuHotRemoveEnabled	string	-	Whether removing vCPUs while the virtual machine is running is enabled. Whether it actually works depends on the supported operating system and the settings
memoryHotAddEnabled	string	-	Whether adding memory while the virtual machine is running is enabled. When true, memory can be expanded without powering off the virtual machine
maxMksConnections	string	count	Maximum number of MKS connections that can access the virtual machine console at the same time. Used to check console access limits and operational access control
creationDate	string	-	Date and time the virtual machine was created. Used for lifecycle management and tracking newly created virtual machines
uuid	string	-	BIOS UUID of the virtual machine. It can also be referenced inside the guest operating system, and it can be duplicated depending on how the virtual machine is cloned or recreated
instanceUuid	string	-	UUID that vCenter uses to identify the virtual machine instance. The main basis for identifying a virtual machine within the same vCenter
annotation	string	-	Description or note entered in the vSphere virtual machine settings
powerState	string	-	Power state of the virtual machine. Displayed as poweredOn, poweredOff, suspended, and so on
overallStatus	string	-	Overall status of the virtual machine object. Displayed as green, yellow, red, gray, and so on, and used to quickly check for alarms or configuration problems
vmGuestState	string	-	State seen from the VMware Tools or guest operating system perspective. Displayed as running, notRunning, shuttingDown, standby, and so on
guestHeartbeatStatus	string	-	Guest operating system heartbeat status identified through VMware Tools. Displayed as green, yellow, red, gray, and so on

You can join the virtual machine list and the host list with vHostKey. Use it to find all virtual machines placed on a specific host at once.

Note

The existing vCenter monitoring (feature project) will end service on December 31, 2026 and will be replaced by the vCenter feature of the server monitoring product. For more information, see the vCenter monitoring document.