Server inventory collection items

Apart from the performance metrics, the server monitoring agent periodically collects the hardware configuration of the server. Performance metrics alone do not show which device is connected where, so even when a metric looks abnormal, it is difficult to identify the device causing it. Viewing the inventory items together shows which NUMA node the GPU and the network card are attached to, and how the GPUs are connected to each other. This document describes the collected fields of each inventory item, for operators who run AI and HPC workloads and need to check the device configuration as well.

Inventory items are sent only when a value changes. The agent checks the configuration every hour and sends only the items that were added, changed, or deleted compared with the previous collection.

Fields marked with a version and New in the Remarks column of each table were added in that version. Fields without a mark have been provided since the category was added.

server_rdma_inventory

Configuration information of RDMA (InfiniBand, RoCE) devices and ports. One row is collected per port of an RDMA device, and it provides everything from the device model, firmware, and link speed to the connected operating system network interface.

• Collection target: Linux
• Default collection interval: 1 hour

Field	Type	Unit	Description
interfaceName	string	-	RDMA device name and port number (for example, mlx5_0:1)
rdmaDevice	string	-	RDMA device name (for example, mlx5_0)
rdmaNetdev	string	-	Operating system network interface connected to the device. Multiple names are separated by |
rdmaNetdevIp	string	-	IP address per network interface (CIDR notation). Multiple addresses are separated by |
rdmaDeviceType	string	-	ASIC type of the device (for example, MT4123)
rdmaDeviceModel	string	-	Device model name (for example, ConnectX-6)
rdmaLinkLayer	string	-	Link layer. InfiniBand or Ethernet
rdmaNumaNode	string	-	NUMA node number the device belongs to. -1 when it cannot be determined
rdmaPciBusId	string	-	PCI address (for example, 0000:81:00.0)
rdmaPort	string	-	Port number
rdmaRate	string	-	Port link speed (for example, 100 Gb/sec (4X EDR))
rdmaMtu	string	-	Active MTU of the port
rdmaNetdevMtu	string	-	MTU per network interface. Multiple values are separated by |
rdmaDriverName	string	-	Kernel driver name (for example, mlx5_core)
rdmaDriverVersion	string	-	Driver version
rdmaFirmwareVersion	string	-	Device firmware version
rdmaBondMaster	string	-	Name of the bond the network interface belongs to
rdmaNodeGuid	string	-	Node GUID of the device
rdmaPortGuid	string	-	Port GUID of the port
rdmaSysImageGuid	string	-	System Image GUID of the device
rdmaSubnetPrefix	string	-	Subnet prefix the port belongs to
rdmaVlanId	string	-	VLAN ID per network interface
storageMount	string	-	Remote storage mount point connected through that network interface

Adjust whether to collect and the interval with the following options.

whatap.conf

rdma.inventory.enabled=true      # Whether to collect (Default: true)
rdma.inventory.interval=3600     # Collection interval in seconds (Default: 3600)

server_gpu_topology

The connection relationship between two GPUs. One row is collected per physical GPU pair. Use it to check by which path two GPUs are connected and whether they can exchange data directly.

• Collection target: Linux
• Default collection interval: 1 hour

Field	Type	Unit	Description
baseGpuIndex	string	-	Base GPU number
targetGpuIndex	string	-	Peer GPU number. Always larger than the base GPU number
topologyType	string	-	Connection level between the two GPUs. One of PIX, PXB, PHB, NODE, SYS, or NVn. INCONSISTENT when the values are not consistent
topologyConsistent	string	-	Whether the results queried in both directions match. true or false
interconnectType	string	-	Connection method. NVSwitch, NVLink, PCIe, or Unknown
nvLinkCount	-	count	Number of NVLink connections. 0 when it is not an NVLink connection
baseGpuMigMode	string	-	MIG configuration status of the base GPU. Enabled, Disabled, NotSupported, or Unknown
targetGpuMigMode	string	-	MIG configuration status of the peer GPU
peerToPeerSupported	string	-	Whether direct communication (P2P) between the two GPUs is supported. true, false, or unknown
peerToPeerStatus	string	-	P2P read status code (for example, OK, NS, CNS)

The closer the connection level is to PIX, the physically closer the two GPUs are; the closer it is to SYS, the farther apart they are. Grouping GPUs that handle the same job into close pairs reduces communication latency.

whatap.conf

gpu.topology.inventory.enabled=true      # Whether to collect (Default: true)
gpu.topology.inventory.interval=3600     # Collection interval in seconds (Default: 3600)

Note

If GPU collection is disabled or the GPU driver cannot be accessed, this item is not collected. On a server with a single GPU, no pair is formed, so the collection result is empty.

server_system_topology

Configuration information that links CPU sockets, NUMA nodes, and PCIe devices in parent-child relationships. One connection relationship becomes one row. Use it to view the device layout of the whole server as a tree.

• Collection target: Linux, Windows
• Default collection interval: 1 hour

Field	Type	Unit	Description
parentDevice	string	-	Name of the parent device (for example, SYSTEM, NUMA_NODE:0, PCI_BRIDGE:0000:00:01.1)
childDevice	string	-	Name of the child device (for example, CPU_SOCKET:0, GPU0, ens192)
deviceType	string	-	Type of the child device. CPU_SOCKET, NUMA_NODE, CPU_CORES, MEMORY, PCI_ROOT, GPU, NVSWITCH, RDMA_NIC, NIC, NVME, PCI_BRIDGE, HOST_BRIDGE, or PCI_DEVICE
pciBusId	string	-	PCI address for devices in the PCI layer, and an identifier in the socket:0 or node:0 form for devices in the CPU layer
pcieDepth	string	-	Depth of the PCIe path. 0 for the CPU, NUMA, and memory layers

The name of the top of the tree is SYSTEM. The collection uses only the information provided by the kernel (sysfs on Linux, device information on Windows), so it does not run any separate command.

whatap.conf

system.topology.enabled=true      # Whether to collect (Default: true)
system.topology.interval=3600     # Collection interval in seconds (Default: 3600)

server_numa_inventory

Configuration information per NUMA node. One NUMA node becomes one row, and it provides the CPUs and memory that belong to the node, together with the GPU, RDMA, and NVMe devices attached to the same node.

• Collection target: Linux
• Default collection interval: 1 hour

Field	Type	Unit	Description	Remarks
numaNodeId	string	-	NUMA node number
cpuList	string	-	List of CPUs that belong to the node (for example, 0-7,16-23)
logicalCores	-	count	Number of logical CPUs that belong to the node
memorySize	-	byte	Memory size that belongs to the node
gpuIndexes	string	-	List of NVIDIA GPU indexes attached to the node. Separated by a comma and a space
rdmaDevices	string	-	List of RDMA device names attached to the node (for example, mlx5_0, mlx5_1). Separated by a comma and a space
rdmaNetDevs	string	-	List of network interfaces bound to those RDMA devices. Separated by a comma and a space
nvmeDevices	string	-	List of NVMe device names connected to that NUMA node (for example, nvme0, nvme1). Separated by a comma and a space	New in 2.9.22
gpuRdmaAffinity	string	-	The closest RDMA device for each GPU of that NUMA node, displayed in the GPU0:mlx5_0(PIX) form. Multiple entries are separated by a comma and a space	New in 2.9.22

The value in parentheses in gpuRdmaAffinity is the distance between the GPU and the RDMA device. From closest to farthest, the order is PIX, PXB, PHB, NODE, and SYS, and only the device at the closest level is displayed for each GPU. Use it to check whether the two devices are attached to the same node in configurations where GPU communication goes through an RDMA device.

Adjust whether to collect and the interval with the following options.

whatap.conf

numa.inventory.enabled=true      # Whether to collect (Default: true)
numa.inventory.interval=3600     # Collection interval in seconds (Default: 3600)