System topology

Home > Select Project > Labs > System topology

GPU job performance depends not only on the GPU itself but also on which CPU socket and NUMA node the GPU is attached to, and how close the RDMA devices used with it are. If a job uses cores or memory on a different socket from the GPU, data crosses between sockets and latency increases. System topology is a Labs feature that shows, as a diagram, a single server's CPU sockets, NUMA nodes, and the GPUs, RDMA devices, NVMe devices, network cards, and NVSwitches connected under them. Use it to decide which CPU cores to pin a GPU job to, to choose the RDMA device to use with a GPU, or to find ports where RDMA communication fails. It is intended for operators who need to check device placement on AI and HPC servers.

Note

Supported agent versions

Supported only on Linux and Windows agents 2.9.25 or later. AIX, HP-UX, and Solaris servers and servers with earlier versions do not appear in the server selection list. Servers without GPUs can also be queried. The data shown on the screen is the server_system_topology, server_numa_inventory, and server_rdma_inventory inventory items collected by the agent. For collected fields and options, see Server inventory collection items.

To view system configuration information as tables, see System configuration info. To view GPU-to-GPU connections, see GPU Topology. For how to access Labs features and precautions, see Laboratory.

Screen layout

Area	Description
Header	Server selection, supported agent information, agent status, last configuration change time, Usage guide, refresh
Issue summary line	Number of issues such as unplugged-cable or inactive RDMA ports and GPUs not recognized by NVML
Diagram	CPU sockets, NUMA node cards, and device chips shown as a tree
Information panel	Information on the selected device, nearby resources to use with it, and How to read

Header

• Server selection: Select a server from the dropdown. The list uses the format "Hostname (Public IP)" and shows the private IP on the second line. You can search by hostname, public IP, or private IP.
• Last configuration change: The time the device configuration last changed. This is not the data collection time.
• If the agent is inactive, a yellow warning bar below the header shows the last collection time. Configuration changes after that are not reflected on the screen.
• Click the Usage guide button to see how to use the screen in specific situations, such as pinning CPU cores, choosing RDMA devices, and checking RDMA communication.

Diagram

The diagram is laid out so that the whole view fits on one screen without dragging. The layout varies with the number of sockets.

Socket	Layout
1–2	Horizontal tree. NUMA cards shown vertically under each socket
3–4	Cross layout. NUMA nodes collapsed into summary boxes
5 or more	Ring summary. Click a socket to expand only that socket as a tree. Click View all to return to the ring

A NUMA card shows the CPU range and core count of the node, memory, and the device chips connected to that node. Each device type starts on a new line.

Chip	Description
GPU	GPU and model name
RDMA	RDMA network port (InfiniBand, Ethernet)
NVSwitch	NVLink device connecting GPUs. Connects GPUs regardless of NUMA
NVMe·NIC	NVMe storage devices and regular network cards

A dashed border on an RDMA chip indicates an inactive port, and a red border indicates an alert port, such as one with an unplugged cable.

Adjust the zoom level with the Fit, Zoom out, and Zoom in buttons. A manually set zoom level ranges from 50% to 250%. Click Fit to return to the zoom level at which the whole diagram fits on one screen.

Device selection and information panel

Click a device chip or NUMA summary box to select the device and open the information panel. Click the same chip again to clear the selection. Cards and chips unrelated to the selected device are dimmed.

• When you select a GPU, the CPU range and memory of the same NUMA node are highlighted. RDMA devices on the same node get a proximity badge (PIX, PXB, PHB, NODE, SYS). The closer to PIX, the physically closer the device is to the GPU.
• When you select an RDMA chip, the information panel shows the port status.

Port status	Description
Active	Link is up and can communicate
Inactive	Port is down
Cable unplugged	Looking for the peer (Polling). Check the cable and switch
No Subnet Manager	Physical link is up, but the Subnet Manager does not respond (LinkUp, Init)
Other status	A status not listed above. Check the raw value in the RDMA port table
Status not collected	The agent does not send the port status

Use cases

Pinning CPU cores for a GPU job

Click a GPU chip to highlight the CPU range and memory of the same NUMA node as that GPU. Pinning the job to these cores keeps memory access from crossing sockets.

Choosing an RDMA device to use with a GPU

Select a GPU and check the proximity badges. If you use an RDMA device with a close level such as PIX or PXB, data moves without going through the CPU.

When RDMA communication fails

Click an RDMA chip with a red border to check the port status. If the status is Cable unplugged, check the cable and switch port.

When the screen has no data

Message	Cause
No system configuration has been collected yet	Before the agent's first collection. The first collection takes up to 1 hour
Could not collect CPU and NUMA information	An environment where NUMA information is not exposed, such as a container
Unsupported OS	The server specified in the URL is not Linux or Windows
Unsupported agent version	The agent of the server specified in the URL is earlier than 2.9.25. Agent update required
No servers are collecting topology	No server meets the support conditions (Linux or Windows 2.9.25 or later)
Cannot verify the agent version	No agent version information for the server specified in the URL
The requested server cannot be found	The server specified in the URL has been excluded or deleted

If the project has no servers, an agent installation guide is shown.