GPU Topology
Home > Select Project > GPU > GPU Topology
When you combine multiple GPUs for a single training or inference job, the connection paths between GPUs greatly affect job speed. GPUs directly connected via NVLink communicate fast, but GPUs connected via a PCIe path that crosses CPU sockets communicate more slowly. GPU Topology shows the GPU-to-GPU connections of a single server with summary cards and an N×N matrix. Use it to choose the GPU combination to assign to a job, or to find why only a specific server is slow among servers of the same model. It is intended for operators who run GPU servers and need to check GPU placement and connection status.
Supported agent versions
Supported only on Linux agents 2.9.25 or later. Windows servers and servers whose version information cannot be verified do not appear in the server selection list. The data shown on the screen is the server_gpu_topology inventory item collected by the agent. For collected fields and options, see Server inventory collection items.
Screen layout
| Area | Description |
|---|---|
| Header | Server selection, supported agent information, agent status, last configuration change time, Usage guide, refresh |
| Summary cards | Number of GPU pairs per connection type. Click a card to highlight only the cells of that type |
| Matrix | One cell per GPU pair. Each cell shows the connection path code |
| Right panel | Three sections: Selected connection, How to read, and GPU list |
Header
- Server selection: Select a server from the dropdown. The list uses the format "Hostname (Public IP)" and shows the private IP on the second line. You can search by hostname, public IP, or private IP.
- Supported agent information: Hover over the icon to check the supported OS and minimum version.
- Last configuration change: The time the GPU connection configuration last changed. This is not the data collection time.
- If the agent is inactive, a yellow warning bar below the header shows the last collection time. Configuration changes after that are not reflected on the screen.
- Click the Usage guide button to see how to use the screen in specific situations, such as assigning GPUs, comparing training speed, and checking MIG settings.
Summary cards
| Card | Description |
|---|---|
| All | All GPU pairs. The subtext shows the number of GPUs and the number of GPUs with MIG enabled |
| NVLink connection | Pairs directly connected via NVLink |
| Via PCIe | Pairs that communicate via PCIe. If MIG is enabled on both GPUs, shown as "Via PCIe (between MIG-enabled GPUs)" |
| MIG direct communication unavailable | Pairs where MIG is enabled on only one GPU |
| MIG asymmetric | MIG asymmetric pairs that support P2P |
| Needs review | Pairs whose results queried in both directions differ, which MIG settings do not explain |
Click a card to keep only the cells of that type highlighted and dim the rest. Click the same card again to return to All.
Matrix
Rows and columns are GPU numbers, and the cell where two GPUs meet shows the connection path code. Diagonal cells where a GPU meets itself show -. GPUs with MIG enabled have a MIG badge in the header.
| Code | Description |
|---|---|
NVn | Directly connected via NVLink. n is the number of links; higher is faster |
PIX | Connected through a single PCIe bridge. The closest PCIe path |
PXB | Connected through multiple PCIe bridges |
PHB | Connected through the CPU's PCIe host bridge |
NODE | Connected across host bridges within the same NUMA node |
SYS | Connected across CPU sockets. The slowest path |
For PCIe paths, from PIX toward SYS, the two GPUs are physically farther apart and communicate more slowly. If direct communication (P2P) between two GPUs is not supported, a P2P unavailable badge appears in the upper right of the cell.
Click a cell to pin the result in the Selected connection section on the right. Click the same cell again or press Esc to clear the selection. Within the matrix, you can move between cells with the arrow keys.
Right panel
- Selected connection: Shows the result and explanation for the two selected GPUs, and How to use them together in one job. As detailed values, you can check the connection path, connection method (NVSwitch, NVLink, PCIe), NVLink count, P2P support, and BDF (PCI address).
- How to read: Explains cell colors and codes. Items not present on the current server are dimmed. Click the title to collapse or expand it; the collapsed state is remembered in the browser.
- GPU list: Shows the number and NVLink connection count of each GPU. For GPUs with MIG enabled, the NVLink connection cell shows "Not used with MIG".
Interpreting results
Needs review requires a settings check
If the results queried in both directions differ and MIG settings do not explain it, the pair is shown as Needs review. Check the server's ACS, IOMMU, and GPU driver settings. Comparing the matrix with another server of the same model shows whether a pair that should be NVLink appears as PCIe.
When the screen has no data
| Message | Cause |
|---|---|
| No servers are collecting topology | All GPU servers are Windows, or the agent version is earlier than 2.9.25 |
| Unsupported OS | The server specified in the URL is not Linux |
| Unsupported agent version | The agent of the server specified in the URL is earlier than 2.9.25. Agent update required |
| No topology has been collected yet | Before the agent's first collection. The first collection takes up to 1 hour |
A server with a single GPU has no GPU pairs to compare, so the matrix is empty. If the project has no GPU servers, a guide for enabling the GPU monitoring feature is shown. For setup details, see GPU Feature.