GPU Metrics
WhaTap server monitoring collects and provides GPU performance and status metrics from NVIDIA GPU environments across four perspectives: the GPU server, individual GPUs, GPU-using processes, and GPU fault events. You can check GPU utilization, compute resource activity, memory usage, power, temperature, clock speeds, PCIe/NVLink throughput, ECC errors, performance throttle states, and Xid events — enabling step-by-step analysis from server-wide GPU utilization to per-GPU bottleneck analysis, process-level root cause tracing, and driver/hardware fault event inspection. Some metrics are collected only for Physical GPUs, and in MIG environments the collection targets and interpretation criteria may differ, so it is recommended to check the GPU Type and collection targets together. In addition, Xid events are collected based on the fault/event history that the NVIDIA driver records in the system log. When analyzing the root cause, check the event timestamp, GPU identification, Xid Code, PID, Task Name, and detailed message together.
Note
Agent configuration is required to enable GPU monitoring. For more information, see GPU Feature Settings.
GPU performance metrics are provided in 4 categories based on the analysis perspective: server-wide GPU aggregation, per-GPU detail, GPU-using process, and GPU Xid events.
-
infra_gpu
A category that aggregates the status of all GPUs installed in the server, allowing you to check the performance and status from a server-wide GPU perspective. It provides GPU utilization, memory, power, temperature, clock speeds, PCIe/NVLink, ECC, and Violation states based on averages, sums, and maximums, making it easy to quickly assess the overall utilization level and potential bottlenecks of the GPU server.
-
infra_gpu_id
A category that provides performance and status at the individual GPU level. You can check per-GPU utilization, compute resource activity, memory, temperature, power, clock speeds, PCIe/NVLink, ECC, and Violation states to analyze bottlenecks or throttle conditions occurring on a specific GPU. Collectible metrics may differ between Physical GPUs and MIG instances, so GPU Type and collection targets must be checked together.
-
infra_gpu_process
A category that provides resource usage information at the per-process level for GPU-using processes. Along with PID, process name, user, and GPU UUID, you can check SM utilization, memory controller utilization, encoder/decoder utilization, FB memory usage, and more to trace which processes are consuming GPU resources. Note that some per-process GPU utilization metrics may have different measurement criteria than the overall GPU metrics, so when assessing individual process contributions, interpret them together with the overall GPU metrics.
-
infra_gpu_xid
A category that provides GPU Xid event information recorded by the NVIDIA driver. Based on the
NVRM: Xidlogs recorded in/var/log/messagesor the system log, it collects the Xid Code, GPU Index, PCI Bus ID, occurrence time, PID, Task Name, and detailed message, allowing you to trace GPU anomaly events such as GPU driver, hardware, PCIe, memory, and application errors. Since not every Xid event immediately indicates a hardware failure, it is recommended to interpret them together with the meaning of the Xid Code, occurrence frequency, concurrent GPU performance metrics, process information, and system logs.
GPU Metrics Detail
The following sections describe the detailed metrics collected per category.
infra_gpu
A category that collects the aggregated usage status of all GPUs installed in the server. It provides GPU utilization, memory usage, power, temperature, clock speeds, PCIe/NVLink throughput, ECC, and violation states recalculated as averages, sums, and maximums from a server-wide GPU perspective. Use this to understand the overall utilization level, bottlenecks, and throttle states of the entire GPU server.
- To collect DCP (Datacenter Profiling) metrics, the nvidia-dcgm service (or nv-hostengine process) must be running.
- For Agent 2.9.16 and later, when nvidiav2.collector=true is set, metric data is collected automatically via the connectable NVML and DCGM Library. (nvidia-smi or dcgmi queries are not used.)
- Default collection interval: 30s
| Field Category field name | Title Metric name in Performance Summary / Performance Detail / Metrics Explorer | Unit | Description | Notes | Collection Condition | Min. Agent Version |
|---|---|---|---|---|---|---|
| name_50 | GPU Name | string | Represents the model name of the GPU. The display string may vary slightly depending on the driver version or collection path. | 2.9.3 | ||
| model_family | GPU Model Family | string | A normalized value that maps GPU model names — which may appear differently depending on driver version, firmware, or collection path — to representative family names such as A100, H100, or L40S. Used as an identifier for consistent per-model aggregation, filtering, and comparison. | 2.9.7 | ||
| gpu_count | GPU Count | string | The number of Physical GPUs recognized by the server. Used as the baseline when interpreting average, total, and ratio metrics across all GPUs. | 2.9.3 | ||
| gpu_utilization | GPU Utilization | % | The average utilization across individual GPUs in the server. In Physical GPU environments, this represents the standard GPU Util value; in MIG environments, the GR Engine Active value per instance can be used as a representative GPU utilization. Shows the overall utilization level of all GPUs in the server. | AVG of installed Physical GPUs | Required to run nvidia-dcgm service or nv-hostengine process in MIG Enabled environments | 2.9.6 |
| enc_util_206 | Encoder Util | % | The average encoder engine utilization across individual GPUs in the server. Shows how much video encoding work is being handled by the GPUs. Higher values indicate greater use of NVENC-family engines. | AVG of installed Physical GPUs | 2.9.3 | |
| dec_util_207 | Decoder Util | % | The average decoder engine utilization across individual GPUs in the server. Shows how much video decoding work is being handled by the GPUs. Higher values indicate greater use of NVDEC-family engines. | AVG of installed Physical GPUs | 2.9.3 | |
| gr_engine_active_1001 | GR Engine Active Util | % | The average Graphics/Compute Engine active ratio across individual GPUs in the server. In MIG instance environments, this metric can be used as a representative GPU utilization for each instance. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| sm_active_1002 | SM Active Util | % | The average SM (Streaming Multiprocessor) active ratio across individual GPUs in the server. Shows the overall utilization level of SM resources used in actual computation. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| sm_occupancy_1003 | SM Occupancy | % | The average SM Occupancy across individual GPUs in the server. Indicates how fully the SMs are loaded with active warps — unlike a simple active ratio, this shows the density of SM resource utilization. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| tensor_active_1004 | Tensor Core Util | % | The average Tensor Core active ratio across individual GPUs in the server. Useful for understanding the workload mix that uses Tensor Cores, such as AI/deep learning computation. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fp64_active_1006 | FP64 Compute Util | % | The average FP64 compute active ratio across individual GPUs in the server. Indicates the double-precision compute load level. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fp32_active_1007 | FP32 Compute Util | % | The average FP32 compute active ratio across individual GPUs in the server. Indicates the single-precision compute load level. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fp16_active_1008 | FP16 Compute Util | % | The average FP16 compute active ratio across individual GPUs in the server. Indicates the half-precision compute load level, which can be important for AI inference/training workloads. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| memory_copy_util_204 | Memory Copy Util | % | The average memory copy engine utilization across individual GPUs in the server. Shows how actively data movement paths — such as GPU memory to internal compute areas, Host ↔ Device copies, and Device ↔ Device copies — are being used. Higher values may indicate that data movement load is greater than computation itself, or that memory copy operations are becoming a bottleneck. Note that this value represents the activity level of the memory copy/transfer engine, not memory capacity utilization. | AVG of installed Physical GPUs | 2.9.3 | |
| dram_active_1005 | DRAM Active Util | % | The average DRAM active ratio across individual GPUs in the server. Represents the proportion of time the GPU DRAM memory subsystem was active. Shows how actively the GPU memory bandwidth is being used; tends to be high in memory-intensive or memory-access-heavy workloads. Sustained high values may indicate that GPU memory bandwidth is a bottleneck rather than compute performance. Note that this value is an activity indicator showing how busy the GPU DRAM was — not a memory capacity metric like FB Memory Used. | AVG of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fb_total_250 | FB Total Memory | MiB | The sum of FB (Frame Buffer) total memory capacity across individual GPUs in the server. Represents the total GPU memory capacity of the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| fb_free_251 | FB Free Memory | MiB | The sum of FB free memory capacity across individual GPUs in the server. Represents the total available GPU memory of the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| fb_used_252 | FB Used Memory | MiB | The sum of FB used memory capacity across individual GPUs in the server. Represents the total GPU memory currently in use across the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| fb_resv_253 | FB Reserved Memory | MiB | The sum of FB reserved memory capacity across individual GPUs in the server. Represents the total GPU memory reserved by the system, drivers, firmware, etc. | SUM of installed Physical GPUs | 2.9.3 | |
| fb_USDP_254 | FB Memory Usage | % | The overall FB memory utilization for the entire server. Rather than a simple average of individual GPU utilization rates, this is recalculated as FB Used sum / FB Total sum * 100. | AVG of installed Physical GPUs | 2.9.3 | |
| bar1_total_90 | BAR1 Total Memory | MiB | The sum of BAR1 total capacity across individual GPUs in the server. BAR1 is the memory-mapped region used by the host to access GPU memory; this represents the total BAR1 capacity of the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| bar1_used_92 | BAR1 Used Memory | MiB | The sum of BAR1 usage across individual GPUs in the server. Represents the total BAR1 memory currently in use across the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| bar1_free_93 | BAR1 Free Memory | MiB | The sum of BAR1 free capacity across individual GPUs in the server. Represents the total available BAR1 capacity of the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| bar1_usdp | BAR1 Memory Usage | % | The overall BAR1 utilization for the entire server. Rather than a simple average of individual GPU ratios, this is recalculated as BAR1 Used sum / BAR1 Total sum * 100. | AVG of installed Physical GPUs | 2.9.3 | |
| ecc_sbe_aggregate_total_312 | ECC SBE Total | count | The sum of cumulative ECC Single Bit Error counters across individual GPUs in the server. Represents the total accumulated single-bit ECC errors across the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| ecc_dbe_aggregate_total_313 | ECC DBE Total | count | The sum of cumulative ECC Double Bit Error counters across individual GPUs in the server. Represents the total accumulated double-bit ECC errors across the entire server. | SUM of installed Physical GPUs | 2.9.3 | |
| ecc_sbe_aggregate_delta_312 | ECC SBE Delta | count | The sum of ECC Single Bit Error increments across individual GPUs in the server during the collection interval. Represents the number of new single-bit ECC errors that occurred in the recent interval. | SUM of installed Physical GPUs | 2.9.16 | |
| ecc_dbe_aggregate_delta_313 | ECC DBE Delta | count | The sum of ECC Double Bit Error increments across individual GPUs in the server during the collection interval. Represents the number of new double-bit ECC errors that occurred in the recent interval. | SUM of installed Physical GPUs | 2.9.16 | |
| gpu_temp_150 | GPU Temperature | °C | The maximum temperature among individual GPUs in the server. Represents the highest temperature across all GPUs in the server, used to quickly identify whether a specific GPU is overheating. | MAX of installed Physical GPUs | 2.9.10 | |
| memTemp_140 | GPU Memory Temperature | °C | The maximum memory temperature among individual GPUs in the server. Typically the HBM or GDDR memory temperature; used to check for memory thermal issues or cooling anomalies. In memory-intensive workloads, this may rise before the GPU core temperature, so it is recommended to monitor both together. | MAX of installed Physical GPUs | 2.9.16 | |
| power_usage_155 | Power Usage | watt | The sum of power consumption across individual GPUs in the server. Represents the total power currently consumed by all GPUs in the server, in watts (W). | SUM of installed Physical GPUs | 2.9.3 | |
| energy_cumulative_daily_kwh | Energy Cumulative | kWh | The cumulative GPU energy consumption of the entire server since 00:00 of the current day. This is a reference estimate calculated by integrating the powerUsage values collected via NVIDIA drivers/DCGM at each collection interval, and may differ from precise power measurements from a dedicated energy meter. Use for reference to track GPU power usage trends and daily usage patterns. | SUM of installed Physical GPUs | 2.9.16 | |
| fan_speed_191 | Fan Speed | % | The average fan speed across individual GPUs in the server. Indicates the cooling device activity level for GPU models equipped with fans. | AVG of installed Physical GPUs | 2.9.3 | |
| sm_clock_100 | SM Clock | MHz | The average SM clock speed across individual GPUs in the server. Represents the average core clock level at which the compute units are operating. | AVG of installed Physical GPUs | 2.9.3 | |
| memory_clock_101 | Memory Clock | MHz | The average memory clock speed across individual GPUs in the server. Represents the operating speed level of GPU memory. | AVG of installed Physical GPUs | 2.9.3 | |
| video_clock_102 | Video Clock | MHz | The average video engine clock speed across individual GPUs in the server. Represents the operating speed level of video processing engines. | AVG of installed Physical GPUs | 2.9.3 | |
| pcie_tx_bytes_1009 | PCIE TX | bytes/s | The sum of PCIe transmit throughput across individual GPUs in the server. Represents the total PCIe data transmitted from the GPUs to the host or external destinations, in bytes/s. | SUM of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| pcie_rx_bytes_1010 | PCIE RX | bytes/s | The sum of PCIe receive throughput across individual GPUs in the server. Represents the total PCIe data received by the GPUs from the host or external sources, in bytes/s. | SUM of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| pcie_tx_rx_bytes | PCIE Total | bytes/s | The sum of PCIe transmit and receive throughput across individual GPUs in the server. Represents the total PCIe data movement (PCIE TX + PCIE RX) across the entire server. | SUM of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.16 |
| nvlink_tx_bytes_1011 | NVLink TX | bytes/s | The sum of NVLink transmit throughput across individual GPUs in the server. Represents the total data transmitted between GPUs via NVLink, in bytes/s. | SUM of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| nvlink_rx_bytes_1012 | NVLink RX | bytes/s | The sum of NVLink receive throughput across individual GPUs in the server. Represents the total data received between GPUs via NVLink, in bytes/s. | SUM of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| nvlink_tx_rx_bytes | NVLink Total | bytes/s | The sum of NVLink transmit and receive throughput across individual GPUs in the server. Represents the total GPU-to-GPU communication (NVLink TX + NVLink RX) across the entire server. | SUM of installed Physical GPUs | nvidia-dcgm service or nv-hostengine process must be running | 2.9.16 |
| power_violation_rate_240 | Power Violation Rate | % | The representative value showing the highest power cap violation rate among individual GPUs in the server. Used to quickly identify if any GPU is experiencing performance throttling due to power constraints. | MAX of installed Physical GPUs | 2.9.16 | |
| thermal_violation_rate_241 | Thermal Violation Rate | % | The representative value showing the highest thermal limit violation rate among individual GPUs in the server. Used to quickly identify if any GPU is experiencing performance throttling due to overheating. | MAX of installed Physical GPUs | 2.9.16 | |
| low_util_violation_rate_244 | Low Util Violation Rate | % | The representative value showing the highest low-utilization violation rate among individual GPUs in the server. Used to check if any GPU is in a state that meets the configured low-utilization conditions. | MAX of installed Physical GPUs | 2.9.16 | |
| board_limit_violation_rate_243 | Board Limit Violation Rate | % | The representative value showing the highest board-level hardware limit violation rate among individual GPUs in the server. Used to identify the GPU most affected by board-level constraints such as power delivery and board design limits. | MAX of installed Physical GPUs | 2.9.16 | |
| sync_boost_violation_rate_242 | Sync Boost Violation Rate | % | The representative value showing the highest sync boost violation rate among individual GPUs in the server. Used to identify the GPU most affected by constraints imposed to synchronize the operating speeds of multiple GPUs. | MAX of installed Physical GPUs | 2.9.16 | |
| reliability_violation_rate_245 | Reliability Violation Rate | % | The representative value showing the highest reliability protection violation rate among individual GPUs in the server. Used to identify the GPU most affected by constraints for stability and hardware protection. | MAX of installed Physical GPUs | 2.9.16 | |
| power_violation_240 | Power Violation | ns | The sum of time individual GPUs in the server spent in a power cap throttled state during the collection interval. In nanoseconds (ns); shows the total duration of performance throttling due to power constraints across the entire server. | SUM of installed Physical GPUs | 2.9.16 | |
| thermal_violation_241 | Thermal Violation | ns | The sum of time individual GPUs in the server spent in a thermal limit throttled state during the collection interval. In nanoseconds (ns); shows the total duration of performance throttling due to overheating across the entire server. | SUM of installed Physical GPUs | 2.9.16 | |
| low_util_violation_244 | Low Util Violation | ns | The sum of time individual GPUs in the server spent in a low-utilization throttled state during the collection interval. In nanoseconds (ns); shows the total duration of low-utilization condition violations across the entire server. | SUM of installed Physical GPUs | 2.9.16 | |
| board_limit_violation_243 | Board Limit Violation | ns | The sum of time individual GPUs in the server spent in a board-level hardware limit throttled state during the collection interval. In nanoseconds (ns); used to identify the GPU most affected by board-level constraints. | SUM of installed Physical GPUs | 2.9.16 | |
| sync_boost_violation_242 | Sync Boost Violation | ns | The sum of time individual GPUs in the server spent in a sync boost throttled state during the collection interval. In nanoseconds (ns); used to identify the GPU most affected by constraints imposed to synchronize the operating speeds of multiple GPUs. | SUM of installed Physical GPUs | 2.9.16 | |
| reliability_violation_245 | Reliability Violation | ns | The sum of time individual GPUs in the server spent in a reliability protection throttled state during the collection interval. In nanoseconds (ns); used to identify the GPU most affected by constraints for stability and hardware protection. | SUM of installed Physical GPUs | 2.9.16 | |
| response_timeout | Response Timeout | string | If no data is collected through NVML for a certain period, this value is recorded as False. | 2.9.19 |
infra_gpu_id
A category that collects per-GPU performance and status information for each individual GPU installed in the server. Provides per-GPU utilization, SM/DRAM/Tensor/FP compute utilization, memory capacity/usage, temperature, power, clock speeds, PCIe/NVLink, ECC, and violation states. Use this to check for overheating, power throttling, memory shortage, or interconnect bottlenecks on a specific GPU.
- Criteria for distinguishing collection targets as Physical/MIG: For metrics that represent the overall hardware status of the board/card, only Physical metrics are collected where Physical-only collection suffices.
- To collect DCP (Datacenter Profiling) metrics, the nvidia-dcgm service (or nv-hostengine process) must be running.
- For Agent 2.9.16 and later, when nvidiav2.collector=true is set, metric data is collected automatically via the connectable NVML and DCGM Library. (nvidia-smi or dcgmi queries are not used.)
- Default collection interval: 30s
| Field Category field name | Title Metric name in Performance Summary / Performance Detail / Metrics Explorer | Unit | Description | Target | Collection Condition | Min. Agent Version |
|---|---|---|---|---|---|---|
| index | GPU Index | string | The index value used to identify the GPU within the server, representing the sequence number assigned by the OS or driver. | Physical, MIG | 2.9.3 | |
| name_50 | GPU Name | string | Represents the model name of this GPU. The display string may vary slightly depending on the driver version or collection path. | Physical, MIG | 2.9.3 | |
| uuid_54 | GPU UUID | string | The unique identifier (UUID) of this GPU. Used to reliably identify the GPU regardless of reboots or device order changes. | Physical, MIG | 2.9.3 | |
| gpuType | GPU Type | string | Displays the type of this GPU as Physical or MIG. | Physical, MIG | 2.9.3 | |
| modelFamily | GPU Model Family | string | Represents the upper model family for this GPU. Since the same GPU may display slightly different model names depending on driver version or collection path, this is provided as a simplified model name normalized to a common standard such as A100, H100, or L4. Used for consistent classification, aggregation, and comparison of GPUs of the same family. | Physical, MIG | 2.9.7 | |
| gpuUtilization | GPU Utilization | % | The utilization of this GPU. In Physical GPU environments, this represents the standard GPU Util value; in MIG environments, the GR Engine Active value can be interpreted as a representative GPU utilization. | Physical, MIG | Required to run nvidia-dcgm service or nv-hostengine process in MIG Enabled environments | 2.9.6 |
| encUtil_206 | Encoder Util | % | The encoder engine utilization of this GPU. Higher values indicate greater use of NVENC-family engines. | Physical | 2.9.3 | |
| decUtil_207 | Decoder Util | % | The decoder engine utilization of this GPU. Higher values indicate greater use of NVDEC-family engines. | Physical | 2.9.3 | |
| grEngineActive_1001 | GR Engine Active Util | % | The Graphics/Compute Engine active ratio of this GPU. In MIG instance environments, this metric can be used as a representative GPU utilization for each instance. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| smActive_1002 | SM Active Util | % | The SM (Streaming Multiprocessor) active ratio of this GPU. Shows the overall utilization level of SM resources used in actual computation. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| smOccupancy_1003 | SM Occupancy | % | The SM Occupancy of this GPU. Shows how fully the SMs are loaded with active warps — better represents the density of SM resource utilization than a simple active ratio. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| tensorActive_1004 | Tensor Core Util | % | The Tensor Core active ratio of this GPU. Useful for understanding the proportion of workloads that use Tensor Cores, such as AI/deep learning computation. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fp64Active_1006 | FP64 Compute Util | % | The FP64 compute active ratio of this GPU. Shows the double-precision compute load level. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fp32Active_1007 | FP32 Compute Util | % | The FP32 compute active ratio of this GPU. Shows the single-precision compute load level. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fp16Active_1008 | FP16 Compute Util | % | The FP16 compute active ratio of this GPU. Shows the half-precision compute load level, which can be important for AI inference/training workloads. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| memoryCopyUtil_204 | Memory Copy Util | % | The memory copy engine utilization of this GPU. Shows how actively data movement paths — such as GPU memory to internal compute areas, Host ↔ Device copies, and Device ↔ Device copies — are being used. Higher values may indicate that data movement load is greater than computation itself, or that memory copy operations are becoming a bottleneck. Note that this value represents the activity level of the memory copy/transfer engine, not memory capacity utilization. MIG-enabled Physical GPUs cannot collect data. | Physical | 2.9.3 | |
| dramActive_1005 | DRAM Active Util | % | The proportion of time the DRAM memory subsystem of this GPU was active. Shows how actively the GPU memory bandwidth is being used; tends to be high in memory-intensive or memory-access-heavy workloads. Sustained high values may indicate that GPU memory bandwidth is a bottleneck rather than compute performance. Note that this value is an activity indicator showing how busy the GPU DRAM was — not a memory capacity metric like FB Memory Used. | Physical, MIG | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| fbTotal_250 | FB Total Memory | MiB | The total FB (Frame Buffer) memory capacity of this GPU. | Physical, MIG | 2.9.3 | |
| fbFree_251 | FB Free Memory | MiB | The available FB memory capacity of this GPU. Represents the amount of GPU memory currently available. | Physical, MIG | 2.9.3 | |
| fbUsed_252 | FB Used Memory | MiB | The used FB memory capacity of this GPU. Represents the amount of GPU memory currently occupied. | Physical, MIG | 2.9.3 | |
| fbResv_253 | FB Reserved Memory | MiB | The reserved FB memory capacity of this GPU. Represents the GPU memory area reserved by the system, drivers, firmware, etc. | Physical, MIG | 2.9.3 | |
| fbUsdp_254 | FB Memory Usage | % | The FB memory utilization of this GPU. Calculated as FB Used / FB Total * 100. | Physical, MIG | 2.9.3 | |
| bar1Total_90 | BAR1 Total Memory | MiB | The total BAR1 capacity of this GPU. BAR1 is the memory-mapped region used by the host to access GPU memory. | Physical, MIG | 2.9.3 | |
| bar1Used_92 | BAR1 Used Memory | MiB | The BAR1 memory currently in use on this GPU. | Physical, MIG | 2.9.3 | |
| bar1Free_93 | BAR1 Free Memory | MiB | The available BAR1 capacity on this GPU. | Physical, MIG | 2.9.3 | |
| bar1Usdp | BAR1 Memory Usage | % | The BAR1 utilization of this GPU. Calculated as BAR1 Used / BAR1 Total * 100. | Physical, MIG | 2.9.3 | |
| eccSbeAggregateTotal_312 | ECC SBE Total | count | The total number of accumulated ECC Single Bit Errors on this GPU. As a cumulative counter, continuous trend observation is important. | Physical | 2.9.3 | |
| eccDbeAggregateTotal_313 | ECC DBE Total | count | The total number of accumulated ECC Double Bit Errors on this GPU. Double-bit errors indicate a more severe condition than single-bit errors. | Physical | 2.9.3 | |
| eccSbeAggregateDelta_312 | ECC SBE Delta | count | The number of newly incremented ECC Single Bit Errors during the most recent collection interval on this GPU. Used to check new single-bit error occurrences in the interval. | Physical | 2.9.16 | |
| eccDbeAggregateDelta_313 | ECC DBE Delta | count | The number of newly incremented ECC Double Bit Errors during the most recent collection interval on this GPU. Used to check new double-bit error occurrences in the interval. | Physical | 2.9.16 | |
| gpuTemp_150 | GPU Temperature | °C | The current temperature of this GPU. The primary indicator for assessing overheating at the individual GPU level. | Physical | 2.9.3 | |
| memTemp_140 | GPU Memory Temperature | °C | The current memory temperature of this GPU. Typically the HBM or GDDR memory temperature; used to check for memory thermal issues or cooling anomalies. In memory-intensive workloads, this may rise before the GPU core temperature, so it is recommended to monitor both together. | Physical | 2.9.16 | |
| powerUsage_155 | Power Usage | watt | The current power consumption of this GPU, in watts (W). | Physical | 2.9.3 | |
| powerRate | Power Rate | % | The power utilization rate of this GPU. Interpreted as Power Usage / Power Limit * 100; shows how close the current power usage is to the power limit. | Physical | 2.9.3 | |
| energyCumulativeDaily | Energy Cumulative | kWh | The daily energy usage of this GPU, accumulated since 00:00 of the current day. This is a reference estimate calculated by integrating the powerUsage values collected via NVIDIA drivers/DCGM at each collection interval, and may differ from precise power measurements from a dedicated energy meter. Use for reference to track GPU power usage trends and daily usage patterns. | Physical | 2.9.16 | |
| pstate_190 | Performance State | P | The current performance state (P-state) of this GPU. Lower numbers indicate higher performance states, with P0 being the highest. The range is typically P0 to P15; when stored, only the numeric portion without the P prefix is used (e.g., P0 is stored as 0, P2 as 2). | Physical | 2.9.3 | |
| fanSpeed_191 | Fan Speed | % | The current fan speed of this GPU as a percentage. Shows the cooling device activity level on models equipped with fans. | Physical | 2.9.3 | |
| smClock_100 | SM Clock | MHz | The SM clock speed of this GPU in MHz. Shows the core clock level at which the compute units are operating. | Physical | 2.9.3 | |
| memoryClock_101 | Memory Clock | MHz | The memory clock speed of this GPU in MHz. Shows the operating speed of the GPU memory. | Physical | 2.9.3 | |
| videoClock_102 | Video Clock | MHz | The video engine clock speed of this GPU in MHz. Shows the operating speed level of video processing engines. | Physical | 2.9.3 | |
| pcieTxBytes_1009 | PCIE TX | bytes/s | The data throughput transmitted from this GPU to external destinations via PCIe, in bytes/s. | Physical | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| pcieRxBytes_1010 | PCIE RX | bytes/s | The data throughput received by this GPU from external sources via PCIe, in bytes/s. | Physical | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| pcieTxRxBytes | PCIE Total | bytes/s | The total PCIe transmit and receive throughput of this GPU in bytes/s. Equals PCIE TX + PCIE RX. | Physical | nvidia-dcgm service or nv-hostengine process must be running | 2.9.16 |
| nvlinkTxBytes_1011 | NVLink TX | bytes/s | The data throughput transmitted from this GPU to other GPUs via NVLink, in bytes/s. | Physical | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| nvlinkRxBytes_1012 | NVLink RX | bytes/s | The data throughput received by this GPU from other GPUs via NVLink, in bytes/s. | Physical | nvidia-dcgm service or nv-hostengine process must be running | 2.9.3 |
| nvlinkTxRxBytes | NVLink Total | bytes/s | The total NVLink transmit and receive throughput of this GPU in bytes/s. Equals NVLink TX + NVLink RX. | Physical | nvidia-dcgm service or nv-hostengine process must be running | 2.9.16 |
| powerViolationRate_240 | Power Violation Rate | % | The proportion of time this GPU was in a power cap throttled state. Shows how much performance throttling occurred due to the power limit. | Physical | 2.9.16 | |
| thermalViolationRate_241 | Thermal Violation Rate | % | The proportion of time this GPU was in a thermal limit throttled state. Shows how much performance throttling occurred due to overheating. | Physical | 2.9.16 | |
| lowUtilViolationRate_244 | Low Util Violation Rate | % | The proportion of time this GPU was in a low-utilization throttled state. Shows how much of the configured low-utilization conditions were triggered. | Physical | 2.9.16 | |
| boardLimitViolationRate_243 | Board Limit Violation Rate | % | The proportion of time this GPU was in a board-level hardware limit throttled state. Shows how much performance throttling occurred due to board-level constraints such as power delivery and board design limits. | Physical | 2.9.16 | |
| syncBoostViolationRate_242 | Sync Boost Violation Rate | % | The proportion of time this GPU was in a sync boost throttled state. Shows how much performance throttling occurred due to constraints imposed to synchronize the operating speeds of multiple GPUs. | Physical | 2.9.16 | |
| reliabilityViolationRate_245 | Reliability Violation Rate | % | The proportion of time this GPU was in a reliability protection throttled state. Shows how much performance throttling occurred due to constraints for stability and hardware protection. | Physical | 2.9.16 | |
| powerViolation_240 | Power Violation | ns | The time this GPU spent in a power cap throttled state during the collection interval. In nanoseconds (ns); shows the duration of performance throttling due to the power limit. | Physical | 2.9.16 | |
| thermalViolation_241 | Thermal Violation | ns | The time this GPU spent in a thermal limit throttled state during the collection interval. In nanoseconds (ns); shows the duration of performance throttling due to overheating. | Physical | 2.9.16 | |
| lowUtilViolation_244 | Low Util Violation | ns | The time this GPU spent in a low-utilization throttled state during the collection interval. In nanoseconds (ns); shows how long the low-utilization condition violation persisted. | Physical | 2.9.16 | |
| boardLimitViolation_243 | Board Limit Violation | ns | The time this GPU spent in a board-level hardware limit throttled state during the collection interval. In nanoseconds (ns); shows the duration of performance throttling due to board-level constraints. | Physical | 2.9.16 | |
| syncBoostViolation_242 | Sync Boost Violation | ns | The time this GPU spent in a sync boost throttled state during the collection interval. In nanoseconds (ns); shows the duration of performance throttling due to constraints imposed to synchronize the operating speeds of multiple GPUs. | Physical | 2.9.16 | |
| reliabilityViolation_245 | Reliability Violation | ns | The time this GPU spent in a reliability protection throttled state during the collection interval. In nanoseconds (ns); shows the duration of performance throttling due to constraints for stability and hardware protection. | Physical | 2.9.16 |
infra_gpu_process
A category that collects GPU and system resource usage information for each individual process using the GPU. Provides GPU index/UUID, PID, process name/type, SM utilization, memory controller utilization, encoder/decoder utilization, FB memory usage, and protected memory (CCPM). It also provides CPU utilization, RSS memory, and process I/O information, enabling process-level analysis of not only GPU resource usage causes but also potential CPU, memory, and I/O bottlenecks.
- From Agent 2.9.16 onward, when nvidiav2.collector=true is set, the mapping between processes and MIG instances can be confirmed in MIG environments.
- Default collection interval: 30s
| Field Category field name | Title Metric name in Performance Summary / Performance Detail / Metrics Explorer | Unit | Description | Notes | Collection Condition | Min. Agent Version |
|---|---|---|---|---|---|---|
| index | GPU Index | string | The index of the GPU on which this process is running. Used as a sequence number to identify GPUs within the server. | Agent 2.9.16+: Can identify processes assigned to MIG | 2.9.3 | |
| uuid | GPU UUID | string | The UUID of the GPU on which this process is running. As the GPU's unique identifier, it reliably identifies the GPU regardless of device order changes, unlike the index. | 2.9.3 | ||
| pid | Process ID | string | The PID (Process ID) of the process using the GPU. | 2.9.3 | ||
| process_name | Process Command | string | The name of the process using the GPU. For example, python, cuda_program, Xorg, etc. may be displayed. | 2.9.3 | ||
| process_type | Process Type | string | The type of the running GPU process. C indicates a Compute process (CUDA, AI/compute tasks), and G indicates a Graphics process (GUI, rendering, display acceleration). Both types may coexist depending on the environment. | 2.9.3 | ||
| owner | Owner | string | The user account that launched this process. | 2.9.16 | ||
| args | Arguments | string | The command-line arguments of this process. | 2.9.16 | ||
| stime | StartTime | string | The start time of this process in epoch time. | 2.9.16 | ||
| gpu_utilization | GPU Utilization | % | Displays the overall GPU Utilization of the GPU device to which this process is bound at the time of collection. This is not a directly measured per-process GPU utilization; rather, it shows the overall utilization of the connected GPU alongside the process information. As a result, multiple processes using the same GPU may show identical or similar values, and this value alone should not be used to determine how much each individual process contributed to GPU utilization. | 2.9.16 | ||
| sm_util | SM Util | % | The proportion of the GPU's SM (Streaming Multiprocessor) compute resources being used by this process. Shows the utilization level of GPU core resources used in actual computation; higher values indicate the process is making heavy use of GPU compute resources. Note that this is a process-perspective SM utilization and may use different measurement criteria than the device-wide SM Active Util (%). Therefore, even if a specific process shows high SM Util, the GPU's SM Active Util may appear lower or different depending on idle time in the overall observation window, inter-kernel gaps, scheduling with other processes, MIG/Time-slicing effects, etc. (For processes using MIG Instances, per-process SM Util values may not be collected or displayed due to limitations in the driver and monitoring interface.) | 2.9.3 | ||
| mem_controller_util | Memory Controller Util | % | The proportion of the GPU memory controller being used by this process. This is not how much VRAM capacity is occupied, but rather how actively read/write and data movement requests to GPU memory were made. Higher values may indicate the process is making heavy use of memory bandwidth or memory access paths. Note that this is a process-perspective memory controller utilization and differs in meaning from the device-wide Memory Copy Util (%) or DRAM Active Util (%). Memory Copy Util is closer to the overall GPU memory copy/transfer engine utilization, while DRAM Active Util is closer to the proportion of time GPU DRAM was active with read/write requests during the observation window. (For processes using MIG Instances, per-process SM Util values may not be collected or displayed due to limitations in the driver and monitoring interface.) | 2.9.15 | ||
| enc_util | Encoder Util | % | The proportion of the GPU encoder engine being used by this process. Primarily increases during video encoding, streaming, and video save/compression tasks. | 2.9.3 | ||
| dec_util | Decoder Util | % | The proportion of the GPU decoder engine being used by this process. Primarily increases during decoding of compressed video streams, playback, and video analysis preprocessing. | 2.9.3 | ||
| fb_used | FB Used | MiB | The Frame Buffer (GPU memory) size being used by this process, in MiB. Can be interpreted as the VRAM usage occupied by this process. | 2.9.3 | ||
| ccpm | CCPM | MiB | The Confidential Computing Protected Memory usage of this process, in MiB. Available on GPUs that support Confidential Computing, such as the NVIDIA H100. May display as 0 on GPUs that do not support this feature. | 2.9.3 | ||
| cpu | CPU | % | The total CPU utilization used by this process. | 2.9.15 | ||
| rss | RSS | Byte | The RSS (Resident Set Size) of this process; the amount of memory currently loaded in system memory. | 2.9.15 | ||
| total_char_io | Process IO Total | Bytes/s | The total character-based I/O (char I/O) throughput generated per second by this process. May include I/O to pipes, terminals, buffers, and cache layers in addition to disk, so it can be larger than actual physical disk I/O. | 2.9.15 |
infra_gpu_xid
A category that collects GPU fault/event history by parsing the NVRM: Xid logs that the NVIDIA driver records in /var/log/messages. It provides the Xid Code, GPU Index, PCI Bus ID, related PID/process name, and raw message, and is used to trace when a GPU error occurred, the affected GPU, the related process, and the error type. Since Xid can involve a wide range of causes — from simple application errors to driver, hardware, and PCIe issues — it is important to analyze the code meaning, occurrence frequency, GPU status metrics, and system logs together.
- Supported from Agent 2.9.19 or later.
- Default collection interval: 60s
| Field Category field name | Title Performance summary/detail/Metrics Explorer name | Unit | Description | Note | Collection condition | Minimum supported Agent Version |
|---|---|---|---|---|---|---|
| index | GPU Index | string | The index of the GPU where the Xid event occurred. A number used to identify the GPU device within the server; matched with the GPU Index shown in nvidia-smi to identify the GPU where the fault occurred. | 2.9.19 | ||
| pcieBusId | PCI Bus ID | string | The PCI Bus ID of the GPU where the Xid event occurred. Can be used as a secondary identifier for the physical GPU in environments where the GPU Index may change, and is referenced when checking the GPU device location within the same server. | 2.9.19 | ||
| agentTimestamp | Agent Timestamp | string | The actual event occurrence time recorded in /var/log/messages. | 2.9.19 | ||
| xidCode | XID Code | string | The Xid error code reported by the NVIDIA driver. A code that distinguishes various GPU anomaly states such as GPU driver, hardware, PCIe, memory, and application errors; used together with the meaning of each code to analyze the fault cause and determine whether an RMA (Return Merchandise Authorization) is needed. | 2.9.19 | ||
| pid | PID | string | The process ID included in the Xid log. Used to trace the responsible process when a GPU error occurs while a specific process is running. The value may be empty if the log has no PID information or the event occurred at the driver/hardware level. | 2.9.19 | ||
| taskName | Task Name | string | The Task Name recorded by the NVIDIA driver when the Xid occurred. A short name corresponding to the Linux /proc/{pid}/comm or per-thread /proc/{pid}/task/{tid}/comm value, it does not mean the full command line (cmdline). In multi-threaded applications, even for the same PID, different names such as python3 or pt_nccl_watchdg may appear depending on the task where the error was recorded. Therefore, when identifying the actual application, it is recommended to interpret it together with the PID, GPU PCI address, GPU UUID, and cmdline information. | 2.9.19 | ||
| message | Message | string | The detailed message of the NVRM: Xid log parsed from /var/log/messages. In addition to the Xid code, it may include the GPU UUID, PCI information, channel information, process information, and detailed error text; used for checking the raw text during fault cause analysis. | 2.9.19 |