GPU Service Grouping

Note

For an overview of GPU monitoring, see the WhaTap GPU Monitoring document.

WhaTap Operator's GPU monitoring feature attaches Kubernetes labels from application (GPU-using) pods to GPU metrics, enabling grouping and querying of GPU usage by service or project in WhaTap.

• Target: WhatapAgent CR (monitoring.whatap.com/v2alpha1)

• Related fields: spec.features.k8sAgent.gpuMonitoring.groupLabel, groupLabelAllowlistRegex

• Key result label: whatap_kube_label_gpu_group

Tip

What problem does this solve?

Default DCGM metrics (e.g., DCGM_FI_DEV_GPU_UTIL) include GPU/node/pod identifiers, but do not contain "service"-level information that users operate (e.g., project ID, team, workload type).

As a result:

• Queries like "total GPU utilization used by Project A" are difficult.

• Cost and usage accounting in multi-tenant environments (multiple teams sharing a cluster) is difficult.

With this feature, labels attached to pods (e.g., prjId=projectA) are automatically propagated to DCGM metric labels and normalized to WhaTap's common label whatap_kube_label_gpu_group, making them immediately available as grouping keys in dashboards and searches.

Quick Start

Enable GPU Monitoring + Group Label

apiVersion: monitoring.whatap.com/v2alpha1
kind: WhatapAgent
metadata:
  name: whatap
  namespace: whatap-monitoring
spec:
  features:
    k8sAgent:
      gpuMonitoring:
        enabled: true
        # Label key to use as grouping key
        groupLabel: "prjId"
        # (Optional) Regex for exposed label keys. If not set, ^(prjId)$ is applied automatically
        # groupLabelAllowlistRegex: "^(prjId|team)$"
        # (Optional) Attach cluster label to all GPU metrics
        clusterName: "prod-gpu-cluster"

Add Labels to GPU-Using Pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-train
  namespace: ml
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ml-train
  template:
    metadata:
      labels:
        app: ml-train
        prjId: "projectA"        # ◀ Grouping key (same as groupLabel)
    spec:
      containers:
        - name: trainer
          image: my-registry/trainer:latest
          resources:
            limits:
              nvidia.com/gpu: 1

Note

Labels must be applied at the Pod level (spec.template.metadata.labels), not at the Deployment level.

Use Cases

Usage by Project (prjId)

• groupLabel: prjId
• Pod labels: prjId: projectA, prjId: projectB
• Group by whatap_kube_label_gpu_group in WhaTap dashboard → compare GPU utilization and memory usage by project

Billing by Team

gpuMonitoring:
  enabled: true
  groupLabel: "team"
  groupLabelAllowlistRegex: "^(team)$"

• Pod labels: team: search, team: recsys, …

Exposing Multiple Labels Together

While only one label is normalized via groupLabel (whatap_kube_label_gpu_group), you can use groupLabelAllowlistRegex to also expose additional pod labels as metric labels.

gpuMonitoring:
  enabled: true
  groupLabel: "prjId"
  groupLabelAllowlistRegex: "^(prjId|team|env)$"

• whatap_kube_label_gpu_group ← value of prjId
• team, env labels are also exposed as-is in DCGM metrics (not normalized)

Verification

Check dcgm-exporter Environment Variables

kubectl -n whatap-monitoring get ds whatap-node-agent \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="dcgm-exporter")].env}' | jq

The output must contain the following two environment variables for pod label collection to be active.

• DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS=true
• DCGM_EXPORTER_KUBERNETES_POD_LABEL_ALLOWLIST_REGEX=^(prjId)$

Query Metrics Directly (Pod must have labels)

# Connect to one dcgm-exporter Pod
POD=$(kubectl -n whatap-monitoring get pod -l name=whatap-node-agent -o name | head -1)
kubectl -n whatap-monitoring exec "$POD" -c dcgm-exporter -- \
  wget -qO- localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL | head

DCGM_FI_DEV_GPU_UTIL{gpu="0", ..., pod="ml-train-xxxx", namespace="ml", prjId="projectA"} 87

Check OpenAgent Scrape Configuration

kubectl -n whatap-monitoring get configmap whatap-open-agent-config -o yaml | grep -A2 whatap_kube_label_gpu_group

The output must contain the following relabel block. The value of source_labels must match the configured groupLabel key.

- action: replace
  source_labels: [prjId]
  target_label: whatap_kube_label_gpu_group

Troubleshooting

Symptom	Cause / Checklist
prjId label not visible in metrics	(1) Check if prjId is set in pod labels (spec.template.metadata.labels), (2) Check if dcgm-exporter has been restarted to apply new env vars
prjId is visible but whatap_kube_label_gpu_group is not	OpenAgent pod must be restarted after ConfigMap update. kubectl rollout restart deploy/whatap-open-agent -n whatap-monitoring
groupLabel value contains -, ., /	Per Prometheus label key rules, source_labels is composed with _-replaced keys. The allowlist regex uses the original key, so set it as-is
Want to expose multiple labels	Specify groupLabelAllowlistRegex as ^(prjId|team|env)$
Need cluster identification on all GPU metrics	Set the clusterName field (attaches cluster=<value> to all metrics)

Field Reference

Field	Description	Default
gpuMonitoring.enabled	Enable GPU monitoring (dcgm-exporter)	false
gpuMonitoring.groupLabel	Pod label key used for grouping. When set, normalized to whatap_kube_label_gpu_group	-
gpuMonitoring.groupLabelAllowlistRegex	Regex for pod label keys that dcgm-exporter exposes	^(<groupLabel>)$
gpuMonitoring.clusterName	Attach cluster label to all GPU metrics	-

Reference

• git repo: https://github.com/whatap/whatap-operator.git

• Implementation: internal/controller/install_agents.go (addDcgmExporterToNodeAgent, GPU scrape config generation)

• CRD definition: api/v2alpha1/whatapagent_types.go (GpuMonitoringSpec)

• Examples: examples/whatap-agent-all-options.yaml, examples/whatap-agent-gpu-custom-env.yaml