Most of the AI infrastructure writing I read is about getting GPUs. Quota requests, spot capacity, which instance family is actually available in your region this week. Very little of it is about the GPUs you already have and aren’t using.
That gap turns out to be expensive, and it has a specific shape: Kubernetes schedules GPUs as an integer resource. A pod either holds nvidia.com/gpu: 1 or it doesn’t. Nothing in the scheduler, the autoscaler, or the node lifecycle has the faintest idea whether that GPU is running kernels or sitting at zero percent while someone’s notebook stays open over a long weekend.
I built a full stack to measure this on EKS — DCGM into Prometheus for the measurement, Karpenter on spot for the provisioning, and a small detector that closes the gap between them. This post is the part worth knowing even if you never run the code.
The blind spot
Here’s the whole problem in three rows:
| State | Does the cluster notice? | Who reclaims it |
|---|---|---|
| GPU node with no pods | yes | Karpenter consolidation |
| GPU node with pods, GPU busy | yes | nothing to reclaim — correct |
| GPU node with pods, GPU idle | no | nothing |
The first two rows are solved problems. Karpenter handles empty nodes well, and a busy GPU is a GPU doing its job.
The third row is the one that costs money. The node looks fully allocated, so consolidation won’t touch it. The pod is Running and healthy, so nothing restarts it. It’s invisible in exactly the way that lets it run for weeks.
Why GPU_UTIL lies to you
The obvious signal is DCGM_FI_DEV_GPU_UTIL, and it’s wrong in a way that specifically defeats what you’re hunting for.
That counter reports whether any kernel was resident on the device. It does not report whether that kernel did arithmetic. A tight polling loop that burns a core waiting on a queue reads as 100% utilized. Alert on it and you’ll miss precisely the workloads you’re looking for, while feeling confident you’ve got coverage.
The counter you want is DCGM_FI_PROF_GR_ENGINE_ACTIVE — the fraction of wall-clock time the graphics/compute engine was genuinely occupied. It comes from the profiling API, which means the exporter needs SYS_ADMIN:
securityContext:
capabilities:
add: ["SYS_ADMIN"]
That’s a real privilege escalation on the DaemonSet and worth a deliberate decision rather than a copy-paste. I set it explicitly and in one place for that reason.
The other thing to know: DCGM learns the GPU-to-pod mapping from the kubelet pod-resources socket. Mount it, and every metric arrives already labeled with namespace and pod. That single detail is what makes the rest of this possible — without it you know a GPU is idle but not who’s holding it, which is an observation rather than something anyone can act on.
Defining “idle” so it survives contact with reality
A GPU is idle when it is allocated — some pod is holding it — and inactive — engine activity has averaged under 5% for 15 minutes.
Both halves matter. Unallocated GPUs are Karpenter’s job and it does that job well. Allocated-and-inactive is the gap, and it needs a recording rule:
- record: gpu:engine_active:ratio
expr: |
max by (node, gpu, UUID, modelName, namespace, pod, container) (
DCGM_FI_PROF_GR_ENGINE_ACTIVE
)
or
max by (node, gpu, UUID, modelName, namespace, pod, container) (
DCGM_FI_DEV_GPU_UTIL / 100
)
- record: gpu:idle:indicator
expr: |
(avg_over_time(gpu:engine_active:ratio{pod!=""}[15m]) < bool 0.05)
- record: gpu:idle:ratio
expr: gpu:idle:count / clamp_min(gpu:allocated:count, 1)
The or falls back to the coarse counter on hardware or driver versions where profiling isn’t available — degraded, but not blind.
The {pod!=""} filter is doing the load-bearing work. And note the denominator on that last rule: allocated, not total.
It matters because the numerator only ever counts GPUs that are allocated and inactive. Divide that by total GPUs and every unallocated GPU in the fleet dilutes the number — spin up spare capacity nobody has claimed yet and your waste ratio politely falls, without one wasted GPU-hour being reclaimed. Dividing by allocated keeps the metric answering the question you actually asked: of the GPUs somebody has claimed, what fraction are doing nothing?
Price it from what you actually paid
A percentage doesn’t start conversations. A dollar figure does.
The temptation is a hardcoded price table. Don’t — it goes stale, and worse, it prices everything at on-demand, which understates your spot savings and makes the whole exercise look less valuable than it is.
Karpenter already publishes the number it used to make the purchasing decision, current spot price included:
# what idle GPUs cost per hour, priced from Karpenter's live estimate
sum(
gpu:idle:indicator
* on (node) group_left() (node:hourly_price_usd / clamp_min(node:gpu_count, 1))
)
Getting there takes one join through kube_node_labels, which is an info metric carrying node labels as its own labels. Relabel instance_type, capacity_type and zone to the short names Karpenter’s price series uses, and the two join directly:
- record: node:hourly_price_usd:detail
expr: |
max by (node, instance_type, capacity_type, zone) (
node:meta:info
* on (instance_type, capacity_type, zone) group_left()
karpenter_cloudprovider_instance_type_offering_price_estimate
)
Node price divided by GPUs on that node, summed over the idle ones. Now you have dollars per hour.
Alert on pods, not percentages
“GPU utilization is low” is an observation. “This pod has held an idle A10G for 40 minutes, costing $0.14/hr” is something a person will act on. Every alert names the pod and the money:
- alert: GPUIdleWhileAllocated
expr: avg_over_time(gpu:engine_active:ratio{pod!=""}[15m]) < 0.05
for: 15m
annotations:
summary: >-
GPU {{ $labels.gpu }} on {{ $labels.node }} is allocated to
{{ $labels.namespace }}/{{ $labels.pod }} but idle.
The one I’d add second is memory-held-without-compute:
- alert: GPUMemoryHeldWithoutCompute
expr: |
gpu:memory_used:ratio{pod!=""} > 0.2
and
avg_over_time(gpu:engine_active:ratio{pod!=""}[15m]) < 0.02
for: 20m
That’s the abandoned notebook, or a training process that died inside the container without exiting it. The CUDA context survives, the allocation survives, and from outside nothing looks wrong.
Except that alert, as written above, can never fire. I only found out by running it.
PromQL’s and matches on the entire label set of both sides. My two recording rules don’t produce the same one:
# carries modelName
max by (node, gpu, UUID, modelName, namespace, pod, container) (...) # engine_active
# does not
max by (node, gpu, UUID, namespace, pod, container) (...) # memory_used
One side carries modelName="Tesla T4", the other doesn’t, so no pair of series ever matches and the expression is permanently empty. Not an error, not a red panel — just an alert that sits there looking configured and never evaluates to anything. The fix is and ignoring (modelName), or aggregating both rules over identical labels.
And when I fixed that, it still didn’t fire, for a completely separate reason. My idle hog pins about 1.1 GiB on a T4 with 15,094 MiB of framebuffer — a ratio of 0.076, comfortably under the > 0.2 I’d written. The threshold was picked to sound reasonable rather than measured against the workload it was meant to catch.
Two independent bugs stacked in five lines of YAML, both silent, in a rule I’d have told you was obviously correct. Which is the real lesson: an alert you have never watched fire is not an alert, it’s a hypothesis. Force the condition on purpose — deploy something that genuinely holds memory and does nothing — and confirm you get paged before you trust it.
Closing the loop, carefully
Detection without action just produces a dashboard people stop looking at. But a script that deletes GPU pods is also an excellent way to lose a twelve-hour training run, so the reaper is opt-in twice over: it must be running with DRY_RUN=false, and the pod must carry gpu-cost.io/reapable=true.
Default posture is to inform. It annotates the pod with what it found and emits a Kubernetes Event:
msg = (
f"GPU {gpu} on {node} allocated to {ns}/{pod} but idle "
f"(engine active {activity:.1%} over 15m, ~${per_gpu:.3f}/hr wasted)."
)
The whole thing is standard library only — no pip install, so it runs on a stock python image with no build pipeline or registry behind it. It’s a CronJob, not a controller, because the decision cadence here is minutes and a controller would be more machinery than the problem deserves.
One deliberate detail: the per-GPU share of fleet waste is computed per event, so each one carries a real number instead of the aggregate. “$0.14/hr” attached to your pod lands differently than “the fleet is wasting $2.40/hr” in a channel you mute.
Proving the detector actually works
A detector you haven’t tried to fool is a detector you don’t have. Three workloads:
gpu-busy— real matmul on a three-minutes-on, two-minutes-off duty cycle. This is the important negative test: it proves the detector does not fire on a healthy training loop that pauses between epochs.gpu-idle-hog— reserves a GPU, pins some framebuffer so it looks alive to anything watching memory, then does nothing forever. This is the case the project exists to catch, and the reason memory-based detection alone isn’t enough.gpu-burst— a finite Job that works hard and exits. The control: node empties, consolidation reclaims it, cost drops.
The hog is worth reading, because it’s about six lines and it defeats most naive monitoring:
import torch, time
dev = torch.device("cuda")
held = torch.zeros(512, 1024, 1024, device=dev, dtype=torch.float16)
print("allocated 1GiB, now doing nothing forever", flush=True)
while True:
time.sleep(60)
To Kubernetes that node is fully allocated. To Karpenter it’s in use and must not be consolidated. To the bill it’s a full GPU-hour, every hour, until someone notices.
Do the arithmetic on that tensor, though, and it’s 1 GiB, not the 2 GB I’d written in the original — 512 × 1024 × 1024 half-precision elements at two bytes each. Measured on the T4 it settles at 1,144 MiB of 15,094 MiB once the CUDA context is counted, a ratio of 0.076. Which matters, because I’d set the memory alert to fire above 0.2. My own bait workload wasn’t big enough to trip my own alert — a threshold I’d picked because it sounded reasonable rather than by measuring the thing it was meant to catch.
What happened when I ran it
Two 45-minute phases, identical workloads, two g4dn.xlarge in us-east-1. Baseline is on-demand with WhenEmpty consolidation and the reaper in report-only mode. Optimized asks for spot, consolidates WhenEmptyOrUnderutilized after a minute, and lets the reaper reclaim.
| Metric | Baseline | Optimized | Change |
|---|---|---|---|
| Idle GPU ratio | 0.500 | 0.514 | +2.8% |
| Wasted spend | $0.526/hr | $0.607/hr | +15.3% |
| Total GPU spend | $1.052/hr | $1.133/hr | +7.7% |
The optimization delivered nothing. I’m publishing that because the reasons are specific, and because a post that only ever shows the configuration winning isn’t worth much.
What did work was the measurement. For 90 consecutive minutes the detector held at exactly one of two GPUs idle, with no flapping. gpu-busy was never falsely flagged — its duty cycle averages around 60% engine activity against a 5% threshold, so a healthy training loop that pauses between epochs sailed through untouched. Pricing resolved to $0.526/hr straight from Karpenter’s live estimate, the correct on-demand rate. The reaper fired nine times, each event naming the pod and what it was costing.
Nine times, for zero reclaimed GPU-hours. Which is the first reason.
The ReplicaSet beat the reaper. gpu-idle-hog is a Deployment, and the reaper’s only reclamation action is DELETE on the pod. Kubernetes did exactly what you’d want it to: recreated the pod within seconds, which re-took the same GPU, and the idle ratio settled back to 0.5. Nine evictions, nine resurrections.
That’s a design error in my detector, not in Kubernetes. Deleting a controller-managed pod isn’t reclamation, it’s a restart with extra steps — and the honest fix is to act on the owning controller, scaling it to zero or evicting with the owner reference, rather than swatting pods that something else is contractually obliged to bring back.
Spot wasn’t available, so there was no pricing delta either. Karpenter tried repeatedly across g4dn, g5 and g6:
InsufficientInstanceCapacity: There is no Spot capacity available
that matches your request.
VcpuLimitExceeded: You have requested more vCPU capacity than your
current vCPU limit of 8 allows for the instance bucket ...
My spot quota was fine at 32 vCPU. AWS just had no single-GPU spot capacity in those AZs that evening, so the pool fell back to on-demand — correct behaviour, and precisely why you configure fallback — but it means this run compared on-demand against on-demand. The slight increase in the optimized phase is churn from replacement nodes that kept failing to launch, not a regression.
So: the thing I built to find waste found it, accurately, and the two mechanisms meant to act on it were defeated by a controller doing its job and a spot market that had nothing to sell. Worth knowing before you trust either in production.
The bug that would have published a lie
Partway through the second phase I checked the headline number and found gpu:idle:ratio reading 2.5. A ratio bounded at 1. Idle spend was $2.104/hr against total spend of $1.052/hr — waste at double the entire bill.
The numerator and denominator didn’t share time semantics:
# numerator: resurrects every series seen in the last 15 minutes
sum(avg_over_time(gpu:engine_active:ratio{pod!=""}[15m]) < bool 0.05)
# denominator: an instant vector, counts only what exists now
count(gpu:engine_active:ratio{pod!=""})
Every eviction mints a new pod name and therefore a new series. The numerator was accumulating each dead pod for fifteen minutes — five of them — while the denominator counted the two alive.
Read that failure mode again, because the direction is the point: the more effectively the reaper reclaimed, the worse the fleet’s waste appeared. The metric penalized its own remedy, and did it only in the “after” scenario — the exact place a before/after comparison is most likely to be believed and least likely to be questioned. Left alone, my results file would have reported a 209% idle ratio and a 233% regression, and every one of those digits would have been generated by working code.
The fix is to gate the average on the raw instant vector, so only GPUs still allocated right now can count:
(avg_over_time(gpu:engine_active:ratio{pod!=""}[15m]) < bool 0.05)
and (gpu:engine_active:ratio{pod!=""})
Numerator 5 → 1. Ratio 2.5 → 0.500. Every figure above uses the corrected expression.
Gotchas and tips
- The 5% / 15-minute threshold is a starting point, not a constant. Inference services with bursty traffic legitimately sit near zero between requests. Tune it per workload class or you’ll page someone about a service that’s working correctly.
- Widen your instance families before you blame spot. Each
(family, size, AZ)combination is an independent capacity pool. Restrict to one family and you get one pool, which means interruptions arrive in a batch.["g4dn", "g5", "g6"]instead of["g4dn"]is the single highest-leverage spot setting I know of. - Set
consolidateAfteraggressively on spot. Baseline setups commonly run 30m. Spot nodes are cheap to re-acquire, so 1m costs little and saves the entire idle window. - Cap consolidation with disruption budgets (
nodes: "30%") so reclamation can’t stampede the fleet. - G and VT vCPU quota defaults to zero on new AWS accounts. Mirror your actual quota in the NodePool
limitsso Karpenter refuses cleanly instead of getting throttled by EC2 and leaving you to debug pending pods. - Don’t hardcode an alert duration that contradicts your
consolidateAfter. I wrote aGPUNodeUnclaimedTooLongalert withfor: 15magainst a NodePool set toconsolidateAfter: 30m. It fires a quarter of an hour before Karpenter is even permitted to act, so it reports normal waiting as a fault. Derive the alert window from the NodePool setting, or at minimum keep it comfortably longer. And when an empty node genuinely does overstay, the usual culprit is a PodDisruptionBudget that’s too strict to let the last pod go, or a pod with no controller that nothing will reschedule — a restrictive PDB blocks drain, not a missing one.
Where I’d take this next
The two failures above are the next work, in order: reclaim through the owning controller rather than the pod, and re-run the comparison when spot capacity exists so the pricing half of the story can actually be measured.
Beyond that, three directions I think are interesting:
MIG and time-slicing. Everything here assumes whole-GPU allocation. Once you’re slicing an A100 seven ways, “allocated” gets considerably more interesting and the pod-resources mapping needs revisiting.
Feeding it back into requests. The idle data is a request-sizing signal. A workload that never exceeds 30% engine activity is a workload that should probably be sharing a device, and that recommendation could be generated rather than argued about.
Idle-aware scheduling. Right now the detector reacts. The more useful version influences placement — bin-packing by observed activity rather than by declared requests, which is where the real fleet-level savings are.
If you’re running GPUs on Kubernetes, the cheapest thing you can do this week is add one recording rule and look at gpu:idle:ratio for a few days. I’d be genuinely curious what you find.