Technical6 min28 July 2026

What effective GPU utilization actually measures

If you run nvidia-smi against a busy inference node and see GPU-Util: 94%, the honest interpretation is not "this GPU is 94% busy." It is closer to "in 94% of the sampling intervals, at least one kernel was resident on the device."

Those are very different statements, and the gap between them is where most self-hosted inference budgets quietly go.

Three different numbers, all called "utilization"

Device utilization is what nvidia-smi reports as GPU-Util, sampled from NVML. It answers one question: was any kernel executing during the sample window? A single small kernel occupying two streaming multiprocessors out of 132 registers exactly the same as a fully saturated matmul. For LLM inference — which alternates between a compute-bound prefill and a memory-bandwidth-bound decode — this metric is close to useless as a capacity signal. A decode-heavy workload can pin GPU-Util near 100% while the SMs idle on memory reads.

Achieved occupancy and memory bandwidth utilization get closer to the hardware truth. During decode, most LLM serving is bandwidth-bound: you are streaming weights and KV cache out of HBM, and the arithmetic units wait. If you are reading 80% of theoretical HBM bandwidth, you are genuinely near the hardware ceiling for that phase, regardless of what GPU-Util says.

Effective utilization is the one that governs the bill. It asks: of the GPU-hours you paid for, what fraction produced tokens that a user or an agent actually consumed? That framing folds in everything the device-level metrics cannot see.

What effective utilization has to account for

A GPU-hour can fail to become useful work in at least five ways, and only one of them is visible on the device:

Idle capacity. The cluster is sized for peak and the peak is four hours a day. Nights and weekends are paid for at full rate.
Over-provisioned replicas. Four replicas of a model that needs one and a half, because the autoscaler is conservative and nobody has revisited the floor since launch.
Running-but-unreachable deployments. A model is up, healthy, consuming memory, and not registered anywhere a caller can reach it. This is more common than it sounds, particularly after a migration.
Requests on the wrong tier. A classification call that a 3B model handles fine is being served by a 70B, because the routing rule was written when there was only one model.
Speculative and cancelled work. Tokens generated for requests that were abandoned client-side, or drafted by a speculative decoder and rejected.

None of these are serving-engine faults. vLLM is not doing anything wrong when it faithfully serves a 70B model to a request that didn't need one.

The published numbers are worse than most teams guess

External measurements of enterprise GPU estates are consistently grim. Cast AI's 2026 State of Kubernetes Optimization Report analysed roughly 23,000 clusters across AWS, Azure and GCP and found average GPU utilization of about 5%. Broader surveys put most enterprise clusters under 10%, with roughly two-thirds of organizations reporting peak utilization below 70%.

Two caveats, because they matter. First, those figures are cluster-wide and Kubernetes-wide — they include training, batch and experimentation, not just production inference. They are consistent with poor inference utilization; they do not prove it. Second, inference-specific numbers depend enormously on batching strategy: static batching leaves utilization in the 15-30% range, while continuous batching moves it dramatically higher.

That second point deserves emphasis, because it is frequently misattributed.

Continuous batching is vLLM's win, not a control plane's

The single largest utilization improvement available to most self-hosted inference stacks is continuous batching with paged attention — and if you are running vLLM or SGLang, you already have it. PagedAttention is the reason vLLM exists. Any vendor claiming credit for moving your utilization from the twenties into the sixties is describing your serving engine's headline feature.

We think this distinction is worth being pedantic about. It separates two genuinely different problems:

Making one deployment efficient. Batching, quantization, KV-cache management, tensor parallelism, prefix caching. This is the serving engine's job, and the open-source engines are very good at it now.
Making a fleet efficient. Which models exist, how many replicas each has, who is allowed to call them, which tier a given request lands on, and what happens to capacity at 3am. This is a scheduling and governance problem that sits above the engine, and it is where most of the remaining waste lives once your serving layer is well configured.

A team with a beautifully tuned vLLM deployment can still be at 25% effective utilization across the fleet, because eleven models are running and three of them serve nothing.

How to measure it without buying anything

You can get a defensible effective-utilization number from telemetry you probably already emit:

effective_utilization ≈
    (tokens_served_to_completed_requests × cost_per_token_at_hw_ceiling)
  ÷ (provisioned_gpu_hours × cost_per_gpu_hour)

In practice that means collecting four things per model, per team, over a full weekly cycle:

Provisioned GPU-hours, from your scheduler — not from the device.
Completed-request token counts, input and output separately, from the serving engine's metrics endpoint.
A hardware ceiling for that model and GPU class, established once with a benchmark sweep rather than assumed. GuideLLM is the usual tool.
Request outcomes, so cancelled and rejected work is excluded from the numerator.

Cover a full week, including the weekend. Utilization measured over a Tuesday afternoon is a measurement of your Tuesday afternoon.

The number nobody likes

The uncomfortable part of effective utilization is that it is a business metric wearing engineering clothes. It divides tokens people actually used by money you actually spent, which means it implicates capacity planning, routing policy, model proliferation and organizational ownership all at once — not just the serving configuration.

Which is usually why it goes unmeasured. It is nobody's single responsibility, and the first honest number is rarely flattering.

It is still the right number to start with. If you cannot state your effective utilization per team and per application, you cannot tell whether your next GPU purchase is capacity you need or waste you are about to duplicate.

Schedule your 30-minute Inference Expert Audit. We measure effective GPU utilization across your fleet, attributed per app and per team, on the stack you already run — and we will not quote you a savings figure before we have looked.