Cost per task, and why nobody can calculate it

Ask a platform team what a GPU-hour costs and you will get an answer in about four seconds. Ask what one support-ticket summarization costs and you will get a pause, then a spreadsheet, then a caveat.

That gap is the problem. GPU-hours are what you buy. Tasks are what the business consumes. Every conversation about whether inference is worth what it costs happens in the second unit, and almost nobody can produce it reliably.

Why cost per token isn't enough

Cost per token is easy and mostly useless for decisions.

A task is not a token. A single "summarize this thread" might be one call at 4,000 input tokens and 300 output. An agentic workflow answering the same user request might be fourteen calls across three models, two of them retried, one of them a tool-selection step that generates nine tokens. Both are one unit of business value. Their token profiles are not remotely comparable.

The task is the unit your product manager reasons about, so it has to be the unit of cost. Which means the attribution has to survive being aggregated across calls, models and retries.

Where the arithmetic breaks

Five places, roughly in order of how much damage they do.

Input and output tokens do not cost the same

Prefill is compute-bound and parallel. Decode is memory-bandwidth-bound and sequential — one token at a time per sequence, streaming weights and KV cache out of HBM for each one.

The practical consequence: an output token typically costs several times what an input token costs on the same hardware, and the ratio depends on your model, GPU class and batch composition. Any cost model using a single blended per-token rate will systematically misprice workloads. Long-context summarization looks expensive and is cheap. Chatty generation looks cheap and is not.

Shared capacity has to be divided, and the divisor is a policy choice

A GPU serving eight tenants concurrently produces one electricity bill and one depreciation line. Splitting it requires a decision, and the options give materially different answers:

• By token share — attribute in proportion to tokens served. Simple. Penalizes nobody for idle capacity, which means nobody pays for the 3am replica.
• By occupancy share — attribute in proportion to time-weighted GPU memory or SM occupancy. Fairer to bursty workloads, considerably harder to instrument.
• By reserved capacity — a team that demanded a dedicated replica pays for that replica whether it uses it or not. Politically the most honest and the least popular.

There is no correct answer. There is only a stated answer. The failure mode is not choosing wrong, it is never choosing, so each team computes its own and the numbers never reconcile.

Idle time belongs to someone

If your fleet is provisioned for peak and the peak is four hours a day, the other twenty hours are real money. Attribute them to nobody and your per-task costs look great while the invoice does not. Spread them evenly and the small tenant subsidizes the large one. Attribute them to whoever forced the peak and you need capacity-request history, which most teams do not keep.

This single decision moves a per-task figure by more than any tuning work you might do afterwards.

Retries, cancellations and speculation

A request cancelled by the client at 80% completion consumed 80% of the compute and delivered nothing. A retry after a timeout costs twice. Rejected speculative-decode drafts cost real forward passes. Rate-limited requests that were queued then dropped consumed scheduler time and possibly KV cache.

Naive attribution counts completed requests only, which understates true cost — sometimes badly, on a flaky upstream.

Prefix cache hits change the price of identical-looking calls

Two requests with the same token count can differ several-fold in cost depending on whether the prefix was already resident. Long shared system prompts are the common case: the first call pays for the prefill, the next thousand mostly do not.

This is good news operationally and awkward for accounting. Cost per task becomes dependent on traffic shape and ordering, not just request content. Teams that share a system prompt are effectively cross-subsidizing each other, and if you cannot see cache hit rates per tenant you cannot explain why one team's identical workload costs less this month.

A cost model that survives scrutiny

Nothing here needs new infrastructure. It needs the four inputs kept together and reconciled against the actual invoice.

cost_per_task =
    Σ over calls in task (
        input_tokens  × cost_per_input_token(model, gpu_class)
      + output_tokens × cost_per_output_token(model, gpu_class)
      + wasted_tokens                                          # retries, cancellations, rejected drafts
    )
  + idle_allocation(team, period)                              # per your stated policy

To make it real:

1. Establish per-token costs empirically, per model and GPU class. Benchmark the deployment to find achievable throughput at your latency target, then divide the GPU-hour cost by it. Do not take a vendor's tokens-per-second figure — it was measured on a different batch composition than yours.
2. Carry a task identifier through every call. This is the single highest-leverage change, and it is a header. Without it, multi-call agentic work cannot be reassembled, and agentic traffic is where cost surprises now originate.
3. Record outcomes, not just completions. Cancelled, retried, rate-limited, rejected. Otherwise waste is invisible by construction.
4. State the idle policy in writing and apply it consistently, even if people dislike it.
5. Reconcile monthly against the actual bill. If the modelled total and the invoice disagree by more than a few percent, the model is wrong and you now know to hunt for the gap.

The organizational half of the problem

Everything above is tractable engineering. The reason it usually does not happen is that cost per task has no natural owner.

Finance owns the invoice and cannot see tokens. The platform team owns the tokens and is not accountable for the invoice. Application teams own the workload and see neither. Each group holds one third of the calculation, and no individual is measured on producing it.

So it becomes a quarterly spreadsheet exercise instead of a number on a dashboard — which means it arrives too late to change a routing decision, a replica count, or a model choice while any of those are still cheap to change.

The fix is boring and structural: pick the attribution policy, instrument the task identifier, and put the resulting number somewhere both the platform team and finance look at the same time. The arithmetic is the easy part.

Schedule your 30-minute Inference Expert Audit. We measure your real cost per task, attributed to model, team and deployment choice, using the metrics you already emit.