Inference has no owner

Here is a test you can run on your own organization this week. Latency on your primary copilot degrades 40% between 09:00 and 09:20 on a Tuesday. Nobody deployed anything. Ask the question: whose incident is this?

In most organizations running self-hosted models, four teams investigate and none of them can close it alone. Infrastructure confirms the GPUs are healthy. The ML team confirms the model is unchanged. The platform team confirms the serving config is unchanged. The application team confirms their traffic is normal — for them. Somebody eventually notices that a different team shipped a new agent on Monday afternoon that is issuing nine times the calls per user action, on the same tier.

Every individual answer was correct. The system-level answer took four teams and two hours.

The gap is structural, not a staffing problem

Inference sits at a genuinely awkward intersection. It is not one of the three things organizations already know how to own.

Infrastructure owns hardware, capacity and the bill. It reasons in nodes, GPU-hours and quarterly procurement. It cannot see which prompt caused the spike.

ML and data science own model selection, evaluation and quality. They reason in benchmarks, accuracy and drift. They are not usually on the pager at 03:00, and were not hired to be.

Application teams own the product surface and the user experience. They reason in features and latency budgets. They treat inference as an API that should simply work.

Inference is the layer that turns owned hardware into consumed tokens. It requires simultaneous authority over routing, capacity, quotas, SLOs, model lifecycle and cost attribution — a set that cuts across all three org charts and belongs cleanly to none.

The result is not that nobody cares. It is that everybody cares about their own slice, and no single person is accountable for the SLO or the invoice together.

This is a new problem, and it is about to get worse

Two shifts make the ownership gap urgent rather than merely untidy.

The first is the workload mix. McKinsey's projection is that by 2030 inference will surpass training as the dominant AI workload. Training is a project: it has a start, an end, an owner and a budget line. Inference is an operation — permanent, load-bearing, and behaving much more like a production database than like a research run. Organizations built the AI org chart around the project shape and are now running the operational shape through it.

The second is self-hosting. Roughly half of enterprise inference spend now goes to open-source or custom models rather than commercial APIs, on McKinsey's numbers. When you call a hosted API, most of these problems are somebody else's: they own the fleet, the routing, the utilization. The moment the GPUs are yours, all of it becomes yours too — and the ownership question can no longer be deferred to a vendor's status page.

There is a third data point worth sitting with. Gartner has found that at least half of generative AI projects are abandoned after proof of concept. Not all of that is operational, plainly. But the distance between "works in a notebook" and "runs reliably at cost inside an SLO" is exactly the distance this unowned layer is supposed to cover. A pilot that cannot answer "what will this cost per task at 100x volume, and who keeps it inside its latency budget?" tends not to survive its own business case.

Why the obvious fixes underdeliver

"Give it to the platform team." Reasonable, and often right — but authority has to come with it. A platform team that can observe traffic and cannot enforce a quota, or can enforce a quota and cannot see cost per team, has been handed the pager without the controls. The pattern that fails most often is responsibility without enforcement.

"The serving engine handles it." vLLM, SGLang and their peers are excellent at making a deployment efficient — continuous batching, paged attention, prefix caching. That is real engineering and it solved a hard problem. But no serving engine decides that a classification call should not be hitting your 70B, or that eleven models are running when six would do, or which tenant's traffic to shed when capacity tightens. Those are fleet-level policy questions, deliberately outside the engine's scope.

"We'll build it internally." Many teams do, and the first 60% is genuinely straightforward: a dashboard, a gateway, some quota logic. The expensive part is the long tail — cost attribution that reconciles with the invoice, SLO enforcement that degrades gracefully, alerts that clear themselves instead of training everyone to ignore them, and keeping all of it working as serving frameworks change every few months. It is a product, and it competes for headcount with the thing your company actually sells.

"Add more GPUs." Occasionally correct. But buying capacity to solve a utilization problem duplicates the waste at a larger scale, and the reason it is chosen so often is that it requires no organizational change — which is precisely the tell.

What good looks like

The organizations handling this well have not necessarily hired anyone new. They have made three things explicit:

1. A named owner for inference operations — one person accountable for both the SLO and the cost per task, not one for each. This is the whole thing, and it is an org decision, not a tooling one.
2. Attribution before optimization. Utilization and cost per task, broken out by app, team and model, before anyone tunes anything. You cannot govern a shared resource whose consumption you cannot attribute, and the first honest number is usually the thing that gets the org decision made.
3. Enforcement that actually binds. Quotas, rate limits and SLO tiers that change system behaviour under load rather than merely labelling intent. A quota nobody enforces is a comment.

None of that requires replacing your serving stack. It requires deciding that the layer above it is somebody's job.

The alternative is the Tuesday morning incident — four correct answers, two hours, and no owner.

Schedule your 30-minute Inference Expert Audit. We measure utilization and cost per task attributed by app and team, which is usually the number that settles who should own inference operations.