llm-d 1.0: Keep Agent Traffic Moving

A fast model does not guarantee a responsive service. One agent can send a series of long requests. Another user may need a short answer now. Both requests reach the same machines, but they do not have the same needs.

On 20 October 2026, the llm-d project announced version 1.0. It is a distributed inference stack that works with model servers such as vLLM and SGLang. It does not replace the model or turn every workload into a faster one. Its purpose is to coordinate the service around those servers. Source 1 Source 3

The important change is operational. Earlier releases already included prefix-aware routing, cache offload, and separate prefill and decode pools. Version 1.0 adds a stable configuration API and stronger controls for shared traffic, upgrades, and agent sessions. Source 1

A queue is part of the service

Consider a shared service with two kinds of work. People ask short, interactive questions. Background agents inspect large codebases. The agents can submit several requests at once.

A simple queue treats arrival order as the main rule. That rule is easy to understand. It cannot express every service promise, however. A background burst can compete with the next interactive request.

In llm-d 1.0, flow control can group requests by tenant or workload and by priority. It maintains separate queues. It selects by priority, then shares turns among tenants in the same priority band. When servers are full, the router can hold requests instead of adding them to overloaded server queues. Source 1

This policy is not automatic. The defaults still resemble first-come, first-served operation. Operators must configure the multi-tenant policy. Priority also cannot create capacity. Teams still need to decide which work can wait, which work can be refused, and what delay is acceptable.

Send context where it can be reused

An agent often returns to the same conversation. Its next request can share a long prefix with the previous request. Sending that request to a server with the relevant cached state can avoid repeated prompt work.

Google describes llm-d as a coordinated pool behind one endpoint. Its router considers cached prefixes as well as load. Cache can also move through different memory tiers, rather than remain only in accelerator memory. Source 2

The useful mental model is a workshop with several benches. A worker benefits from returning to the bench that holds the current job. But a crowded bench is not always the best choice. This analogy describes a trade-off, not an exact scheduling algorithm.

Do not confuse cache reuse with permanent conversation storage. Cache is an execution resource. Application records, access rules, and retention policies still need their own design.

Separate the phases without splitting responsibility

Prefill processes the input prompt. Decode generates the output tokens. These phases place different demands on hardware. A disaggregated deployment assigns them to separate pools so that each pool can scale for its work. Source 2

That separation creates an operational question: how do the pools change version together?

A version change can make the two phases incompatible. DisaggregatedSet labels the components by revision. With the rollout screener configured, routing selects one revision for a request and keeps both phases within it. A new revision receives no traffic until every required role has a Ready pod. Revision-gating guide

This needs explicit setup. The current llm-d rollout guide marks the screener Alpha. It requires allow-experimental-plugins: true, LeaderWorkerSet 0.11.1 or later with DisaggregatedSet enabled, and correct router configuration. Installing the controller alone does not enable revision-aware routing.

Graceful drain also matters. A prefill component can still hold state that an active decode request needs. The guide requires shutdown settings that allow requests and dependent cache transfers to finish. Revision selection does not replace that step.

These controls address a specific coordination risk. They are not a guarantee that every upgrade or failure is safe. Check the documented prerequisites and your serving-engine versions before use.

Our diagram shows one request moving through both phases. It is a conceptual example, not a benchmark. The important detail is continuity: the request must reach compatible components throughout its path.

Choose an operating policy, not a headline number

IBM frames the release around “day 2” operations: keeping a service useful after its first successful deployment. It connects inference engines with the routing and operational controls needed for shared infrastructure. Source 3

For a small team, that does not mean every application needs a distributed cluster. Start with a concrete problem. Is repeated context expensive? Are background jobs delaying users? Are split-pool upgrades difficult to coordinate?

llm-d 1.0 makes these operating decisions more explicit. The gain is not a universal speed promise. It is a clearer way to control how a shared inference service behaves.

Sources

1. llm-d maintainers: version 1.0 announcement, 20 October 2026

2. Google Cloud: llm-d 1.0 distributed inference, 20 October 2026

3. IBM Research: llm-d 1.0 and production operations, 20 October 2026

4. llm-d: safe rollouts of disaggregated deployments, v1.0 guide

5. LeaderWorkerSet: revision gating with llm-d, updated 20 October 2026