Google Cloud has released llm-d 1.0, the first stable version of its open-source, Kubernetes-native stack for distributed LLM inference. Announced at the PyTorch Conference in San Jose on October 20, the release adds production-grade controls, multi-tenant serving, zero-downtime upgrades and deeper TPU integration, positioning llm-d as a vendor-neutral alternative for organizations that want to run their own models instead of renting inference from a single provider.
The project, a Cloud Native Computing Foundation (CNCF) Sandbox effort, is co-maintained by Google, Red Hat and IBM, which co-founded it last year with CoreWeave and NVIDIA. More than 500 contributors across 25 organizations, including AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI, have joined the effort.
What Was Announced?
llm-d sits between the two communities its name bridges: PyTorch, where serving engines like vLLM and SGLang run models, and Kubernetes, where distributed systems scale in production. It does not replace those engines. Instead it turns a cluster of model servers into a coordinated serving pool with one router, one KV cache index and one flow-control policy in front of the whole fleet.
Version 1.0 is about operational maturity rather than new research tricks. Google says the five core techniques, cache-aware routing, tiered KV cache offloading, request batching, prefill/decode disaggregation and wide expert parallelism, were already in the project. What is new is that they now fit together with the controls enterprises need to run inference as a critical service.
How llm-d 1.0 evolves distributed inference, from generic load balancing to cache-aware, multi-tenant production serving. Image: Google Cloud
What’s New in 1.0
- Multi-tenant quality of service. Priority and fairness-aware routing via llm-d flow control lets latency-sensitive requests and bursty agent workloads share a fleet without starving each other, so paid tiers can hold their latency targets.
- A stable foundation. v1 APIs and a hardened request path, validated on multiple clouds and accelerator types.
- Safer upgrades. Router high availability and zero-downtime updates, plus DisaggregatedSet, which orchestrates prefill/decode-aware rolling updates without dropping context or disrupting sessions.
- Agent-aware serving. Session-aware routing, turn-priority scheduling and native Responses API state help agents reuse context across turns.
- Expanded TPU integration. Native PyTorch model serving with TorchTPU co-operative time-slicing to raise throughput across Google Cloud TPU fleets.
The release also introduces an async inference processor that pulls requests from queues like Pub/Sub or Redis and routes them alongside real-time traffic, plus TPU and GPU time-slicing that interleaves async inference into the idle cycles of reinforcement learning training, lifting aggregate accelerator duty cycles from around 40% to more than 70%, according to Google.
Benchmarks and Production Proof
The llm-d 1.0 stack on Google Cloud: multi-node orchestration over single-node model engines on TPUs and GPUs. Image: Google Cloud
Google backed the release with measured results. On an agentic coding workload on Ironwood TPU, prefix-aware routing plus host memory offload cut median time-to-first-token by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026. In a published benchmark across a 17,000-node multi-region GKE deployment, the llm-d router scaled throughput 2.9x across three clusters with a 99.9% success rate and less than 1% routing overhead. And on an A3 Mega cluster serving a 70B-parameter model, multi-connector KV cache offloading delivered roughly 40% better time-to-first-token and 30% lower end-to-end latency on 50,000-token prompts while cutting GPU-hour requirements by nearly 60%.
Production users are already on record. “At Snap, we are integrating llm-d into our production AI infrastructure to facilitate high-performance inference at scale. By employing prefix-cache-aware routing, we have achieved prefix cache hit rates ranging up to 75-80%,” said Vinay Kola, Senior Manager of Software Engineering at Snap. Google also cites the Gemini platform, where GKE Inference Gateway doubled prefix cache hit rates from 35% to 70%, and Zenteiq.ai, which reported over 4x higher aggregate inference throughput for synthetic data generation.
“At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage,” said Ruchir Puri, IBM Research chief scientist, in comments reported by SiliconANGLE. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity.”
Carlos Costa, Distinguished Engineer at IBM Research and an llm-d co-founder and lead maintainer, said version 1.0 adds the maturity to run llm-d like any other critical service, pointing to a stable API, multiple tenants with competing priorities sharing the same infrastructure, and upgrades that do not interrupt operations, as reported by Unite.AI.
Why It Matters
The timing is deliberate. Google argues inference workloads have shifted: a year ago most traffic was chat, one prompt in and one response out. Today, multi-step agents and reinforcement learning rollouts fan out across tools, share long prefixes and arrive in bursts, and serving them on a single machine or behind a generic round-robin load balancer breaks down. During the PyTorch Conference keynote, Google demonstrated llm-d handling 1,000 sandboxed agents hitting a single endpoint across vLLM, SGLang and a mixed fleet of TPUs and GPUs.
For enterprises, the pitch is control: run any model, on any accelerator, in any cloud, without depending entirely on one proprietary provider. The project also ships Prism, an open-source dashboard for exploring published benchmarks with standardized recipes, so teams can inspect the exact configurations behind the numbers or upload their own runs.
