GKE Inference Gateway: KV-Cache-Aware LLM Routing Explained

Round-robin load balancing works fine for stateless services. For LLM inference, it's actively harmful. Each vLLM replica holds a different KV cache state, and routing to the wrong replica means throwing away computed work and re-running prefill from scratch. Google's GKE Inference Gateway, announced at Google Cloud Next 2025, addresses this directly with cache-aware routing powered by the llm-d Endpoint Picker (EPP). The interesting part: the routing logic underneath is open source, built on the Kubernetes Gateway API Inference Extension, and runs on any cluster. For teams already running llm-d for disaggregated inference on Kubernetes, the EPP is the same component that drives cache-aware request dispatch in that stack too.

What the GKE Inference Gateway Actually Does

GKE Inference Gateway is a managed Kubernetes gateway with inference-aware routing built in. You point it at a pool of vLLM replicas, and it takes over request dispatch from a standard load balancer.

Three things make it different from a dumb L7 proxy:

Model-aware routing: The gateway reads the model field from every incoming /v1/chat/completions request and routes it to the correct backend pool. Multiple models behind a single endpoint, no separate Services per model.

Prefix cache routing: Before forwarding a request, the Endpoint Picker hashes the incoming token prefix and checks which replica most likely holds that prefix in its KV cache. Requests with matching prefixes go to the replica that already computed them.

KV cache utilization balancing: When no replica has a strong prefix match, the EPP falls back to routing based on vllm:kv_cache_usage_perc, sending new requests to the replica with the most available cache capacity.

Here is what the architecture looks like:

Client
  |
  v
GKE Inference Gateway (Envoy + EPP plugin)
  |
  v
Endpoint Picker (EPP) -- scrapes /metrics from each replica
  |         |          |
  v         v          v
vLLM-0   vLLM-1   vLLM-2
(KV cache A) (KV cache B) (KV cache C)

The GKE layer manages the Envoy proxy, the load balancer provisioning, and the Kubernetes control plane. The EPP is where the routing intelligence lives. It is a Go binary that plugs into the gateway as an extension filter.

Crucially, the EPP is not proprietary. It ships in the open-source gateway-api-inference-extension repository under kubernetes-sigs. The GCP-specific parts are the managed control plane, the Google Cloud Load Balancer provisioning, and the IAM integration. The routing logic, the CRDs, and the vLLM integration are identical whether you run on GKE or on your own Kubernetes cluster.

Why Round-Robin Fails for LLM Serving

A stateless web server is a good fit for round-robin because each replica has identical capacity and no local state that makes one replica better than another for a specific request. LLM serving breaks both assumptions.

Each vLLM replica maintains its own KV cache. When a request comes in with a long system prompt, the first replica to process it computes the full KV cache for that prompt and stores it. The next identical request, if routed to the same replica, skips that computation entirely via prefix caching. But with round-robin across 8 replicas, the same system prompt gets distributed across all 8 replicas, and each replica sees the prefix only about 12.5% of the time. Cache hit rate per replica is near zero.

The math on wasted compute is ugly. A 128K-token system prompt on a 70B FP8 model takes roughly 11 seconds of prefill on a multi-GPU setup capable of holding the full context (the model weights alone are ~70GB, so running 128K-token contexts requires tensor parallelism across multiple H100s). With round-robin across 8 replicas, almost every request hits that full prefill. With KV-aware routing, the first request per replica pays the full cost, and every subsequent request with the same prefix hits cache in under 2 seconds. On a chatbot handling 1,000 requests per hour with a shared system prompt, that's the difference between 11,000 GPU-seconds wasted on redundant prefill versus under 2,000.

This scales worse as you add replicas. More replicas means lower per-replica cache hit rate with round-robin. KV-aware routing gets more efficient as you scale because it has more cache capacity to route into, not less.

Here is a concrete before/after comparison on a chatbot workload with a 4K-token system prompt, 8 vLLM replicas (illustrative estimates based on modeled workload behavior):

The TTFT improvement compounds at longer contexts and higher concurrency. For RAG pipelines with document-length system prompts, the difference is even larger.

For a deeper explanation of why KV cache state matters and how PagedAttention allocates cache blocks, see the KV cache optimization guide.

Under the Hood: the llm-d Endpoint Picker and Gateway API Inference Extension

The routing logic lives in two components: the Gateway API Inference Extension CRDs and the Endpoint Picker binary.

The Endpoint Picker

The EPP runs as a Kubernetes Deployment alongside the gateway. It maintains a background scrape loop that polls each registered vLLM replica's Prometheus /metrics endpoint every 100ms. From these metrics it reads:

• vllm:kv_cache_usage_perc: fraction of KV cache blocks currently occupied
• Queue depth and running request counts for load pressure signals

Prefix-cache locality is tracked by the EPP itself, not reported by replicas. The EPP maintains an in-memory prefix map that hashes each request's token prefix and records which replica last computed it.

On each incoming request, the EPP:

1. Hashes the token prefix of the request (system prompt + conversation history up to this turn)
2. Looks up which replicas have seen this prefix hash before
3. Among those replicas, selects the one with available KV cache capacity (low vllm:kv_cache_usage_perc)
4. If no replica has a prefix match, falls back to the replica with the lowest vllm:kv_cache_usage_perc
5. Forwards the request to the selected replica and returns the response

The whole decision takes under 2ms of added latency. The EPP is stateful: it keeps an in-memory prefix map that tracks which replicas have computed which prefixes. When a replica restarts and clears its KV cache, the EPP's prefix map ages out the stale prefix entries for that replica and stops routing those prefixes there until the cache warms up again.

The Gateway API Inference Extension CRDs

The Inference Extension adds a custom resource type to standard Kubernetes Gateway API:

InferencePool: Groups a set of backend pods into a named pool. Points to a pod selector and port. The EPP uses this to know which pods to poll and route to.

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: llama-70b-pool
spec:
  selector:
    matchLabels:
      app: llama-70b-worker
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: llama-70b-epp

The InferenceModel CRD was replaced by InferenceObjective when the project graduated to inference.networking.k8s.io/v1. InferenceObjective carries request priority and criticality settings that the EPP uses during scheduling. Model-name routing in the current API is handled through HTTPRoute rules that reference the InferencePool directly by name. For multi-model clusters, create one InferencePool per model and use separate HTTPRoutes (or header-based routing rules) to direct traffic to the correct pool.

For broader context on model routing architectures and how this fits into a tiered serving strategy, see the LLM inference router guide.

The Lock-In Question: What Is GKE-Specific vs Portable

Before committing to GKE Inference Gateway, it is worth being precise about which parts are Google-specific and which are not.

The routing intelligence, the CRDs, and the vLLM integration are fully portable. You pay GCP for the managed control plane and the load balancer provisioning. If you already operate Kubernetes clusters and want cache-aware inference routing without a managed service, you can deploy the exact same EPP binary and the same CRDs from the kubernetes-sigs/gateway-api-inference-extension repository.

The egress cost is worth calling out. At 1 TB/month of outbound traffic, GCP charges $80-120 in egress fees. Bare-metal GPU clusters do not carry this per-GB markup. For high-throughput inference serving, egress costs compound quickly.

Replicating the Stack on Any GPU Cloud

Running the same open-source stack outside GCP takes about 45 minutes end to end. Here is the deployment path on Spheron bare-metal GPUs.

Step 1: Provision GPU Nodes

For a 70B model at FP8, you need at least two H100 SXM5 nodes for the vLLM decode pool plus one smaller node for the Gateway and EPP process. The EPP is a lightweight Go binary and runs fine on a CPU node or a smaller GPU.

Deploy your H100 nodes from Spheron with per-minute billing. For 13B-40B models, A100 80GB instances work well and cost less. SSH into each node and verify GPU access:

nvidia-smi
# Should show your H100 SXM5 or A100 devices

For the vLLM baseline setup, see the Spheron vLLM quick guide.

Step 2: Install the Gateway API CRDs

On your Kubernetes cluster (you can use k3s or kubeadm on the provisioned nodes):

# Gateway API CRDs (v1.3.0+ required)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml

# Gateway API Inference Extension CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/latest/download/manifests.yaml

This installs the InferencePool CRD (inference.networking.k8s.io/v1) into your cluster.

Step 3: Deploy vLLM with Prefix Caching

On each GPU node, start vLLM with the options the EPP needs:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --dtype fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --port 8000

The --enable-prefix-caching flag is what makes the EPP routing effective. Without it, the KV cache does not persist prefix blocks between requests, so routing to a specific replica gives no benefit. The EPP scrapes each replica's standard Prometheus /metrics endpoint (exposed on the same port as the vLLM API server) to read vllm:kv_cache_usage_perc and queue-depth counters.

Step 4: Deploy the EPP

Clone the Inference Extension repo and apply the EPP deployment:

git clone https://github.com/kubernetes-sigs/gateway-api-inference-extension.git
cd gateway-api-inference-extension

# Deploy the EPP
kubectl apply -f config/default/

Step 5: Apply the InferencePool

# inferencepool.yaml
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: llama-70b-pool
  namespace: default
spec:
  selector:
    matchLabels:
      app: llama-70b-worker
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: endpoint-picker

kubectl apply -f inferencepool.yaml

Step 6: Wire Up Envoy Gateway

Install Envoy Gateway (simpler than Istio for this use case):

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  -n envoy-gateway-system \
  --create-namespace

Create the Gateway and HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
  namespace: default
spec:
  gatewayClassName: envoy
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
  namespace: default
spec:
  parentRefs:
  - name: inference-gateway
  rules:
  - backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: llama-70b-pool
      port: 8000

Test it:

curl http://<GATEWAY_IP>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'

Cost Comparison: Managed GKE Gateway vs Self-Hosted on a GPU Marketplace

Here is a concrete side-by-side for a two-replica 70B serving stack.

Spheron bare-metal H100 SXM5 configuration:

• 2x H100 SXM5 nodes for vLLM: 2 x $3.17/hr = $6.34/hr
• 1 CPU node for EPP + Envoy Gateway: ~$0.10/hr
• Total: ~$6.44/hr
• Egress cost: $0 markup on bare-metal

Equivalent GKE configuration:

• 2x a3-highgpu-8g nodes (H100 equivalent): ~2 x $3.20/hr = $6.40/hr (approximate)
• GKE managed control plane: $0.10/hr per cluster
• Google Cloud Load Balancer: $0.025/hr + per-rule charges
• Egress at 1 TB/mo: ~$80-120 one-time
• Total: ~$6.50/hr plus egress

The compute gap closes if you factor in GKE's operational overhead savings (managed node upgrades, built-in monitoring). The egress gap is harder to close. At 1 TB/month outbound, GCP charges roughly $100 that you do not pay on bare-metal.

The EPP is a lightweight process. It does not justify a managed service tier on its own. The value of GKE Inference Gateway is the managed control plane, not the EPP binary.

Pricing fluctuates based on GPU availability. The prices above are based on 22 Jun 2026 and may have changed. Check current GPU pricing for live rates.

Benchmarking Cache-Aware vs Round-Robin

The performance gap between round-robin and KV-aware routing depends heavily on workload shape. Here is a concrete scenario: 1,000 requests on an 8-replica vLLM cluster, 4K-token system prompt shared across all requests (a typical chatbot or RAG pattern).

Note: The numbers below are illustrative estimates based on modeled workload behavior, not production-measured benchmarks. Actual results vary by model size, context length, hardware configuration, and traffic distribution.

Setup:

• 8x vLLM replicas, each on an H100 SXM5
• Model: Llama 3.1 70B, FP8
• System prompt: 4,096 tokens (about 3,000 words)
• User messages: random, 50-200 tokens each

Results:

The throughput gain comes from freeing prefill capacity. With round-robin, most of the GPU time on each replica goes toward recomputing the same 4K-token system prompt. With KV-aware routing, each replica computes the system prompt once and cache hits make up the rest. That freed GPU capacity goes toward decode, which is what the user waits for.

The EPP adds 1.4ms per request on average, which is negligible against the 11s prefill baseline.

For teams wanting to extend further than single-node prefix caching, LMCache for distributed KV cache sharing adds a Redis-backed shared cache tier so any replica can serve prefixes computed on any other replica, removing the constraint that the routing must always hit the same node.

If you want prefix caching and KV-aware routing without GCP lock-in or egress bills, the same open-source EPP stack runs on Spheron bare-metal GPUs at a fraction of the managed cost. Deploy on H100, A100, or L40S with per-minute billing and no contracts.

H100 on Spheron → | View all GPU pricing → | Get started →