Inference routerp99 latency
−43%
lower p99 latency from routing repeat prompts to the replica that already holds their KV cache.
p99 latency, indexed · before = 100
router.cpp · C++20
79%cache hit rate
−43%p99 latency
<0.9 msrouting cost
50,000 / 50,000generations token-identical
load test
⚡ replica failure injected
0 divergence
01 · The problem
Round-robin wastes the GPU’s memory.
LLM serving engines like vLLM cache the attention state (the KV cache) for prompts they have already seen. A load balancer that ignores this spreads repeat prompts across every replica, so each one recomputes the same prefix from scratch.
02 · The idea
Route by prefix, on the hot path.
I wrote a C++20 router that keeps a prefix tree of which replica holds which prompt prefix, and pins each request to the replica that already has its KV cache. Matching costs under 0.9 ms per request, behind an OpenAI-compatible API.
03 · The result
−43% p99 latency. 79% cache hits.
Tail latency drops because repeat prompts skip recomputation. The fleet-wide cache hit rate rose to 79%, and p99 latency fell 43%.
04 · The proof
Faster, and provably the same output.
A router must never change what the model says. I checked 50K generations token-for-token against direct-to-engine output, under load and with an injected replica failure. Zero divergence.
C++20vLLMOpenAI-compatible APIDockerKubernetes
Read the code on GitHub ↗