4ROUTER / Performance
The anatomy of a KV cache
The state behind generation, and how it uses memory.
Engineering note
What the cache holds
Autoregressive generation repeatedly attends to previous context. A KV cache retains attention keys and values so later steps can reuse earlier work. It is request state, separate from model weights, and consumes device memory as context grows.
Why block allocation matters
Requests vary in length and completion time. Reserving one large contiguous region can waste space. Block-based allocation maps a logically continuous context onto physical blocks. Block size, reclamation and reuse depend on the serving implementation and workload.
Reuse needs boundaries
A shared prefix is not the only condition for safe reuse. Model versions, token inputs, positions and execution settings must be compatible. Multi-tenant systems also need an isolation policy. Evaluate cache behavior alongside memory use and request delay.
Before implementation
- Context-length distribution
- Memory budget and reclamation
- Prefix reuse and isolation
