Systems
4ROUTER / ENGINEERING
Behind every request.
Notes on the systems, performance and infrastructure behind inference.
Mechanisms and experiment design ↗cuBLAS · cuDNN · PagedAttention · RadixAttention · LMCache
All stories
Performance
Tuning the inference path
Serverless
Making inference serverless
Routing
Inside our routing engine
Systems
Scheduling across GPUs
Performance
The anatomy of a KV cache
Performance
Continuous batching, explained
Serverless
Designing for cold starts
Serverless
Scaling with the workload
Routing
When a route needs a fallback
Observability
Following a request end to end
Reliability