4ROUTER / Performance
Continuous batching, explained
Coordinating generation as requests arrive and finish.
Engineering note
Why a batch changes
Online requests do not start or finish together. Continuous batching lets a serving scheduler admit new sequences and retire completed ones as generation progresses. This reuses available capacity, while throughput and latency still depend on workload and scheduling choices.
Prefill and decode share resources
Long-prompt processing and token-by-token generation have different compute profiles. Sharing an execution queue can make long inputs affect active generations. Policies such as chunked prefill change this interaction and should be tested against interactive workloads.
Bound the batch to protect latency
More simultaneous work does not necessarily improve the user experience. Capacity pressure can lengthen queues and reduce cache headroom. Observe queue delay, first-token time, generation intervals and failures together, and preserve cancellation and timeout handling.
Before implementation
- Arrival pattern
- Mixed prompt lengths
- Batch limits and waiting targets
