4ROUTER / Performance
Tuning the inference path
Inside the work of balancing latency and throughput.
Engineering note
Separate the waiting time
A request can wait at admission, prompt processing, token generation or transport. Total duration alone does not identify the bottleneck. Record arrival, execution start, first token and completion, then interpret these stages alongside prompt and output lengths.
Keep the workload comparable
Keep model, hardware, request samples and arrival patterns fixed when comparing settings. Aggregate throughput can hide growing queues, while isolated latency says little about concurrency. Define acceptable waits for interactive and batch work before choosing which improvement matters.
Validate one change at a time
Batch limits, cache policy, precision and kernels change resource use in different ways. Establish a baseline and record each change against the same workload. When precision or kernels change, check task quality and numerical behavior as well as speed.
Before implementation
- Stage-level latency records
- Concurrency and arrival patterns
- Performance and task-quality checks
