4ROUTER / Serverless
Scaling with the workload
Matching serving capacity to the shape of demand.
Engineering note
Choose a useful signal
GPU utilization describes device activity, not the caller’s wait. Queue length, delay, active requests and generation pace offer different views. A scaling policy should state which signals it uses and why they represent the workload.
Scaling has a lead time
Collection, policy execution, placement and model loading all take time. Scaling after a traffic spike may not relieve the queue already formed. Capacity planning combines warm resources, queue limits and explicit overload responses.
Drain before scaling down
Stop assigning new work before removing a replica and define how active requests finish. Long generations affect shutdown time. Repeated scaling can waste loading work and destabilize service, so policies need stabilization and explicit minimum capacity.
Before implementation
- Workload signals
- Startup lead time and headroom
- Draining and stabilization
