4ROUTER / Serverless
Making inference serverless
How workloads become services that scale on demand.
Engineering note
Define the service contract
Callers need defined inputs, streaming behavior, timeouts and errors. Operators need model loading, concurrency limits, health states and shutdown rules. A stable contract connects these boundaries while the number of running replicas changes.
Separate alive from ready
A running process may still be downloading weights, loading the device or warming execution. Readiness should reflect whether new work can be accepted. During shutdown, stop admission before draining or explicitly terminating in-flight work.
Allow time for capacity to arrive
Queue delay, active requests and spare resources can inform scaling, but new replicas take time to become ready. Warm capacity trades resource cost for shorter waits. Describe the boundary of a deployment instead of equating on-demand capacity with instant availability.
Before implementation
- Request and error contract
- Readiness and shutdown behavior
- Cold-start and warm-capacity policy
