4ROUTER / Serverless
Designing for cold starts
Understanding the path from launch to readiness.
Engineering note
Startup has multiple stages
From placement to a usable endpoint, startup may include image preparation, process launch, weight reads, device loading and warmup. Recording stages separately reveals where the wait occurs. Process creation is not a substitute for readiness.
Caches and warm capacity have costs
Local weight caches reduce repeated downloads but need capacity and version management. Warm replicas reduce startup waits while retaining resources. Choose a policy against request frequency, model changes and acceptable waiting time.
Make failures recoverable
Corrupt weights, load failures and resource shortages need distinct errors and recovery paths. Do not admit traffic before readiness. During scaling and rolling updates, inspect what callers experience while waiting, not just the eventual successful state.
Before implementation
- Startup-stage records
- Weight-cache versions
- Readiness and retry policy
