4ROUTER / Systems
Scheduling across GPUs
Placing work where the right resources are available.
Engineering note
A resource count is a starting point
GPU count alone does not determine placement. Memory, accelerator type, interconnect, cached models and competing workloads also matter. Explicit scheduling constraints are easier to maintain than relying on someone to remember each machine.
Choose the isolation unit
Kubernetes exposes GPUs through device plugins and resource requests. The serving runtime still manages concurrency and memory inside the application. Node placement and request scheduling operate at different layers and need separate observations.
Make waiting understandable
When a job is waiting, distinguish insufficient resources, unmatched constraints, loading and node failure. Explicit queue and cancellation policies help prevent duplicate submissions. Node maintenance should state whether work migrates, restarts or waits.
Before implementation
- GPU type and memory
- Placement and queue policy
- Maintenance and cancellation
