4ROUTER / Systems
Building an inference cluster
From individual GPUs to a coordinated serving system.
Engineering note
Start with the workload
Plan around models and requests. Weights, context length, active sequences and output length all affect memory demand. Interactive applications care about time to the first token and the pace of generation. Offline jobs care about finishing the workload. A shared pool needs an explicit policy for both.
Choose the parallelism boundary
A model that fits on one GPU gives a useful single-replica baseline. Splitting a model introduces communication and synchronization. Tensor and pipeline parallelism address model placement, while additional replicas address request capacity. Test these choices on the interconnect you will actually deploy.
Make the operating boundary explicit
Keep model and runtime versions, launch settings, health probes and rollback instructions together. Acceptance should cover long prompts, cancellation, node loss and insufficient capacity. Document which alerts need intervention and what files and permissions are required to replace a replica.
Before implementation
- Model and prompt/output distributions
- GPU memory and node interconnect
- Workload isolation and failure handling
