Services
Inference
Make models callable, measurable and maintainable. From a multi-provider gateway to dedicated inference endpoints, design access, capacity, budgets and failure handling around real workloads.
Engineering field guide
From requests to a service you can operate.
Start with the request contract, then validate quality, capacity and failure behavior. Gateway integration and dedicated serving follow different implementation paths.
Choose your starting point
Bring the bottleneck first; choose the configuration after it.
- An application with multiple providers
- Use the compatible endpoint to align credentials, candidate scope, budgets and request records. Test the streaming, tools and structured output your client actually uses.
- An open model that needs its own endpoint
- Plan memory and serving processes around weights, context distribution, precision and concurrency. Accept the serving deployment and tenant gateway separately.
- Stable quality, unpredictable cost or tail latency
- Keep a pinned model as the baseline and replay representative requests. Locate queueing, prefill, generation or provider variation before changing scheduling or routing.
What a request passes through
Each layer needs a success condition and a failure path. A successful status code alone does not make the result useful.
Access & constraints
Agree identity, input format, context limits and client timeouts. Routed requests can narrow the pool with allow / deny; pinned requests provide a controlled baseline.
Execution & streaming
Record the selected model, first content event and completion or cancellation reason. Define behavior for client disconnects, upstream timeouts and repeated submissions.
Accounting & replay
Join request identifiers, usage, cost and decision records for investigation and policy comparison. Agree redaction and retention for business content in logs.
Deliverables & acceptance evidence
Choose the scope at kickoff and use the same workload and constraints for acceptance.
- Request contract & integration examples
- Endpoint, authentication, request examples, error categories and stream-reading notes; verify normal, rejected, timed-out and cancelled requests.
- Capacity & latency report
- Record hardware or provider, input/output lengths, concurrency and arrival pattern; separate queueing, first-token time, completion time and errors.
- Operations & fallback runbook
- Define limits, alerts, investigation entry points, fallback triggers and owners. Configure and explain hard budgets separately from soft latency targets.
Decisions before rollout
These decisions change cost, request behavior and the scope of validation.
Should every request be routed?
No. Pin workflows with a specific model requirement; evaluate routing for traffic with meaningful task variation. Compare quality and cost on held-out samples first.
Is a latency target a service guarantee?
No. The gateway’s latency_target_ms is a soft target. End-to-end time includes network, queueing and generation and must be measured under the agreed load.
What should I bring?
Request samples, model and tool dependencies, input/output length distribution, peak traffic, deployment region, budget constraints and representative failures.
Continue reading
Explore the integration contract4ROUTER / SYSTEMS RESEARCH
Technical foundations of the service
Understand how operators, KV state and scheduling shape an inference path.
Read the systems notes in Resources ↗FlashAttention & PagedAttention: different memory problems
One optimizes data movement; the other organizes persistent state.
↗PREFIX / REUSERadixAttention: reuse prefixes, not answers
Reuse a prefix state with compatible computational history.
↗SCHEDULING / SLOFrom prefill to decode: designing the scheduling experiment
When throughput rises, know which requests are waiting.
↗Let’s talk infrastructure.
Connect to the platform, or discuss models, performance and deployment.