Services

Private deployment

Deploy model services in your environment with explicit data paths, access controls and operational ownership. Turn the deployment boundary into configuration and verification records.

Engineering field guide

Make the data boundary a deployment boundary.

Define what stays in your environment and which dependencies require external access, from ingress to models, logs and backups.

Map the boundary first

A private deployment needs verifiable network and data paths.

Models & business data: Identify weight sources, data locations, inference processes and output storage. Stage models, dependencies and images in advance when offline operation is required.
Identity & administration: Separate tenant credentials from administrative credentials. Define access boundaries, permissions and rotation; treat administration separately from public inference ingress.
Logs, monitoring & backups: Decide whether prompts and responses are retained, who can read them, retention periods and backup locations. Include alerts and telemetry in egress review.

From environment to callable endpoint

Keep configuration and verification records at every step so upgrades can repeat the process.

Prepare the environment
Check hardware, OS, drivers, container runtime, networking, certificates and storage. Pin software and model versions.
Deploy serving & optional gateway
Establish the inference endpoint and validate inputs and outputs. Add gateway and database components separately when tenant access, decisions and a ledger are required.
Validate & hand over
Test access controls, egress, workload behavior and recovery. Hand over runbooks, backup/restore steps and upgrade/rollback records.

An operational handoff

Put these artifacts in the project checklist and agree the support scope before implementation.

Deployment & dependency manifest: Service topology, versions, configuration locations, secret injection points, certificate renewal and external dependencies.
Health & traffic checks: Validate liveness, readiness and real requests separately. Enthalpy /healthz checks process liveness; /readyz checks traffic prerequisites. They are not interchangeable.
Restore & upgrade plan: Define database, model and configuration backups. Validate restores and version compatibility in isolation before a production switch.

Deployment constraints in practice

A software example is a starting point; production requirements depend on the environment.

Is the Compose example a high-availability design?

No. The repository example is a single-node gateway and database. TLS, replicas, edge rate limiting and secret management need environment-specific implementation and validation.

Can external providers still be used?

Evaluate connectivity separately from data policy. A no-egress requirement calls for restricted candidates, outbound access and logging paths, verified against actual traffic.

What should be ready at kickoff?

The site, hardware inventory, network zones, data policy, identity system, certificate and image sources, operational owners and recovery requirements.

Prepare the handoff with the checklist

4ROUTER / SYSTEMS RESEARCH

Technical foundations of the service

Understand how operators, KV state and scheduling shape an inference path.

Read the systems notes in Resources ↗

Let’s talk infrastructure.

Connect to the platform, or discuss models, performance and deployment.

Talk to us Integration docs

Prepare the environment

Deploy serving & optional gateway

Validate & hand over

KV cache: from tensor shapes to capacity budgets

RadixAttention: reuse prefixes, not answers

LMCache: placing KV state in a storage hierarchy

Let’s talk infrastructure.