4ROUTER / Routing
Inside our routing engine
Tracing a request through selection and dispatch.
Engineering note
Begin with the candidate set
The routing engine selects candidates from the gateway model pool. Model descriptions, task context and cost constraints inform the decision. Candidate scores explain selection; they do not guarantee the quality of an individual response.
Record decision and execution separately
Choosing a model and successfully calling it are separate events. Troubleshooting needs the candidate set, the selection and the attempts that followed. The engine page contains recorded decisions; its evaluation page states the pool, workload and measurement context.
Keep the engine inside the platform boundary
Routing is one part of the platform alongside integration, accounts, billing and audit. Clients can use model selection or pin a model where the interface supports it. Integration also needs error, stream-completion and usage-record checks, beyond a successful response.
Before implementation
- Model pool and constraints
- Candidate and execution records
- Errors and stream completion
