SELECT
One selection, one call. Latency = the routing prefill plus one upstream completion; cost = one completion. The default.
Engine
Scoring and selection
Enthalpy computes a score for every model card in the pool, subtracts the cost term, takes the maximum, and keeps the full candidate table. This page describes the scoring function, the four routing modes, and the result of the generalisation measurement.
The same at 1,138 candidates — the shapes of W and P have no model-count dimension.
- backbone
- Qwen/Qwen3-0.6B
- Upstream calls in this step
- 0
Recorded decisions
The full candidate tables for four real queries at each cost weight, recorded from the current checkpoint.
The full candidate table for one decision
One query: raising the cost weight from 0 to 1 changes the model that is selected. The raw scores do not move, the cost term does, and the final ordering follows.
Query: “A one-line rename”. claude-opus-4-6 has the highest raw score (+15.132), and the highest estimated cost ($0.006240). Raising γ from 0 to 1 gives it a cost term of -0.485, while kimi-k2.5 takes -0.048; the final score +14.678 then exceeds +14.647, a difference of 0.0307.
The probability column is identical at both values of γ: the softmax runs on the raw logits and the cost term does not reach it. What changes the selection is an explicit price term that can be turned back to zero.
This is not a special case exported for a landing page. The gateway writes every route into the route_decisions table unconditionally: explain in the request decides only whether the candidate table is included in the response body, never whether it is stored. A customer can open any request they have sent in the console, and this is the table they see.
Scoring runs locally and makes no upstream call. One local forward pass decides which model the request goes to; the selected model is called only after that.
The scoring function
The model pool is configuration, not a retrained head.
The query makes one pass through the backbone and gives h(q) ∈ R^1024. The descriptor paragraph of a model card passes through the same backbone, is projected by P and normalised, and gives e_j ∈ R^128. The score is a bilinear product:
s_j = (h(q)ᵀ W) · e_j
What matters is that the shapes of W and P carry no “number of models” dimension. A model is an input, not a parameter.
Scoring head parameters
| Tensor | Shape | Parameters |
|---|---|---|
P | 1024 × 128 | 131,072 |
W | 1024 × 128 | 131,072 |
b_acc | scalar | 1 |
w_acc | 1024 | 1,024 |
| Total | 263,169 |
The checkpoint holds 8 further groups, 8,192 floats in all, which are the cached backbone output for those 8 descriptors. They are cached inputs, not weights — counting them as parameters would concede that the head grows with the number of models, and that is the claim this architecture exists to refute.
Parameter count is independent of pool size
Adding or removing a model changes the descriptor text and its cached vector; W, P, w_acc and b_acc do not move a byte. At 8 models in the pool the head is 263,169 parameters, and at a thousand candidates it is the same number.
Generalisation to untrained models: not achieved
Across 1,138 candidates in 12 groups, holding out 12.5% leaves 142 candidates the head never trained against. The result:
| Slice | Router | Best single model | oracle | Random |
|---|---|---|---|---|
val | 0.7571 | 0.7667 | 0.9758 | 0.4865 |
test | 0.7516 | 0.7637 | 0.9749 | 0.4915 |
holdout | 0.5702 | 0.6596 | 0.8668 | 0.4863 |
On the held-out slices beats_best_single is false and gap_closed is -0.4313 — negative, meaning it does worse than the best single model. What still holds is beats_random (true). So: the scoring function needs no retraining when the pool changes, but the quality after a change of pool has no evidence today, and the evidence that does exist points the other way. Every positive number above comes from inside one pool.
Four modes
One endpoint, four paths. The first two differ in whether the answer is scored; the last two differ in who decides how many calls are made.
ESCALATE
Starts at the lower-cost candidates and works up the order, for at most max_attempts attempts. The result of the last attempt is always returned; when the threshold is not reached, notes records that.
ORCHESTRATE
The planner emits a graph and the executor runs it in parallel, in depth waves. A step sees the original question plus the earlier outputs named in its access list — not the other sub-tasks, not the plan itself, and not the outputs of sibling steps in the same wave.
access may only point backwards, which is why every edge in the graph above runs to the left. A DAG whose edges all run the same way is itself the proof of that isolation contract. A plan that violates it is refused rather than executed first and checked afterwards.
PINNED
model in the request names a real id from the pool, and the router is skipped entirely. Auth, admission, billing, and the decision log carry on as before. This is the setting for a migration: move one endpoint across, keep it pinned, and leave it pinned until you are willing to trust the routing.
Only SELECT and PINNED stream from upstream for real — the other two do not know which model will produce the answer until it exists, so they replay it as SSE once they have it and note in notes that the frame timing is not the upstream timing. Streaming out step 0 of an escalate and then withdrawing it is worse than a short wait.
Three design trade-offs
Fugu made the opposite choice three times over in the same kind of work. Each item below states where the evidence is, and where it stops.
The model pool is replaceable
The shape of the head does not change with the size of the pool: 263,169 parameters at 8 candidates, and the same number at 1,138 candidates. Adding a model means writing a paragraph of description, not retraining a classifier head.
That is a mechanical fact, and it can be checked. It does not mean “quality survives a change of pool” — that second claim was measured, and on 142 candidates the head had never trained against, it did not hold (section 3 above prints the raw numbers for all three slices). The two sentences are kept apart because their evidence differs by an order of magnitude.
Cost enters the score rather than a post-processing step
γ enters the ranking directly: adjusted = logit − γ·log1p(est_cost / c_ref). It is not a price check applied after the choice, and it is not a 0–1 ratio knob — its unit is the logit, so it competes with the model's confidence on one scale, and which of the two wins can be computed.
The evidence is the frontier curve in section 4, including why the factory default is 0: at γ=0.3 the confidence interval on the advantage crosses zero.
Decisions can be opened
route_decisions is written unconditionally. explain in the request decides only whether this response body carries the candidate table, not whether it is stored — so any past request can afterwards be opened in the console with its full candidate scores, every attempt (including the ones that failed and the ones the accept head refused), and the billed amount to 1e-8 of a dollar.
Fugu's decisions are not exposed by design. Here the decision record is written unconditionally.
4ROUTER / SYSTEMS RESEARCH
Connect configuration choices to system behavior.
Understand how operators, KV state and scheduling shape an inference path.
Read the systems notes in Resources ↗GEMM / KERNELS
From operators to serving: cuBLAS & cuDNN
Identify the limiting resource before selecting the implementation.
↗STATE / CAPACITYKV cache: from tensor shapes to capacity budgets
Context length is a state budget that grows with concurrency.
↗SCHEDULING / SLOFrom prefill to decode: designing the scheduling experiment
When throughput rises, know which requests are waiting.
↗Let’s talk infrastructure.
Connect to the platform, or discuss models, performance and deployment.