Engine · Benchmarks
Benchmark results
Measurements on CodeRouterBench. Every figure is read from the evaluation artifacts at build time and matches the record in reports/.
Every row was run through all ten strategies, with 2,000 paired bootstrap resamples and split seed 0.
- Interval
- 95%
- correct threshold
- 0.5
Policy comparison
On 1,425 held-out rows the reward for enthalpy is 0.4381, and for the best single model in the pool it is 0.4167.
- oracle †0.5212$0.00112
- cheapest_correct †0.5006$8.31e-4
- enthalpy0.4381$0.00468
- knn(k=16)0.4257$0.00340
- best_single0.4167$0.00727
- cluster_balance(k=8,alpha=0.7)0.4162$0.00283
- random(seed=0)0.3488$0.00361
- round_robin0.3420$0.00357
- most_expensive †0.3363$0.00935
- cheapest †0.3317$2.36e-4
enthalpy† bounds, not policiesOther baselinesBar length = reward; the two right-hand columns are reward and the average cost per call
The four rows marked † are bounds, not deployable policies: oracle, cheapest_correct, cheapest, and most_expensive use the recorded result of every row. oracle reaches 0.5212 at an average cost of $0.00112, and the gap to the reward for enthalpy is 0.0831.
The full comparison: correct count, cost per call, cost per correct answer, reward/$, oracle gap, models used
| router | reward | correct | $/query | $/correct | reward/$ | oracle gap | models used |
|---|---|---|---|---|---|---|---|
| oracle † | 0.5212 | 747/1425 | $0.00112 | $0.00215 | 463.5 | 0.0000 | 8 |
| cheapest_correct † | 0.5006 | 747/1425 | $8.31e-04 | $0.00159 | 602.1 | 0.0206 | 8 |
| enthalpy | 0.4381 | 622/1425 | $0.00468 | $0.01072 | 93.6 | 0.0831 | 8 |
| knn(k=16) | 0.4257 | 607/1425 | $0.00340 | $0.00798 | 125.3 | 0.0956 | 8 |
| best_single | 0.4167 | 594/1425 | $0.00727 | $0.01743 | 57.3 | 0.1045 | 1 |
| cluster_balance(k=8,alpha=0.7) | 0.4162 | 587/1425 | $0.00283 | $0.00686 | 147.2 | 0.1051 | 5 |
| random(seed=0) | 0.3488 | 489/1425 | $0.00361 | $0.01053 | 96.6 | 0.1725 | 8 |
| round_robin | 0.3420 | 477/1425 | $0.00357 | $0.01065 | 95.9 | 0.1793 | 8 |
| most_expensive | 0.3363 | 460/1425 | $0.00935 | $0.02895 | 36.0 | 0.1849 | 6 |
| cheapest † | 0.3317 | 463/1425 | $2.36e-04 | $7.26e-04 | 1406.1 | 0.1896 | 4 |
Over 1425 held-out rows, enthalpy beats best_single on reward: 0.4381 vs 0.4167, Δ=+0.0214, 95% CI [+0.0093, +0.0336], sign test p=0.0019, reward per dollar 1.63x.
Deltas and confidence intervals
Each horizontal line is a 95% paired bootstrap confidence interval; the vertical rule is zero. An interval that contains zero marks a delta that cannot be distinguished from zero.
The third interval contains zero: knn(k=16) is not distinguishable from best_single. Wins/losses/ties for enthalpy against the best single model are 78/43/1304.
Against k-NN the reward per dollar is 0.75×, below one: higher quality, lower efficiency per unit of cost. The trade-off in the cost weight γ is the next section.
The full paired tests: bootstrap p, sign p, wins/losses/ties, win rate, reward/$ ratio
| router | Δreward | 95% CI | bootstrap p | sign p | W/L/T | win rate | reward/$ ratio |
|---|---|---|---|---|---|---|---|
| oracle | +0.1045 | [+0.0901, +0.1193] | 0.0000 | 0.0000 | 308/0/1117 | 0.608 | 8.08 |
| cheapest_correct | +0.0839 | [+0.0688, +0.0997] | 0.0000 | 0.0000 | 274/129/1022 | 0.551 | 10.50 |
| enthalpy | +0.0214 | [+0.0093, +0.0336] | 0.0010 | 0.0019 | 78/43/1304 | 0.512 | 1.63 |
| knn(k=16) | +0.0089 | [-0.0040, +0.0222] | 0.1950 | 1.0000 | 86/85/1254 | 0.500 | 2.18 |
| cluster_balance(k=8,alpha=0.7) | -0.0005 | [-0.0147, +0.0136] | 0.9390 | 0.3711 | 115/130/1180 | 0.495 | 2.57 |
| random(seed=0) | -0.0680 | [-0.0860, -0.0503] | 0.0000 | 0.0000 | 107/304/1014 | 0.431 | 1.68 |
| round_robin | -0.0747 | [-0.0917, -0.0576] | 0.0000 | 0.0000 | 104/307/1014 | 0.429 | 1.67 |
| most_expensive | -0.0804 | [-0.0975, -0.0634] | 0.0000 | 0.0000 | 70/279/1076 | 0.427 | 0.63 |
| cheapest | -0.0851 | [-0.1061, -0.0661] | 0.0000 | 0.0000 | 134/365/926 | 0.419 | 24.52 |
| router | Δreward | 95% CI | bootstrap p | sign p | W/L/T | win rate | reward/$ ratio |
|---|---|---|---|---|---|---|---|
| oracle | +0.0956 | [+0.0814, +0.1098] | 0.0000 | 0.0000 | 295/0/1130 | 0.604 | 3.70 |
| cheapest_correct | +0.0750 | [+0.0606, +0.0901] | 0.0000 | 0.0000 | 256/130/1039 | 0.544 | 4.81 |
| enthalpy | +0.0125 | [+0.0018, +0.0231] | 0.0220 | 0.0022 | 68/36/1321 | 0.511 | 0.75 |
| best_single | -0.0089 | [-0.0222, +0.0040] | 0.1950 | 1.0000 | 85/86/1254 | 0.500 | 0.46 |
| cluster_balance(k=8,alpha=0.7) | -0.0095 | [-0.0202, +0.0011] | 0.0740 | 0.2698 | 73/88/1264 | 0.495 | 1.17 |
| random(seed=0) | -0.0769 | [-0.0949, -0.0596] | 0.0000 | 0.0000 | 104/302/1019 | 0.431 | 0.77 |
| round_robin | -0.0837 | [-0.1003, -0.0667] | 0.0000 | 0.0000 | 100/300/1025 | 0.430 | 0.77 |
| most_expensive | -0.0893 | [-0.1070, -0.0725] | 0.0000 | 0.0000 | 87/293/1045 | 0.428 | 0.29 |
| cheapest | -0.0940 | [-0.1114, -0.0772] | 0.0000 | 0.0000 | 104/334/987 | 0.419 | 11.22 |
Distribution of model selections
The four queries in the hero all select the same model at γ=0 — four prompts say nothing about a distribution.The distribution over the whole held-out set does:
enthalpy
Selection entropy 2.2589 bit
best_single
Selection entropy 0.0000 bit
Method (a standing footer for this section, not a tooltip): CodeRouterBench, 4,781 rows, split at 30% into 1,425 held-out rows (3,356 rows for training, split seed 0). 2,000 paired bootstrap resamples, seed 0, 95% intervals; correct threshold 0.5; all 8 models in the candidate pool are eligible. The resolution floor of a bootstrap p is 1/2,000, so a 0.0000 in these tables should be read as “smaller than that”.
At a cost weight of 0.3, the confidence interval for that advantage includes zero. The shipped default is therefore 0.
The cost term and the frontier
After scoring, γ · log1p(est_cost / c_ref) is subtracted — the unit is log cost, not a ratio between 0 and 1. Raising γ moves the router towards the cheaper side, and it moves inside the same argmax, with no second set of rules.
Every point on the frontier: γ, reward, cost per query, models used
| γ | reward | $/query | Models used | Gap to oracle |
|---|---|---|---|---|
| 0 | 0.4381 | $0.00468 | 8 | 0.0831 |
| 0.3 | 0.4282 | $0.00355 | 8 | 0.0930 |
| 1 | 0.4167 | $0.00231 | 6 | 0.1045 |
| 2 | 0.3950 | $0.00108 | 5 | 0.1262 |
| 4 | 0.3692 | $3.39e-4 | 4 | 0.1520 |
| 8 | 0.3609 | $3.04e-4 | 2 | 0.1603 |
The result at a cost weight of 0.3
Over 1425 held-out rows, the strongest policy (enthalpy, reward 0.4282) is NOT distinguishable from best_single (reward 0.4167): Δ=+0.0115, 95% CI [-0.0015, +0.0247] includes zero.
The same data and the same bootstrap, with γ moved from zero to 0.3: reward falls a little and each query costs noticeably less, but the confidence interval crosses zero. Crossing zero means the advantage can no longer be distinguished statistically from no advantage at all. That is why the factory default carries no cost penalty — the cheaper setting was measured, it does save, it does not win by enough to be certain, and so it is not the default.
| router | Δreward | 95% CI | bootstrap p | sign p | W/L/T | win rate | reward/$ ratio |
|---|---|---|---|---|---|---|---|
| oracle | +0.1045 | [+0.0901, +0.1193] | 0.0000 | 0.0000 | 308/0/1117 | 0.608 | 8.08 |
| cheapest_correct | +0.0839 | [+0.0688, +0.0997] | 0.0000 | 0.0000 | 274/129/1022 | 0.551 | 10.50 |
| enthalpy | +0.0115 | [-0.0015, +0.0247] | 0.0830 | 0.1055 | 79/59/1287 | 0.507 | 2.10 |
| random(seed=0) | -0.0680 | [-0.0860, -0.0503] | 0.0000 | 0.0000 | 107/304/1014 | 0.431 | 1.68 |
| round_robin | -0.0747 | [-0.0917, -0.0576] | 0.0000 | 0.0000 | 104/307/1014 | 0.429 | 1.67 |
| most_expensive | -0.0804 | [-0.0975, -0.0634] | 0.0000 | 0.0000 | 70/279/1076 | 0.427 | 0.63 |
| cheapest | -0.0851 | [-0.1061, -0.0661] | 0.0000 | 0.0000 | 134/365/926 | 0.419 | 24.52 |
Routing latency
A selection is a real prefill, so it grows with the length of the prompt. Below are the raw measurements, with the conditions stated under the table; whether they earn their place ahead of an upstream call is yours to judge.
| prompt tokens | Characters | p50 (warm) | p95 (warm) | min–max | First call (cold) |
|---|---|---|---|---|---|
| 15 | 60 | 71 ms | 77 ms | 67 ms – 80 ms | 180 ms |
| 256 | 1,293 | 264 ms | 334 ms | 247 ms – 369 ms | 251 ms |
| 1,024 | 5,229 | 944 ms | 998 ms | 917 ms – 1029 ms | 963 ms |
A short prompt (15 tokens) takes 71 ms; at the truncation ceiling of 1,024 tokens it takes 944 ms. ENTHALPY_ROUTER_MAX_TOKENS is therefore a latency control and not only a memory one — the routing signal sits in the statement of the task, the statement of the task sits near the start of the prompt, and truncation cuts from the left.
The cold and warm columns have to be read together. The first decision in a process takes 180 ms, and only at steady state is it 71 ms — the backend is building its kernels. Send a warm-up request before a new replica goes behind the load balancer. Loading the weights takes a further 2.7 s on top of that, once per process start.
Conditions: backbone Qwen/Qwen3-0.6B, device cpu, dtype float32, 8 candidates, 25 iterations plus 5 warm-ups, router_max_tokens 1,024. Checkpoint sft-2e9f, pool 3520b40d — note that this is not the checkpoint the accuracy figures above came from. Two measurements from two artefacts do not share one line of provenance.
Evaluation setup
- Dataset
- CodeRouterBench, 4,781 rows in total; the source probing / id_test split provides 1,425 held-out rows. Hash fraction and seed do not determine this split.
- Correctness threshold
- 0.5. The same threshold applies to every router, including the oracle.
- Confidence interval
- 2,000 paired bootstrap resamples, seed 0, 95% interval. Paired means the comparison is made on the same row, not between the means of two independent samples.
- Model pool
- All 8 models qualify; checkpoint
sft-f23aa90cd7c68f27e9b702a0, pool fingerprint3520b40db3b55b1e. - Latency
- Qwen/Qwen3-0.6B on cpu, float32, 8 candidates, 25 iterations plus 5 warmups. No GPU.
- Cost
- Billed against actual token counts at each provider's published list prices, not estimates; the oracle and cheapest baselines use the same price table.
Let’s talk infrastructure.
Connect to the platform, or discuss models, performance and deployment.