Resources / Field guides

Ask the right question of a benchmark.

A useful routing evaluation names the tasks, candidate pool, reference policy and cost. This guide connects reading a report, designing an experiment and interpreting its result.

Define the comparison

An evaluation should inform a deployment choice, not simply rank policies.

Tasks & reward: Use tasks representative of the workload and state scoring rules and the correctness threshold. Reward on code tasks is not accuracy on unrelated domains.
Quality & cost: Read mean reward, cost per request and cost per correct answer together. When there are no correct answers, cost per correct answer is undefined, not zero.
The realistic alternative: best_single selects one model on the training split and pins it for evaluation. It is the deployment baseline against which routing should be compared.

Keep the experiment comparable

Define splitting and model selection before inspecting test outcomes.

Share the test rows
Use one split for all policies. Honor a matrix’s published split when available; otherwise use deterministic hashing of task identity and seed. Reject duplicate task identifiers.
Separate policies from bounds
oracle and cheapest_correct inspect recorded test outcomes and are bounds. The report’s cheapest also uses realized per-row cost; do not present it as a deployable policy.
Preserve the experiment identity
Save the matrix, pool fingerprint, checkpoint_id, training settings and evaluation logs. Check the report’s split description against eval.split.source or eval.split.hash in the logs.

Read differences with their intervals

The point estimate describes this sample. Intervals and failure slices help set the limits of the conclusion.

Paired differences: Compare policy and reference on each shared task before aggregating differences. Do not compare averages from different task sets.
Intervals crossing zero: When the interval for the difference includes zero, report that the result is not distinguishable. Do not promote a positive mean alone as a gain; retain the sign-test result.
The cost frontier: Inspect which sweep points are beaten on both quality and cost. Choose an operating point acceptable to the workload, then test online latency separately.

Limits of the current evidence

The linked benchmarks retain favorable and unfavorable results.

Same-pool results do not establish transfer

The positive CodeRouterBench result supports that task and pool comparison. RouterEval cross-pool experiments do not establish a general transfer advantage. Re-evaluate after a pool change.

Offline replay is not a serving test

Matrix replay produces no new provider outputs and does not measure queueing, networking or generation latency. A quality table cannot establish serving capacity.

Next: reproduce an evaluation

Evaluation setup

Dataset: CodeRouterBench, 4,781 rows in total; the source probing / id_test split provides 1,425 held-out rows. Hash fraction and seed do not determine this split.
Correctness threshold: 0.5. The same threshold applies to every router, including the oracle.
Confidence interval: 2,000 paired bootstrap resamples, seed 0, 95% interval. Paired means the comparison is made on the same row, not between the means of two independent samples.
Model pool: All 8 models qualify; checkpoint sft-f23aa90cd7c68f27e9b702a0, pool fingerprint 3520b40db3b55b1e.
Latency: Qwen/Qwen3-0.6B on cpu, float32, 8 candidates, 25 iterations plus 5 warmups. No GPU.
Cost: Billed against actual token counts at each provider's published list prices, not estimates; the oracle and cheapest baselines use the same price table.

4ROUTER / SYSTEMS RESEARCH

Operators and attention: put the method on the execution path

This section places cuBLAS, cuBLASLt, cuDNN, FlashAttention and PagedAttention inside the evaluation method: define shapes and traffic before discussing throughput or memory.

NOTE / 01GEMM / KERNELS

From operators to serving: cuBLAS & cuDNN

Identify the limiting resource before selecting the implementation.

A model forward pass becomes matrix products, normalization, activations, layout changes and communication. The optimization target is an execution path at a particular shape, precision and layout, rather than a single GPU-utilization number. This note connects library selection, roofline reasoning and serving validation.

FIG. 01GEMM / KERNELS

Tiled matrix products and accumulationSchematic, not a measured scale or a specific engine layout.

GEMM shape is part of the problem

A linear projection can be written as Y = XW. The number of participating tokens changes the row dimension, so large-matrix tests alone do not represent both prefill and decode. cuBLAS supplies BLAS operations; cuBLASLt exposes additional choices for layouts, compute types, algorithm candidates and workspace. Separate algorithm-selection overhead from repeated execution. [1]

Fusion changes data movement

cuDNN’s graph interface expresses operations and dependencies for supported execution plans; SDPA is an example of a fused path. Avoiding intermediate writes can reduce memory traffic. Availability still depends on precision, masks, shapes, layouts and version. A cuDNN dependency alone does not prove that an inference framework uses a fused path. [2]

Use roofline as a testable hypothesis

Arithmetic intensity divides work in FLOPs by bytes moved. Low-intensity paths can be bandwidth-limited, while larger products may approach compute limits; launches, synchronization and communication can sit outside this simplified model. Measure the operator, inspect timeline gaps, then validate complete requests. Peak FLOPs are not a serving-throughput prediction. [3]

Model & notation

P ≤ min(P_peak, BW × I) I = FLOPs / bytes

P is attainable operation rate, BW is bandwidth at the chosen memory tier, and I is intensity at that tier. This upper-bound model omits queueing, communication and scheduling.

Experimental protocol

Fix the input contract
Record M/N/K, batch size, precision, transpose, strides, alignment and workspace. Use shapes from the actual model, including irregular ones.
Measure execution and correctness
Use device-side timing with an explicit synchronization boundary and report a distribution after warmup. Compare numerical error with a reference, including edge shapes and exceptional values.
Return to the complete service
Keep model and traffic fixed; inspect end-to-end latency, throughput, peak memory and errors. If kernel gains do not survive, examine scheduling and data movement.

References

NOTE / 02ATTENTION / MEMORY

FlashAttention & PagedAttention: different memory problems

One optimizes data movement; the other organizes persistent state.

The attention expression, kernel memory traffic and KV-cache allocation are separate layers. FlashAttention targets data movement during computation; PagedAttention addresses block-based access to growing KV state. Their boundaries matter more than treating them as alternative switches.

FIG. 01ATTENTION / MEMORY

Logical token blocks

A₀A₁A₂A₃

A₁B₀A₀A₃B₁A₂

Physical KV blocks

Physical KV blocksSchematic, not a measured scale or a specific engine layout.

FlashAttention: avoid a full intermediate matrix

Attention can be expressed as softmax(QKᵀ / √d + M)V. A direct implementation may materialize large intermediate tensors in HBM. FlashAttention uses tiling and online softmax to reduce those transfers through on-chip storage. It computes exact attention rather than pruning connections through a sparse approximation; floating-point tolerance still needs validation. [1]

PagedAttention: logical continuity, physical blocks

Sequence lengths change during generation. Reserving a maximum-length contiguous region can waste capacity. PagedAttention maps logical token blocks to physical KV blocks, which the attention kernel accesses through a block table. Shared state also requires reference management and mechanisms such as copy-on-write. Paging reduces allocation and duplication waste, not the intrinsic KV payload. [2]

Compatibility is an implementation question

Flash-style tiling and paged KV access can coexist in an implementation; cuDNN’s SDPA documentation lists support conditions for paged inputs. Verify head dimension, precision, masks and GQA layouts for the actual backend. Matching algorithm names do not imply identical execution paths across versions. [3]

Model & notation

O = softmax(QKᵀ / √d + M)V

Q, K and V denote queries, keys and values; d is query/key head dimension and M represents masks or additive bias. Paging changes addressing; tiling changes execution order and traffic.

Experimental protocol

Separate the ablation axes
Hold cache allocation fixed when comparing attention backends. Hold model, input distribution and kernel conditions fixed when comparing allocation policies.
Connect capacity to throughput
Record allocated blocks, tail occupancy, admitted requests and preemptions. Then measure useful throughput under the same latency constraints, not just free memory.
Retain correctness and edge tests
Cover varying sequence lengths, partial final blocks, shared prefixes and long outputs. Retain reference-output comparisons and peak memory measurements.

References

Original explanatory figures; equations describe mechanisms or capacity models. Results in papers and documentation are not 4Router measurements.

Share the test rows

Separate policies from bounds

Preserve the experiment identity

Evaluation setup

From operators to serving: cuBLAS & cuDNN

01GEMM shape is part of the problem

02Fusion changes data movement

03Use roofline as a testable hypothesis