Sign inSign up

Resources / Field guides

Account for the time in a request.

Scoring, a complete routing decision, time to first token and time to completion measure different things. Define the timing boundary before choosing an optimization.

What each metric includes

Do not attach a component measurement to an entire request.

encode + scores
Encodes routing text and scores candidates. This microbenchmark excludes full mode resolution, cost policy and provider calls.
Router.decide
The complete routing call adds mode resolution, conversation assembly, descriptor stacking, cost estimation and ranking. It selects a model without running generation.
First token & completion
At the client, measure request start to first useful content and to completion. These include network and upstream behavior and should be reported separately from router microbenchmarks.

Make measurements comparable

Record the environment and workload distribution so short inputs and warm state do not hide the actual experience.

  1. Fix the environment

    Record checkpoint, pool, encoder, precision and the actual loaded device. Verify where the model resides rather than trusting the device flag alone.

  2. Separate cold & warm state

    Report the first call separately and document warmup before reporting warm measurements. Time checkpoint loading, encoder initialization and request decisions separately.

  3. Cover realistic inputs & load

    Test representative conversation lengths and pools and retain quantiles and extremes. Follow component tests with serving tests for concurrency, arrival patterns and client timeouts.

Locate the bottleneck

Add measurements that distinguish causes before changing configuration or code.

Long inputs slow down
Compare encoding with complete decide time and inspect routing text length and truncation. Shortening routing inputs can change model selection quality; evaluate both.
The first request is slow
Inspect initialization, weight loading, caches and device warmup. Report cold behavior separately rather than presenting a warm median as the first-request experience.
Healthy components, slow online tails
Measure gateway queueing, upstream first-token time, retries and networking. Control concurrency and arrivals and retain failed and timed-out samples.

Rerun decision timing

Use an existing checkpoint matching the pool in the project environment. This is the documented CPU path; the script does not invoke providers.

Keep the context with the output
Archive the JSON, command, software versions and device information. The published timing uses crb-gen1; do not present it as a new crb-gen2 measurement.
python3 scripts/bench_decide.py \
  --pool configs/pool.coderouterbench.yaml \
  --checkpoint ~/.enthalpy/checkpoints/crb-gen1 \
  --device cpu --out reports/router-latency.cpu.json

Continue reading

Apply the measurements to serving acceptance

Routing latency

A selection is a real prefill, so it grows with the length of the prompt. Below are the raw measurements, with the conditions stated under the table; whether they earn their place ahead of an upstream call is yours to judge.

prompt tokensCharactersp50 (warm)p95 (warm)min–maxFirst call (cold)
156071 ms77 ms67 ms – 80 ms180 ms
2561,293264 ms334 ms247 ms – 369 ms251 ms
1,0245,229944 ms998 ms917 ms – 1029 ms963 ms

A short prompt (15 tokens) takes 71 ms; at the truncation ceiling of 1,024 tokens it takes 944 ms. ENTHALPY_ROUTER_MAX_TOKENS is therefore a latency control and not only a memory one — the routing signal sits in the statement of the task, the statement of the task sits near the start of the prompt, and truncation cuts from the left.

The cold and warm columns have to be read together. The first decision in a process takes 180 ms, and only at steady state is it 71 ms — the backend is building its kernels. Send a warm-up request before a new replica goes behind the load balancer. Loading the weights takes a further 2.7 s on top of that, once per process start.

Conditions: backbone Qwen/Qwen3-0.6B, device cpu, dtype float32, 8 candidates, 25 iterations plus 5 warm-ups, router_max_tokens 1,024. Checkpoint sft-2e9f, pool 3520b40d — note that this is not the checkpoint the accuracy figures above came from. Two measurements from two artefacts do not share one line of provenance.

4ROUTER / SYSTEMS RESEARCH

State, reuse and scheduling: where latency comes from

KV cache, RadixAttention, LMCache and chunked scheduling determine which part of a request waits. The models, figures and protocols below separate that critical path.

NOTE / 03STATE / CAPACITY

KV cache: from tensor shapes to capacity budgets

Context length is a state budget that grows with concurrency.

Autoregressive decoding keeps historical keys and values instead of recomputing that state at each step. The payload depends on layers, KV heads, head dimension, storage precision and retained tokens. This note distinguishes a logical tensor budget from GPU allocation and provides a reproducible capacity calculator.

FIG. 01STATE / CAPACITY
L × H_kv × d_headK + V× bytes × tokens
Model dimensions × retained tokensSchematic, not a measured scale or a specific engine layout.

What the cache contains

In conventional decoder attention, each layer retains K and V for processed tokens. A new token still produces its query and reads relevant history, so a KV cache does not remove attention’s bandwidth cost. Model capacity using retained windows and layer types, not only the advertised maximum context. [1]

GQA uses KV heads in the capacity term

MHA, MQA and GQA differ in how query heads share K/V groups. For GQA, the capacity term uses H_kv, not the query-head count H_q. This is part of model architecture; reducing a configuration field on an existing MHA checkpoint is not lossless compression. [2]

Payload bytes are not allocated bytes

Block rounding, metadata, quantization scales, temporary workspace, weights and framework reservations affect allocation. Tensor parallelism also requires checking KV-head sharding or replication; dividing by GPU count is not universally valid. Prefix sharing can reduce copies, but count the actual physical sharing relationships. [3]

Model & notation

B_KV = 2 × L × H_kv × d_head × b × Σ Tᵢ

For conventional full-attention models with identical KV shapes across layers: 2 counts K and V; L is layers, H_kv KV heads, d_head head dimension, b bytes per element and Tᵢ retained tokens per sequence. Excludes sharing, sharding, block rounding and overhead.

INTERACTIVE / CAPACITY

KV capacity workbench

Illustrative parameters, not a named model. Calculates total logical KV payload, excluding weights, sharding, sharing and engine overhead.

KV capacity workbench
Logical KV payload4.00 GiB
Per token · all layers
128.00 KiB
Per sequence
1.00 GiB

2 × 32 × 8 × 128 × 2 × 8192 × 4 = 4294967296 bytes

Integer multiplication gives bytes; KiB = 2¹⁰ bytes and GiB = 2³⁰ bytes. Displays round to two decimal places.

Inputs: the editable illustrative fields above. Method: the integer formula on this page.

Experimental protocol

  1. Read the model configuration

    Check layer count, KV heads, head dimension and actual cache dtype. Weight quantization does not determine KV precision.

  2. Record the length distribution

    Record input length, output length and simultaneously active requests. Budget actual retained state, including long tails and growth.

  3. Cross-check the runtime

    Compare the logical estimate with engine-reported available blocks and peak allocation. Attribute the difference rather than treating every extra byte as waste.

References

  1. Kwon et al. · KV memory in PagedAttention
  2. Ainslie et al. · GQA · arXiv:2305.13245
  3. vLLM · Paged attention tensor layout
NOTE / 04PREFIX / REUSE

RadixAttention: reuse prefixes, not answers

Reuse a prefix state with compatible computational history.

Multi-turn conversations, shared system prompts and branching generation repeatedly encounter identical prefixes. SGLang’s RadixAttention organizes token prefixes and KV state in a radix tree, making matching, splitting, references and eviction runtime concerns. The key is compatible reusable state, not superficially similar text.

FIG. 01PREFIX / REUSE
Shared prefixsystem / document
query Aquery Bquery C

Independent suffix

Match a common prefix, then compute the suffixSchematic, not a measured scale or a specific engine layout.

How a radix tree locates shared prefixes

An edge can represent a token sequence rather than one node per token. An incoming request matches a stored prefix, splits at a divergence and computes KV for the unmatched suffix. Shared references require protection of state in use and eviction choices under capacity pressure. [1]

Read hits in tokens and time

A request-level hit does not distinguish a short system prompt from a long document. Record reused tokens, avoided prefill work and lookup overhead. Prefix caching primarily saves reusable prefill; it does not remove autoregressive generation of the answer. A high hit rate is not an equivalent end-to-end speedup. [2]

Consider compatibility, isolation and scheduling

An identical prefix is insufficient without compatible model revision, adapter, tokenization and positional semantics. Define cache namespaces and tenant boundaries. Sending work to the instance with the most reusable state may improve reuse but increase queueing; evaluate both rather than maximizing hits alone. [1]

Model & notation

T_cached ≈ T_lookup + T_prefill(suffix | prefix KV) + T_decode

This is a conceptual cost decomposition. Reuse avoids recomputing prefix state, while suffix attention still reads that state. Measure queueing, transfers and scheduling in the complete request.

Experimental protocol

  1. Construct hits and misses

    Keep model and output settings fixed. Test shared prefixes, semantically similar inputs with different tokens, and unrelated inputs separately.

  2. Apply cache pressure

    After warmup, add competing requests that trigger eviction, then measure reuse and recovery. Record reference protection, eviction and waiting.

  3. Accept on workload metrics

    Compare TTFT, inter-token latency, useful throughput and queueing. Define explicit isolation and authorization rules for cross-tenant reuse.

References

  1. Zheng et al. · SGLang / RadixAttention · arXiv:2312.07104
  2. vLLM · Automatic prefix caching and limits
NOTE / 06SCHEDULING / SLO

From prefill to decode: designing the scheduling experiment

When throughput rises, know which requests are waiting.

Serving handles prompt computation and iterative generation together. Continuous batching, chunked prefill and P/D separation change how requests interfere across these phases. This note organizes experiments around joint latency and throughput constraints rather than a universally optimal scheduling switch.

FIG. 01SCHEDULING / SLO
PrefillDecodeComplete / release
APDDD···
B·PDDD··
C··PDDD·
Admit and retire requests between iterationsSchematic, not a measured scale or a specific engine layout.

Continuous batching changes scheduling granularity

Orca’s iteration-level scheduling advances work at generation-iteration granularity, allowing completed requests to leave and waiting work to enter without waiting for an entire static batch. This reduces some idle work with mixed lengths, while actual batch size remains constrained by KV capacity, token budgets and execution policy. [1]

How long prefill affects decode

A long prompt can occupy a long execution interval and interfere with requests already decoding. Sarathi-Serve studies chunked prefill and batching to manage throughput–latency interference. Chunk size cannot be selected on TTFT alone: smaller chunks can add scheduling overhead and change GPU execution efficiency. [2]

P/D separation introduces a transfer boundary

Separate prefill and decode instances permit independent resource planning, but KV must cross the instance boundary and load must fit both capacities. Reuse, transfer overlap and queueing determine the critical path. If either side backs up, isolated phase throughput does not describe the user experience. [3]

Model & notation

TTFT = T_queue + T_prompt_path + T_first_content

This is a client-boundary decomposition, not a universal definition of engine timing fields. Report inter-token latency, completion time and SLO-compliant throughput separately; state whether the first event contains actual content.

Experimental protocol

  1. Fix the workload trace

    Record arrivals, input/output length distributions, shared prefixes, sampling and timeouts. A closed-loop fixed-concurrency test does not represent all production traffic.

  2. Include failed work

    Sweep load on fixed hardware and policies. Report latency distributions, errors, rejections, preemptions and queue growth; define SLO-compliant throughput explicitly.

  3. Separate cold and steady state

    Record warmup, cache fill, measurement interval and unfinished requests at the end. Do not hide overload and recovery by selecting only a stable slice.

References

  1. Yu et al. · Orca · OSDI 2022
  2. Agrawal et al. · Sarathi-Serve · OSDI 2024
  3. LMCache · KV transfer and disaggregation

Original explanatory figures; equations describe mechanisms or capacity models. Results in papers and documentation are not 4Router measurements.