Use this checklist before publishing a benchmark, choosing a system or handing over a deployment. Checks are a local review aid, not automated verification.
Checks remain on this page only, reset on reload and are not uploaded.
01
Experiment identity & inputs
A reader should be able to identify the experiment behind the conclusion.
Record sources, revisions, selection scope, sampling limits and duplicate handling. Distinguish business data from public benchmarks.
Keep source- or hash-split logs and relevant settings. Do not describe ignored hash parameters as the actual split when source splits took precedence.
Record pool fingerprint, checkpoint_id, encoder revision and Stage 1 training weight λ; retain flat_row_rate when that weight is nonzero.
02
Comparisons & conclusions
A claim of improvement needs a reference, a metric and the conditions under which it holds.
Name the reference and distinguish bounds such as oracle. Inspect model usage rather than describing a single-model choice as a multi-model routing gain.
Keep intervals, sign tests and correctness thresholds with differences. State when intervals cross zero and retain failure slices.
State cost sources and scope; retain the frontier and undefined values. Do not extend offline quality results into latency, capacity or cross-domain promises.
03
Timing & reproducibility
A measurement is useful only alongside its boundary.
Record actual device, precision, input length, candidates, warmup and sampling. Separate cold start, complete decision time and generation.
Keep raw JSON, Markdown and sweep files. Use report formatters rather than retyping numbers or dropping bound labels.
Provide commands, environment versions, input preparation and report locations. Identify network access and steps that call paid providers.
04
When moving from evaluation to deployment
These are separate acceptance conditions that an offline evaluation cannot replace.
Assign access, key rotation, monitoring, backup/restore, upgrades, rollback and operational ownership. Hand over verification records with configuration.
Validate quality, concurrency, latency, failures and budget behavior on agreed workloads. Keep untested features and environments explicitly outside verified scope.
Put operators, KV capacity and cache reuse on the handoff checklist so literature mechanisms do not become platform promises by accident.
NOTE / 01GEMM / KERNELS
From operators to serving: cuBLAS & cuDNN
Identify the limiting resource before selecting the implementation.
A model forward pass becomes matrix products, normalization, activations, layout changes and communication. The optimization target is an execution path at a particular shape, precision and layout, rather than a single GPU-utilization number. This note connects library selection, roofline reasoning and serving validation.
FIG. 01GEMM / KERNELS
X
×
W
→
Y
Tiled matrix products and accumulationSchematic, not a measured scale or a specific engine layout.
01GEMM shape is part of the problem
A linear projection can be written as Y = XW. The number of participating tokens changes the row dimension, so large-matrix tests alone do not represent both prefill and decode. cuBLAS supplies BLAS operations; cuBLASLt exposes additional choices for layouts, compute types, algorithm candidates and workspace. Separate algorithm-selection overhead from repeated execution. [1]
02Fusion changes data movement
cuDNN’s graph interface expresses operations and dependencies for supported execution plans; SDPA is an example of a fused path. Avoiding intermediate writes can reduce memory traffic. Availability still depends on precision, masks, shapes, layouts and version. A cuDNN dependency alone does not prove that an inference framework uses a fused path. [2]
03Use roofline as a testable hypothesis
Arithmetic intensity divides work in FLOPs by bytes moved. Low-intensity paths can be bandwidth-limited, while larger products may approach compute limits; launches, synchronization and communication can sit outside this simplified model. Measure the operator, inspect timeline gaps, then validate complete requests. Peak FLOPs are not a serving-throughput prediction. [3]
Model & notation
P ≤ min(P_peak, BW × I) I = FLOPs / bytes
P is attainable operation rate, BW is bandwidth at the chosen memory tier, and I is intensity at that tier. This upper-bound model omits queueing, communication and scheduling.
Experimental protocol
Fix the input contract
Record M/N/K, batch size, precision, transpose, strides, alignment and workspace. Use shapes from the actual model, including irregular ones.
Measure execution and correctness
Use device-side timing with an explicit synchronization boundary and report a distribution after warmup. Compare numerical error with a reference, including edge shapes and exceptional values.
Return to the complete service
Keep model and traffic fixed; inspect end-to-end latency, throughput, peak memory and errors. If kernel gains do not survive, examine scheduling and data movement.
FlashAttention & PagedAttention: different memory problems
One optimizes data movement; the other organizes persistent state.
The attention expression, kernel memory traffic and KV-cache allocation are separate layers. FlashAttention targets data movement during computation; PagedAttention addresses block-based access to growing KV state. Their boundaries matter more than treating them as alternative switches.
FIG. 01ATTENTION / MEMORY
Logical token blocks
A₀A₁A₂A₃
A₁B₀A₀A₃B₁A₂
Physical KV blocks
Physical KV blocksSchematic, not a measured scale or a specific engine layout.
01FlashAttention: avoid a full intermediate matrix
Attention can be expressed as softmax(QKᵀ / √d + M)V. A direct implementation may materialize large intermediate tensors in HBM. FlashAttention uses tiling and online softmax to reduce those transfers through on-chip storage. It computes exact attention rather than pruning connections through a sparse approximation; floating-point tolerance still needs validation. [1]
Sequence lengths change during generation. Reserving a maximum-length contiguous region can waste capacity. PagedAttention maps logical token blocks to physical KV blocks, which the attention kernel accesses through a block table. Shared state also requires reference management and mechanisms such as copy-on-write. Paging reduces allocation and duplication waste, not the intrinsic KV payload. [2]
03Compatibility is an implementation question
Flash-style tiling and paged KV access can coexist in an implementation; cuDNN’s SDPA documentation lists support conditions for paged inputs. Verify head dimension, precision, masks and GQA layouts for the actual backend. Matching algorithm names do not imply identical execution paths across versions. [3]
Model & notation
O = softmax(QKᵀ / √d + M)V
Q, K and V denote queries, keys and values; d is query/key head dimension and M represents masks or additive bias. Paging changes addressing; tiling changes execution order and traffic.
Experimental protocol
Separate the ablation axes
Hold cache allocation fixed when comparing attention backends. Hold model, input distribution and kernel conditions fixed when comparing allocation policies.
Connect capacity to throughput
Record allocated blocks, tail occupancy, admitted requests and preemptions. Then measure useful throughput under the same latency constraints, not just free memory.
Retain correctness and edge tests
Cover varying sequence lengths, partial final blocks, shared prefixes and long outputs. Retain reference-output comparisons and peak memory measurements.
Avoiding prefill requires paying for state access.
When reusable context exceeds GPU-resident capacity, the question becomes where state lives, when to move it and whether the transfer is worthwhile. LMCache connects inference engines to KV storage and transport across CPU, disk and remote tiers. It changes state placement and lifetime.
FIG. 01TIERS / TRANSFER
TIER 00GPU · HBM
TIER 01CPU · DRAM
TIER 02NVMe / Remote
⇄
Store state by tier; load when neededSchematic, not a measured scale or a specific engine layout.
01Separate engine integration from storage
The inference engine executes the model; LMCache connectors and storage components manage KV retrieval and movement. Tiering can retain colder state in larger storage, but a storage hit still needs a path to usable GPU state. Integration versions, chunk granularity and layouts affect that boundary; backend names alone do not establish compatibility. [1]
02Reuse has a break-even point
A hit entails lookup, reads, transfers and possibly layout conversion. It benefits a request when that cost is below recomputing prefill. Asynchronous prefetch may hide part of the cost while consuming bandwidth and buffers. Measure effective transfer rates and queueing instead of substituting advertised link speed for serving behavior. [2]
03Context reuse and P/D separation differ
Context caching retains existing state across requests. Prefill/decode separation places two phases of one request on different execution instances, requiring KV transfer. They can be combined, but historical-prefix reuse does not validate cross-instance transport or recovery. Observe hits, misses, timeouts and fallback for each transfer path. [2]
This is a reuse criterion without overlap, not a throughput predictor. Concurrent transfers contend for bandwidth; with pipelining, measure the critical path rather than blindly summing stages.
Experimental protocol
Test cache states separately
Separate GPU-hot, CPU-hot, disk-hot, remote-hot and fully cold requests. Record model, sequence length, tier, hit bytes and recomputation scope.
Trace data movement
Record lookup, reads, device transfers and waiting; inspect pinned memory, NUMA, PCIe or network contention alongside TTFT.
Verify fallback and cleanup
Test missing objects, exhausted capacity, unavailable backends and incompatible versions. Verify recomputation or explicit failure; define tenant isolation, retention and deletion.