Reproducibility contract
Claims require a workload.
TokenLabs publishes inference results as bounded engineering evidence—not universal framework, model, or quantization rankings. Every conclusion applies only to the disclosed hardware, software, traffic model, and SLO.
Publication rule: a result is “published” only when the run is complete, the comparison controls are disclosed, request failures are accounted for, machine-readable evidence is linked, and the recommendation includes limitations. Everything else remains in the lab notebook.
Metric definitions
| TTFT | Time from request submission to the first output token. Report p50, p95, and p99 when the sample count supports them. |
|---|---|
| ITL / TPOT | Delay between generated tokens / average time per output token. The exact aggregation used by the load generator is disclosed per study. |
| Output throughput | Total generated output tokens divided by measured wall-clock duration. It is paired with per-user latency, not treated as a standalone winner metric. |
| Request throughput | Completed requests per second under the stated arrival model, concurrency, and output distribution. |
| Goodput | Requests or tokens completed while satisfying the study's explicit latency and error SLO. |
| ISL / OSL | Input and requested output sequence lengths. Actual tokenizer output and stop-token behavior are disclosed when they differ. |
Controls for comparative studies
Model and quality
- Same checkpoint, tokenizer, prompt set, and generation settings
- Same quantization when comparing frameworks
- Quality evaluation when comparing quantizations
Serving environment
- GPU, topology, driver, CUDA/ROCm, container, and framework version
- KV dtype, tensor parallelism, memory limits, caching, scheduler, graphs, and token budget
- Warm-up and exclusion policy
Traffic
- Exact or distributed ISL/OSL, concurrency, request count, and seed
- Open-loop or closed-loop arrival model
- Streaming mode, timeouts, retries, and client location
Run validity
A run is valid when
- The intended model and configuration are confirmed from server logs.
- Warm-up is separated from measurement.
- Errors, timeouts, OOMs, and restarts are reported.
- Measured samples and percentile method are documented.
- Raw request metrics and configuration metadata are preserved.
A comparison is not generalized when
- Models or quantizations differ across arms.
- One arm uses caching or scheduler features the other does not.
- Sample count is too small for stable tail percentiles.
- Capacity interruption leaves unmatched cells.
- Network or client placement materially differs.
Reproduce a run
Each published study links its raw CSV/JSON and names the exact workload. Deployment manifests, benchmark harnesses, result artifacts, and experiment notes live in the TokenLabs repository. Study-specific commands take precedence over this general contract.