Reproducible benchmarking
InferCrane delegates load generation and measurement to AIPerf. It does not contain a second load generator.constant, poisson, and gamma arrival patterns model different request timing. The request rate
and pattern must be supplied together. InferCrane pools the record-level results, retains the run
count and load shape, and stores Student-t 95% intervals and coefficient of variation for
throughput, tail latency, and goodput when at least two runs contain the metric.
Profiles define load shape, not runtime tuning claims. interactive, balanced, throughput,
buffered, long-context, long-generation, and overload record their immutable
benchmark-profile-v1 identity inside the workload evidence. Explicit token or concurrency flags
override profile defaults and remain visible in the result.
For a bounded qualification sweep across concurrency 1, 8, 32, and 128 plus long prompt,
long-generation, streaming, buffered, and overload shapes:
.infercrane/performance/RUN_ID/matrix.json. Set
INFERCRANE_BENCHMARK_TTFT_SLO_MS, INFERCRANE_BENCHMARK_TPOT_SLO_MS, and/or
INFERCRANE_BENCHMARK_LATENCY_SLO_MS to compute measured goodput: successful requests per second
that satisfy every declared threshold. A zero threshold means no SLO was declared and goodput stays
unavailable rather than being invented.
The profile matrix deliberately changes workload shape by objective, so its rows are not a
scaling curve. Use the separate same-workload campaign when comparing concurrency behavior:
1, 8, 32, and 128. The resulting
.infercrane/performance/RUN_ID-concurrency-sweep/sweep.json is the only one of these two artifacts
that supports a concurrency scaling comparison. Override the bounded defaults with
INFERCRANE_BENCHMARK_SWEEP_* variables; every override remains in persisted workload evidence.
The CLI submits the benchmark through the authenticated control-plane API. The control plane resolves the active immutable revision, runs AIPerf against the logical InferCrane endpoint, and persists the result. A fixed dataset seed defaults to 17 and can be changed with --random-seed. AIPerf 0.12.0 is pinned so flag semantics and exports cannot drift between candidates.
Release Guard validation can benchmark an isolated candidate explicitly:
null. Goodput remains null without SLO thresholds.
Cost metadata becomes available only when fresh, revision-bound deployment_hourly_rate evidence
covers the run; otherwise it contains available: false and a reason. None are fabricated.
What “fast” means
InferCrane does not promise one globally fastest configuration. It measures the exact combination and lets the operator choose the objective:
Runtime arguments are immutable revision data. Tune a candidate, run the same profile against active
and candidate revisions, and let Release Guard compare matching evidence before promotion. This is
how InferCrane can provide a fast qualified default without turning an upstream vLLM or SGLang flag
into an unsupported universal claim.
For reviewed generation models, scaffold two candidate projects and preserve their different runtime
arguments explicitly:
RECOMMENDED only when candidates share one exact workload digest and the required
metric - and sourced hourly cost for cost efficiency - is present. Unlike workload shapes remain
UNRANKED.
The workload digest covers the load that was sent, including the run count and warmup. Results that
are stored beside the load are excluded: the confidence_95 intervals from repeated runs and an
optimization campaign’s measured control comparison. Two repeated benchmarks of the same load
therefore compare with each other. Lab versions before inference-lab-v4 hashed those intervals,
which gave every multi-run benchmark a workload of its own and kept it out of every ranking.
Optimization campaigns use a stricter method than a single infercrane benchmark command: warmup,
three runs per arm, the campaign control interleaved with each candidate, and a noise floor before
a gain counts. See how a campaign measures candidates.
InferCrane normalizes the unit carried by each AIPerf latency record to milliseconds and rejects
unknown units instead of silently mislabeling evidence. Percentiles reconstructed from those records
use the nearest-rank definition. Small samples remain visibly weak evidence: Release Guard separately
requires the persisted minimum request count, and matching workload/tool identity, before a comparison
can pass.
InferCrane asks AIPerf for the records export level. This contains per-request measurements but not the raw request/response export, so prompt and generated content are not persisted. Results remain in the InferCrane PostgreSQL database and are never uploaded by default.
The control-plane image pins AIPerf 0.12.0. For a standalone control plane, install it with
pipx install aiperf==0.12.0; the benchmark adapter rejects another version rather than silently
mixing tool semantics.
Each AIPerf run starts in its own process group. AIPerf starts worker processes, and when a model
server dies mid-run the main process can exit while those workers keep running. InferCrane then
waits at most 30 seconds for them and stops the whole group, so a measurement cannot hang on them.
A cancelled measurement stops the group at once. A run whose main process exited successfully keeps
its result; a failed run keeps its error.