Skip to main content

Reproducible benchmarking

InferCrane delegates load generation and measurement to AIPerf. It does not contain a second load generator.
Use a versioned workload profile when you want comparable evidence for a specific operating goal:
For repeatable optimization evidence, use independent AIPerf runs instead of treating one run as a stable result. Three runs are a useful engineering smoke test; five are the default target for a decision or public result:
constant, poisson, and gamma arrival patterns model different request timing. The request rate and pattern must be supplied together. InferCrane pools the record-level results, retains the run count and load shape, and stores Student-t 95% intervals and coefficient of variation for throughput, tail latency, and goodput when at least two runs contain the metric. Profiles define load shape, not runtime tuning claims. interactive, balanced, throughput, buffered, long-context, long-generation, and overload record their immutable benchmark-profile-v1 identity inside the workload evidence. Explicit token or concurrency flags override profile defaults and remain visible in the result. For a bounded qualification sweep across concurrency 1, 8, 32, and 128 plus long prompt, long-generation, streaming, buffered, and overload shapes:
The script refuses to send load without explicit approval, archives each control-plane result, validates failure bounds, and produces .infercrane/performance/RUN_ID/matrix.json. Set INFERCRANE_BENCHMARK_TTFT_SLO_MS, INFERCRANE_BENCHMARK_TPOT_SLO_MS, and/or INFERCRANE_BENCHMARK_LATENCY_SLO_MS to compute measured goodput: successful requests per second that satisfy every declared threshold. A zero threshold means no SLO was declared and goodput stays unavailable rather than being invented. The profile matrix deliberately changes workload shape by objective, so its rows are not a scaling curve. Use the separate same-workload campaign when comparing concurrency behavior:
It holds request count, input tokens, output tokens, seed, streaming mode, revision, and SLOs constant while running concurrency 1, 8, 32, and 128. The resulting .infercrane/performance/RUN_ID-concurrency-sweep/sweep.json is the only one of these two artifacts that supports a concurrency scaling comparison. Override the bounded defaults with INFERCRANE_BENCHMARK_SWEEP_* variables; every override remains in persisted workload evidence. The CLI submits the benchmark through the authenticated control-plane API. The control plane resolves the active immutable revision, runs AIPerf against the logical InferCrane endpoint, and persists the result. A fixed dataset seed defaults to 17 and can be changed with --random-seed. AIPerf 0.12.0 is pinned so flag semantics and exports cannot drift between candidates. Release Guard validation can benchmark an isolated candidate explicitly:
The candidate command selects one healthy ready replica deterministically and runs AIPerf directly from the control plane using its server-side worker credential. It never exposes that credential to the CLI. This is explicit synthetic validation, not shadow traffic: InferCrane does not copy any user request or conversation content. It creates additional inference work and may incur provider cost. Active and candidate results are eligible for Release Guard comparison only when AIPerf tool version and workload parameters match exactly. Each result records the immutable ModelArtifact, runtime and configuration, revision, provider, region, GPU type and grounded GPU count, compute mode, workload parameters, TTFT, TPOT, request latency, throughput, SLO goodput when declared, errors, timestamp, and exact reproduction command. The persisted command uses a portable local export prefix rather than the deleted temporary execution directory, and replaces the credential with its environment-variable reference. GPU utilization is attached only when measured DCGM evidence for the exact immutable revision overlaps the complete benchmark window; provider-reported or stale samples remain null. Goodput remains null without SLO thresholds. Cost metadata becomes available only when fresh, revision-bound deployment_hourly_rate evidence covers the run; otherwise it contains available: false and a reason. None are fabricated.

What “fast” means

InferCrane does not promise one globally fastest configuration. It measures the exact combination and lets the operator choose the objective: Runtime arguments are immutable revision data. Tune a candidate, run the same profile against active and candidate revisions, and let Release Guard compare matching evidence before promotion. This is how InferCrane can provide a fast qualified default without turning an upstream vLLM or SGLang flag into an unsupported universal claim. For reviewed generation models, scaffold two candidate projects and preserve their different runtime arguments explicitly:
Deploy them as isolated candidates, benchmark both with the same workload profile, then use Lab or Release Guard. Serving-profile names describe runtime configuration; benchmark-profile names describe load. Neither is evidence until the resulting measurements are persisted. Compare multiple measured configurations with the same workload:
Lab emits RECOMMENDED only when candidates share one exact workload digest and the required metric - and sourced hourly cost for cost efficiency - is present. Unlike workload shapes remain UNRANKED. The workload digest covers the load that was sent, including the run count and warmup. Results that are stored beside the load are excluded: the confidence_95 intervals from repeated runs and an optimization campaign’s measured control comparison. Two repeated benchmarks of the same load therefore compare with each other. Lab versions before inference-lab-v4 hashed those intervals, which gave every multi-run benchmark a workload of its own and kept it out of every ranking. Optimization campaigns use a stricter method than a single infercrane benchmark command: warmup, three runs per arm, the campaign control interleaved with each candidate, and a noise floor before a gain counts. See how a campaign measures candidates. InferCrane normalizes the unit carried by each AIPerf latency record to milliseconds and rejects unknown units instead of silently mislabeling evidence. Percentiles reconstructed from those records use the nearest-rank definition. Small samples remain visibly weak evidence: Release Guard separately requires the persisted minimum request count, and matching workload/tool identity, before a comparison can pass. InferCrane asks AIPerf for the records export level. This contains per-request measurements but not the raw request/response export, so prompt and generated content are not persisted. Results remain in the InferCrane PostgreSQL database and are never uploaded by default. The control-plane image pins AIPerf 0.12.0. For a standalone control plane, install it with pipx install aiperf==0.12.0; the benchmark adapter rejects another version rather than silently mixing tool semantics. Each AIPerf run starts in its own process group. AIPerf starts worker processes, and when a model server dies mid-run the main process can exit while those workers keep running. InferCrane then waits at most 30 seconds for them and stops the whole group, so a measurement cannot hang on them. A cancelled measurement stops the group at once. A run whose main process exited successfully keeps its result; a failed run keeps its error.