Optimizers propose. Real execution measures. InferCrane qualifies or rejects.AIConfigurator output reduces a large search space. It is
modeled evidence, not a benchmark.
Catalog profiles are unmeasured. A result becomes measured only after AIPerf or Replay executes
against the exact serving tuple. Release Guard may then mark that tuple qualified or rejected;
changing an identity-defining input makes the evidence stale.
Exact comparison identity
Every published row must bind all of these dimensions:
A result for Mistral 7B on one L40S does not transfer to Qwen, Llama, DeepSeek, an embedding model,
another precision, another runtime, or another provider.
Model-diverse qualification corpus
The initial corpus represents materially different execution behavior:
Frontier MoE entries are admitted only after the exact artifact, license, engine release, GPU
topology and upstream support are verified. InferCrane must not synthesize a deployable single-GPU
recipe merely because a model is popular.
Workload matrix
For generation runtimes, run the same immutable candidate through:- interactive streaming at concurrency 1;
- balanced streaming at concurrency 8;
- sustained throughput at concurrency 32;
- bounded overload at concurrency 128;
- buffered responses;
- 8,192-token long-context requests;
- 1,024-token long generations;
- a repeated-prefix trace when prefix/KV reuse is claimed;
- tool and structured-output conformance when the model/runtime pair claims them.
Repetition and statistics
For a publishable performance row:- Warm the exact runtime according to the declared protocol.
- Execute at least three independent measured campaigns.
- Use at least 256 requests for ordinary generation rows unless the long-context cost is explicitly bounded and disclosed.
- Publish p50, p95 and p99 when sample size permits, plus median and range across campaigns.
- Retain raw, content-safe AIPerf output and an immutable reproduction command.
- Treat any changed model, artifact, runtime, topology, hardware, workload or cache state as a new row.
Three different comparisons
Do not collapse these into one leaderboard.Engine comparison
Run vLLM, SGLang, TensorRT-LLM, or a Dynamo topology on the same owned hardware with the same model, precision and workload. This isolates serving-engine and configuration effects.Hosted-provider comparison
Call the public endpoint exposed by a provider such as Baseten or Fireworks using the same available model and request corpus. Report the product as observed: latency, throughput, errors, billed or advertised price, region visibility and API behavior. Do not claim identical hidden hardware or runtime configuration.Operational comparison
Measure deployment readiness, update availability, rollback, failure recovery, scale convergence and cleanup. These are control-plane/product properties and should not be mixed into token throughput.Current evidence boundary
The repository currently archives a real AWS Mistral 7B baseline on A10G and a partial L40S matrix in AWS real-infrastructure evidence. Those rows are useful calibration points, not a Baseten or Fireworks comparison and not proof for another model. The model-diverse catalog and qualification paths cover Qwen, Llama, DeepSeek Distill, Qwen Coder, BGE, Gemma/Qwen vision and Granite. Real hardware results remain pending until each exact tuple is executed. No competitor number is copied from a marketing page into InferCrane evidence.Commands
Generate candidates without provisioning:Upstream tools
- AIConfigurator proposes modeled configurations.
- AIPerf executes model and trace workloads.
- Dynamo deployment guidance demonstrates why configuration, deployment and benchmark stages must remain distinct.
- AIConfigurator to AISimulate transition defines the migration gates InferCrane will enforce before changing estimator adapters.