Skip to main content

​
Evidence-gated optimization

InferCrane separates proposing a configuration from proving it. The optimizer starts with reviewed model metadata and locally or really qualified provider/runtime compatibility. When the exact optional estimator toolchain is available, it uses AIConfigurator to reduce the candidate space. Otherwise it creates conservative reviewed catalog candidates. Neither path contacts a control plane or provisions paid capacity. Check the optional estimator once:
Install its exact compatible tuple in an isolated Python 3.11–3.13 environment:
The plotext pin is required: AIConfigurator 0.11.0 is not compatible with plotext 6.0.0 even though its published dependency range permits it. optimize doctor validates the complete tuple, not just the top-level package.
The default --source auto tries the pinned AIConfigurator adapter and visibly falls back to the reviewed catalog when the optional tool is unavailable. Use --source aiconfigurator to require the estimator or --source catalog for deterministic reviewed starting profiles only. The result is a bounded candidate set. Each spec pins:
  • the exact model revision;
  • runtime engine and version;
  • provider adapter, region, and accelerator;
  • reviewed runtime arguments;
  • compute mode, scale bounds, and routing strategy.
The command refuses to overwrite a non-empty candidate directory. Use --output json to archive the input digest, source, compatibility evidence, required evidence, and limitations with the specs.
A proposal is not a benchmark result or recommendation. AIConfigurator rows are marked modeled and proposed-modeled-unqualified; catalog rows are marked unmeasured and proposed-unmeasured. InferCrane never presents an estimate as measured or qualified evidence.

​
The simple and advanced paths

The common path needs an objective, not a collection of runtime flags:
Advanced users can bind the search to an observed content-free Replay digest, choose runtimes, control the workload profile and concurrency, require the estimator, and inspect the complete JSON provenance. Provider credentials are not passed to the estimator. Model metadata network access is off by default and requires --allow-model-metadata-network.

​
Create a durable campaign

Use propose for a completely offline preview. Use create to persist the same immutable proposal without provisioning anything:
Approval records a bounded authority window. It does not automatically promote a candidate. Campaign execution remains separately qualification-gated; canceling a campaign prevents future work and marks unfinished candidates stale. Closing the CLI does not remove persisted campaign state.

​
Plan an optimized artifact

Builders run outside InferCrane. The common path produces a strict immutable plan:
Available plan presets cover FP8, AWQ, GPTQ, NVIDIA-specific NVFP4, EAGLE-3, MTP, DFlash, and TensorRT engine builds. A preset is not a compatibility or speed claim. Every plan requires an exact tool version, digest-pinned builder, hardware constraints, immutable output attestation, exact candidate binding, and passing semantic quality evidence before qualification. Advanced users can provide the complete JSON plan with --plan-file.

​
Propose, prove, release

Use the generated files through the existing lifecycle:
When model precision or model artifacts differ, attach signed semantic evaluation evidence before promotion. When cost is part of the objective, attach sourced, dated cost evidence. Only an exact model, runtime, accelerator, provider, and canonical workload result supports a measured recipe.

​
What the first planner knows

The first source composes: It does not infer that a model fits an accelerator. It does not automatically enable quantization, speculative decoding, LMCache, disaggregated serving, FlashInfer, or TensorRT-LLM. Those mechanisms are version- and workload-specific and require their own compatibility and qualification evidence. Use --include-simulated only for development exploration; simulated candidates are excluded by default.

​
Upstream composition roadmap

InferCrane will reuse upstream execution and optimization work behind replaceable adapters: The durable rule is: the optimizer proposes; InferCrane proves. See ADR 0038 for the ownership and evidence boundary. The pinned adapter and migration decision is recorded in ADR 0039. Durable authority and artifact provenance are recorded in ADR 0040.

​
Benchmarks and competitor comparisons

AIConfigurator estimates help decide what to test; they are not publishable performance claims. Use the competitive benchmark methodology to compare InferCrane-managed serving with Baseten, Fireworks, or another provider on the same model artifact, precision, request dataset, protocol, concurrency, region class, warm/cold definition, and output token accounting. If an external provider does not expose the same artifact or configuration, label the result as a product comparison rather than an engine comparison.