​

Inference optimization execution plan

​

Outcome

model + hardware + workload + SLO + cost boundary
                         ↓
discover compatible upstream mechanisms
                         ↓
propose a small candidate set
                         ↓
explicitly approve bounded infrastructure spend
                         ↓
benchmark + replay + semantic evaluation
                         ↓
rank by SLO, goodput, cost, and quality
                         ↓
persist a qualified Serving Plan
                         ↓
Release Guard + human-approved promotion or rollback
                         ↓
record predicted versus actual outcome

​

Current baseline

• reviewed, immutable model recipes and vLLM balanced/interactive/throughput starting profiles;
• replaceable optimizer.Source and deterministic offline optimize propose;
• optional pinned AIConfigurator 0.11.0 modeled-candidate source behind that boundary, including a real Linux package contract and explicit catalog fallback;
• exact provider/runtime compatibility evidence with simulated candidates excluded by default;
• immutable revisions and DeploymentSpecs;
• AIPerf benchmark execution and persisted TTFT, TPOT, throughput, goodput, GPU-utilization, and sourced-cost evidence;
• exact-workload Inference Lab comparisons;
• evaluator-neutral signed semantic-quality evidence;
• advisory Serving Plans, Release Guard, promotion, rejection, rollback, and Inference Passports;
• experimental Dynamo aggregated/disaggregated topology, KV-aware routing, and bounded KVBM intent.

​

AISimulate decision

• the published aisimulate Python artifact is 0.1.0.dev1;
• the implementation currently lives with Dynamo and the proposed standalone repository is not publicly accessible;
• upstream’s accepted transition plan targets AISimulate 0.12.0, AIConfigurator 0.12.0, and Dynamo 1.5.0 for 2026-09-16;
• upstream explicitly requires public packages, documentation, result-schema mapping, parity tests, performance comparison, critical integration validation, and rollback instructions before deprecating AIConfigurator.

1. Keep the pinned AIConfigurator adapter optional and out of the control-plane runtime dependency graph; never import its objects into InferCrane domain or persistence types.
2. Do not ship aisimulate==0.1.0.dev1 in an InferCrane release.
3. Keep the existing optimizer.Source as the stable InferCrane boundary.
4. Build the campaign and evidence pipeline independently of either upstream package.
5. Replace the candidate source with AISimulate only after the upstream readiness gates below pass.

​

AISimulate admission gates

• stable public release, repository, SDK/CLI documentation, and result schema;
• Apache-2.0 artifact and path-level transitive-license review;
• immutable package/image pins, hashes, SBOM, vulnerability scan, and rollback pin;
• clean installation and smoke tests on InferCrane’s supported execution platform;
• explicit model, GPU, runtime, precision, and topology support inventory;
• deterministic bounded execution with CPU, memory, time, and output-size limits;
• no provider credentials, prompt bodies, response bodies, or customer secrets in inputs or output;
• malformed, partial, unknown-enum, timeout, crash, and version-skew fixture coverage;
• AIC/AISimulate parity results within upstream’s published tolerance;
• InferCrane calibration comparing modeled versus measured results on at least three model families, two workload shapes, and two hardware classes;
• modeled rows remain labeled MODELED and can never satisfy a measured Release Guard requirement.

​

Adapter shape

InferCrane OptimizationRequest v1
                ↓ JSON stdin / bounded process contract
AISimulateSource adapter
                ↓
versioned raw result retained as private provenance
                ↓
InferCrane CandidateProposal v1 (MODELED)

​

Dependency-ordered milestones

​

M0 — Conservative proposal source

• reviewed-catalog source;
• immutable input digest and candidate identity;
• objective-aware ordering;
• strict missing-evidence responses;
• offline CLI and loadable candidate files;
• tests, ADR, capability matrix, and user documentation.

​

M0.5 — Pinned AIConfigurator candidate source

• out-of-process, versioned and bounded estimator protocol;
• exact aiconfigurator==0.11.0 and plotext==5.3.2 compatibility tuple;
• credential-scrubbed, offline-by-default execution;
• explicit modeled evidence and upstream result digest;
• executable-topology filtering and single mutation-owner preservation;
• catalog fallback, doctor UX, real-package contract, multi-model catalog regression, ADR and docs.

​

M1 — Durable optimization campaigns

proposed → awaiting_approval → provisioning → ready → measuring
         → validating → ranked → guarding → guard_passed | rejected | inconclusive
         → approved → promoted → observed
         → cleaned

infercrane optimize create MODEL --objective interactive --profile interactive \
  --max-ttft-p95-ms 200 --max-hourly-cost 3 --max-candidates 3
infercrane optimize inspect CAMPAIGN
infercrane optimize approve CAMPAIGN --max-cost-usd 20
infercrane optimize watch CAMPAIGN
infercrane optimize results CAMPAIGN
infercrane optimize promote CAMPAIGN CANDIDATE --yes
infercrane optimize cleanup CAMPAIGN

• proposal and inspection are free of provider mutation;
• paid execution requires explicit candidate count, maximum spend, expiry, and approval identity;
• every candidate gets a durable idempotency key and immutable revision;
• client and control-plane restarts resume safely;
• partial candidate failure does not erase successful evidence;
• cancellation stops future work and deterministically cleans run-owned resources;
• promotion remains a separate human-authorized Release Guard action;
• predicted and actual outcomes are recorded separately.
• the selected Lab ranking identity is persisted before Release Guard runs, so a crash cannot merge ranking and release evidence into one ambiguous boundary;

• crash/restart, duplicate request, cancellation, stale lease, partial benchmark, failed quality gate, promotion race, and cleanup-to-zero tests pass;
• one local fixture campaign completes from proposal through rejected bad candidate and cleanup;
• no unapproved provider mutation is reachable.

​

M2 — Versioned optimization capability registry

runtime + exact version + model architecture + artifact precision + accelerator
    → supported mechanism + constraints + compiler + qualification state

• vLLM and SGLang continuous batching;
• prefix caching;
• chunked prefill and scheduler token budgets;
• attention backend selection;
• weight and KV-cache precision;
• speculative decoding;
• aggregated versus prefill/decode topology;
• KV reuse/offload.

• exact-version compatibility, enum expansion, conflicting mechanisms, model mismatch, accelerator mismatch, and downgrade tests pass;
• every produced argument links to a capability descriptor and evidence source.

​

M3 — AISimulate modeled-candidate source

• exact AISimulate/Dynamo versions;
• model/system database identity;
• input digest and output digest;
• modeled TTFT, TPOT, throughput, and topology;
• unsupported fields and warnings;
• model age and calibration error where available.

• isolated adapter conformance passes;
• the top-N result is deterministic for pinned inputs;
• unsupported tuples fail closed;
• modeled-to-measured error and top-k recall are recorded, not hidden;
• no modeled candidate can bypass AIPerf and quality gates.

​

M4 — Immutable quantized-artifact pipeline

1. FP8 weight/activation candidates on compatible Hopper/Ada paths;
2. AWQ/GPTQ/Marlin weight-only candidates for memory-constrained serving;
3. KV-cache quantization only after long-context quality and stability qualification.

• corrupted output, incompatible GPU/runtime, calibration mismatch, quality regression, builder crash, duplicate build, and artifact cleanup tests pass;
• quantized candidates cannot promote without signed semantic-quality evidence;
• exact baseline and candidate AIPerf workloads are comparable.

​

M5 — Speculative-decoding candidate adapter

• acceptance rate and accepted tokens per step;
• TTFT, TPOT, output throughput, goodput, memory, and failure rate;
• semantic-quality evidence and deterministic structured/tool-call behavior;
• performance across short, long-generation, and overload workloads.

• incompatible draft/target pairs are rejected before GPU creation;
• low acceptance or increased tail latency causes candidate rejection;
• speculative streaming, cancellation, and rollback pass real-runtime qualification.

​

M6 — LMCache and Dynamo/NIXL qualification

1. engine-native prefix cache baseline;
2. LMCache local CPU/disk reuse;
3. Dynamo KV-aware routing;
4. Dynamo prefill/decode with NIXL transfer;
5. KVBM where the exact backend combination is qualified.

• stale, corrupted, cross-model, cross-tenant, storage-full, worker-loss, transfer-timeout, and topology-downgrade tests pass;
• real multi-GPU results show which workload shapes benefit and which regress.

​

M7 — TensorRT-LLM runtime adapter

• path-level licensing and transitive redistribution review passes;
• engine build and runtime images are digest-pinned;
• build failure, architecture mismatch, CUDA mismatch, plugin mismatch, and rollback tests pass;
• measured benefit survives quality and total-cost comparison including engine-build amortization.

​

M8 — llm-d and AIBrix composition

​

Qualification corpus

• dense instruction: Mistral 7B and Llama 3.1 8B;
• reasoning/distilled: DeepSeek R1 Distill 7B;
• coding: Qwen2.5 Coder 7B;
• embeddings: BGE-M3;
• multimodal: Gemma 3 4B or Qwen2.5-VL 7B;
• later multi-GPU MoE: one current supported DeepSeek, Kimi, or GLM family only after exact runtime, license, and hardware qualification.

• interactive short prompt;
• repeated shared prefix;
• long context;
• long generation;
• structured output and tool calling;
• concurrency 1, 8, 32, and bounded overload;
• burst and steady-state arrival patterns.

​

Product metrics and stopping rules

• TTFT p50/p95/p99;
• TPOT p50/p95/p99;
• request and output-token throughput;
• SLO goodput and goodput per sourced currency unit;
• GPU utilization and memory headroom;
• cold and warm readiness;
• semantic-quality score and pass/fail;
• modeled versus measured error;
• provision, runtime, and request failure rates;
• realized versus predicted cost after promotion.

1. it beats or matches a stock qualified baseline on its declared objective;
2. configured SLO and quality gates pass;
3. results repeat within a declared tolerance;
4. failure, rollback, and cleanup behavior pass;
5. the exact software, model, artifact, hardware, workload, and cost source are reproducible;
6. at least one workload where it should not be used is documented.

​

Recommended execution order

1. M1 durable optimization campaigns.
2. M2 capability registry.
3. Maintain AIConfigurator calibration and prepare AISimulate adapter fixtures, but wait for stable upstream admission gates.
4. M4 quantized artifacts with LLM Compressor.
5. Activate M3 AISimulate when stable and calibrate it against real measurements.
6. M5 speculative decoding.
7. M6 LMCache plus real Dynamo/NIXL qualification.
8. M7 narrow TensorRT-LLM support.
9. M8 llm-d/AIBrix only from demonstrated Kubernetes demand.

​

Primary upstream references

• AIConfigurator transition decision: https://github.com/ai-dynamo/aiconfigurator/issues/1517
• AIConfigurator: https://github.com/ai-dynamo/aiconfigurator
• Dynamo: https://github.com/ai-dynamo/dynamo
• vLLM: https://github.com/vllm-project/vllm
• vLLM Speculators: https://github.com/vllm-project/speculators
• LLM Compressor: https://github.com/vllm-project/llm-compressor
• ModelOpt: https://github.com/NVIDIA/Model-Optimizer
• LMCache: https://github.com/LMCache/LMCache
• TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM
• llm-d: https://github.com/llm-d/llm-d
• AIBrix: https://github.com/vllm-project/aibrix
• Wafer GPU performance engineering resources: https://github.com/wafer-ai/gpu-perf-engineering-resources