Open-weight optimization PoC
This PoC proves the system, not a universal speed claim. It accepts any immutable open-weight model, creates conservative runtime baselines, measures the same workload against isolated candidates, and promotes only evidence that survives quality, SLO, cost, and Release Guard checks. The complete operator-driven search design and research map live in Model-agnostic optimization engine.Stage 0: rehearse on a laptop
Two checks run on a laptop with Docker and no datacenter GPU. Neither produces GPU performance evidence. Check a generated kernel for correctness on CPU. The hidden suite runs in a no-network Linux container with Triton’s interpreter, and the sanitizer and timing stages are reported as not run:llama-server and aiperf on PATH. Control drift above 10% means the laptop was too busy
for a fair comparison.
Stage 1: local and free
Run the contract and evidence tests without allocating a GPU:proposed-unmeasured generic
baselines with no model-specific runtime arguments. A missing immutable revision, runtime version,
or locally qualified provider/runtime path must fail closed.
If a small OpenAI-compatible model is already running locally, exercise the AIPerf adapter through
the normal benchmark API. Use three runs for the smoke result:
Stage 2: one controlled GPU
Start this only after an operator explicitly approves paid resources. Use the same model revision, runtime image, token shape, seed, and SLO for every lane. Measure concurrency1, 4, and 8 for
both the baseline and one candidate, with three runs during development and five for a decision.
The first candidate should be a runtime recipe with a plausible large effect: batching budget,
prefix caching for a workload with real prefix reuse, quantization with semantic-quality evidence,
or model-native speculative decoding. Do not start with a custom kernel.
Emit infercrane.dev/optimization-evidence/v1 and evaluate it locally:
Mechanism ladder
Use the cheapest, broadest mechanisms first:- Runtime scheduling: continuous batching, chunked prefill, queue and batch-token bounds.
- Workload reuse: prefix/KV caching only when the profile proves reusable prefixes.
- Artifact changes: FP8, AWQ, GPTQ, or NVFP4 with immutable build provenance and quality gates.
- Model-native decoding: MTP, EAGLE, DFlash, or another exact verifier pairing with acceptance data.
- Topology: tensor/expert parallelism, disaggregated prefill/decode, KV-aware routing.
- Custom CUDA/Triton/Rust-CUDA kernel only after a profile identifies a dominant kernel and an Amdahl calculation shows material end-to-end headroom.
--require-measured, compile on the exact target GPU,
and pass paired AIPerf plus quality gates.
PoC exit criteria
The unpaid stage is complete when generic proposals are deterministic, mutable revisions fail, AIPerf multi-run fixtures produce pooled metrics and confidence, and evidence evaluation is repeatable. The first paid stage is complete only when baseline and candidate each have all required lanes, repeated runs, zero identity drift, passing semantic quality, sourced cost, and cleanup evidence. Production promotion remains a separate explicit action. The first measured Qwen3-0.6B/H100 campaign, including accepted vLLM evidence, a rejected handwritten CUDA candidate, and a rejected SGLang tuple, is recorded inevidence/qwen3-0.6b-modal-2026-09-20.json,
evidence/qwen3-0.6b-cuda-h100-2026-09-20.json,
and evidence/qwen3-0.6b-sglang-screen-2026-09-20.json.
The remote-only integration of the 6.12B-request Chutes trace and paired
Qwen3-0.6B/Qwen3-1.7B L40S measurements are recorded in
evidence/chutes-public-workload-2026-09-20.json
and evidence/chutes-qwen-l40s-screen-2026-09-20.json.
Existing FastPath manifests can be inspected with: