Optimizer golden suite
The golden suite is a fixed evaluation of InferCrane’s optimizer. Each task replays recorded evidence, names the customer’s current setup as the baseline, and states the decision a careful reviewer would make from the raw numbers. A change to qualification, scoring, or the baseline rule that turns a known loss into a promotion fails the suite. The suite replays files. It starts no servers and allocates no GPUs, so it costs nothing to run.Run it
Run the golden suite
--output json for the full report, including each task’s evaluator reason. The command exits
non-zero on any wrong decision or false win. TestOptimizerGoldenSuite runs the same tasks as part of
go test, so make dev-check fails on the same regressions.
Current result
These numbers describe the evaluator on this task set, not a performance claim. The 44.3% comes from
two H200 screening receipts in which NEXTN speculative decoding lowered cost per 1M successful output
tokens against an SGLang control on one decode-saturation workload. Screening evidence uses 12 to 32
requests per lane and one run.
What the metrics mean
Every task ends in one of three decisions:promote: a candidate replaces the baseline.keep_baseline: the current setup stays.reject_all: nothing qualified, including the baseline, so no setup can be recommended.
A gain is known only when the baseline qualified. When the baseline fails its own SLO, the promoted
candidate has no comparable baseline score, so the gain is reported as unknown and left out of the
median instead of being estimated.
Measured GPU-hours leave out provisioning, model loading, and warmup, so they are a lower bound on
what an answer cost. Two tasks that share an evidence file each count it in full. The laptop task
counts the laptop’s GPU time. A lane with no successful requests or no recorded GPU seconds makes its
task’s GPU time unknown; the report counts those tasks and leaves them out of the average.
The baseline rule
The evaluator accepts two optional fields inselection_policy:
Declare the current setup
baseline_candidate_id is evaluated exactly as before: the best qualified score is
selected, and the input digest does not change. With a baseline, qualification still runs first, and
then:
- If the baseline qualified, the best other qualified candidate is promoted only when its relative gain on the objective is larger than the noise floor. Otherwise the baseline is kept.
- If the baseline failed qualification, the best qualified candidate replaces it, with an unknown gain.
- If nothing qualified, the decision is
reject_allandwinner_idis empty. - If the baseline scored zero, the relative gain is undefined and the baseline is kept.
sglang-0520-nextn-k4-bounded-graphs recipe spanned
4.9% in lane-mean cost across six H200 screening receipts, so a smaller lead cannot be told apart from
re-measurement noise. minimum_relative_gain can raise the floor, for example when a campaign measured
larger drift between its control runs. It cannot lower it: lowering the floor is a code change that
must keep this suite passing. Treat 5% as a minimum; one speculative-decoding recipe varied by 30%
between two receipts, both of which failed quality gates.
The result adds a baseline_decision object with the outcome, the kept or promoted candidate_id,
the best challenger_id, its relative_gain, the noise_floor, a reason, and the rule’s
algorithm_version (baseline-noise-floor-v1). Each candidate also reports baseline and
relative_gain_over_baseline. winner_id is the candidate to serve: the promoted candidate, the kept
baseline, or nothing.
Tasks
The manifest at
internal/optimizationevidence/testdata/golden/manifest.json holds the full reason for
each task. The H200 receipts live in evidence/. The H100 task uses the Qwen3.8-27B-FP8
screening evidence in internal/optimizationevidence/testdata/qwen38-modal-screening.json.
The Qwen3-0.6B H100 campaign summary in
evidence/qwen3-0.6b-modal-2026-09-20.json is not a
task. It has no request counts, quality gates, or GPU time, so expressing it as optimization
evidence would mean inventing those values.
Add a task
1
Start from recorded evidence
Use an
infercrane.dev/optimization-evidence/v1 file from a real run. Recorded receipts live in
docs/testing/evidence/. Put new laptop or fixture evidence in
internal/optimizationevidence/testdata/golden/. Never edit measured values or build evidence to
fit a case; boundary cases such as a gain of exactly 5% belong in unit tests.2
Decide before you run the evaluator
Check every candidate’s quality gates and every lane against the SLO. Then compute the objective
for each lane and its mean, and compare the best challenger with the baseline against the noise
floor. Write the decision and the numbers behind it down first, so the evaluator’s output cannot
shape the expectation.
3
Add the task to the manifest
Each task needs
id, evidence (relative to the manifest), baseline_candidate_id,
expected_decision, categories, and why. A promote task also needs expected_candidate_id
and may list must_not_win candidates, such as one that breaks the SLO. Categories are
clear_win, inside_noise, slo_gate, quality_gate, baseline_fails_slo,
laptop_rehearsal, and recorded_h100. The test fails if any category except
baseline_fails_slo loses its last task.4
Run the suite
Run
make test-optimizer-golden. If the evaluator disagrees with a new task, do not change the
expectation to match. Find out which one is wrong. A pull request that changes an existing
expectation must explain the new reasoning in that task’s why.Planning tasks
The evaluator decides which measured setup wins. The planner decides which setups get measured at all.planning.json (schema infercrane.dev/optimizer-planning-golden/v1) fixes the planner’s
choices: each task is a proposal request with a recorded
endpoint diagnosis and the
order a reviewer would measure setups in. The same command detects the schema and runs the planner
instead of the evaluator. TestPlanningGoldenSuitePasses runs it under go test.
A task fails when any of these differ from the expectation:
Current result: 31 of 31 plans as reviewed, rule set
diagnosis-technique-rules-v4. Version 2
widened the FP8 gate to the reviewed AMD Instinct GPUs; two MI300X tasks check that the same
diagnosis chooses the same plan there and that RunPod’s MI300X name resolves to its 192 GB memory.
Version 3 reads the trace analysis of a GPU profile capture; six gpu_trace tasks check it.
Version 4 adds AMD’s AITER kernels for vLLM on reviewed AMD Instinct GPUs
(accelerator.kernels_off_by_default); five accelerator_kernels tasks check where it is chosen
and where it must not be, and the two earlier MI300X tasks now also forbid it.
The
gpu_trace tasks are written by hand from the trace analysis rules. They are not recorded
captures: no real GPU trace has been planned from yet.
The two RTX 4090 tasks in recorded_gpu_run come from the first real GPU run (Qwen3-0.6B, Vast through SkyPilot, a replay
at one request at a time). vLLM defaults served 246 tok/s; n-gram speculation served 188 tok/s, and
an FP8 KV cache with 2,048-token chunks served 193 tok/s. 2,048 is already vLLM’s default under
70 GB, so that arm changed only the KV cache. Both were hand-picked before the planner read a
diagnosis. The tasks plan on RunPod Pods, because operator-declared clouds such as Vast are
registered only when a control plane starts.
To add a planning task, write the diagnosis from recorded readiness output, decide the order from
the rule table before you run the planner, and record the reason in why. Categories are
kv_pressure, queueing, host_bound, prefill, prefix_reuse, decode_low_concurrency,
decode_high_concurrency, gpu_profile, gpu_trace, not_ready, reliability,
recorded_gpu_run, publisher_checkpoint, pinned_image_setups, and accelerator_kernels, and
the test fails if any of them loses its last task. A change to a rule, priority, or flag value must
bump the rule set version (now diagnosis-technique-rules-v4) and explain each changed expectation
in its task’s why.
Limits
- The tasks replay screening evidence, mostly from one model and one workload family on H200.
- The win rate rests on three expected promotions. Add tasks before reading much into it.
- The suite checks decisions on recorded evidence. It does not check that a new campaign measures correctly; the campaign’s own control runs and noise-floor checks do that.
- Planning tasks check which setups are measured and in what order, not whether they win. Only the two RTX 4090 tasks rest on a measured outcome; the rest encode the rule table.