Skip to main content

Optimizer golden suite

The golden suite is a fixed evaluation of InferCrane’s optimizer. Each task replays recorded evidence, names the customer’s current setup as the baseline, and states the decision a careful reviewer would make from the raw numbers. A change to qualification, scoring, or the baseline rule that turns a known loss into a promotion fails the suite. The suite replays files. It starts no servers and allocates no GPUs, so it costs nothing to run.

Run it

Run the golden suite
The target runs the race-enabled Go tests and then prints both reports, the evaluator’s decisions and the planner’s choices:
Add --output json for the full report, including each task’s evaluator reason. The command exits non-zero on any wrong decision or false win. TestOptimizerGoldenSuite runs the same tasks as part of go test, so make dev-check fails on the same regressions.

Current result

These numbers describe the evaluator on this task set, not a performance claim. The 44.3% comes from two H200 screening receipts in which NEXTN speculative decoding lowered cost per 1M successful output tokens against an SGLang control on one decode-saturation workload. Screening evidence uses 12 to 32 requests per lane and one run.

What the metrics mean

Every task ends in one of three decisions:
  • promote: a candidate replaces the baseline.
  • keep_baseline: the current setup stays.
  • reject_all: nothing qualified, including the baseline, so no setup can be recommended.
A gain is known only when the baseline qualified. When the baseline fails its own SLO, the promoted candidate has no comparable baseline score, so the gain is reported as unknown and left out of the median instead of being estimated. Measured GPU-hours leave out provisioning, model loading, and warmup, so they are a lower bound on what an answer cost. Two tasks that share an evidence file each count it in full. The laptop task counts the laptop’s GPU time. A lane with no successful requests or no recorded GPU seconds makes its task’s GPU time unknown; the report counts those tasks and leaves them out of the average.

The baseline rule

The evaluator accepts two optional fields in selection_policy:
Declare the current setup
Evidence without baseline_candidate_id is evaluated exactly as before: the best qualified score is selected, and the input digest does not change. With a baseline, qualification still runs first, and then:
  1. If the baseline qualified, the best other qualified candidate is promoted only when its relative gain on the objective is larger than the noise floor. Otherwise the baseline is kept.
  2. If the baseline failed qualification, the best qualified candidate replaces it, with an unknown gain.
  3. If nothing qualified, the decision is reject_all and winner_id is empty.
  4. If the baseline scored zero, the relative gain is undefined and the baseline is kept.
The default noise floor is 5%. The unchanged sglang-0520-nextn-k4-bounded-graphs recipe spanned 4.9% in lane-mean cost across six H200 screening receipts, so a smaller lead cannot be told apart from re-measurement noise. minimum_relative_gain can raise the floor, for example when a campaign measured larger drift between its control runs. It cannot lower it: lowering the floor is a code change that must keep this suite passing. Treat 5% as a minimum; one speculative-decoding recipe varied by 30% between two receipts, both of which failed quality gates. The result adds a baseline_decision object with the outcome, the kept or promoted candidate_id, the best challenger_id, its relative_gain, the noise_floor, a reason, and the rule’s algorithm_version (baseline-noise-floor-v1). Each candidate also reports baseline and relative_gain_over_baseline. winner_id is the candidate to serve: the promoted candidate, the kept baseline, or nothing.

Tasks

The manifest at internal/optimizationevidence/testdata/golden/manifest.json holds the full reason for each task. The H200 receipts live in evidence/. The H100 task uses the Qwen3.8-27B-FP8 screening evidence in internal/optimizationevidence/testdata/qwen38-modal-screening.json. The Qwen3-0.6B H100 campaign summary in evidence/qwen3-0.6b-modal-2026-09-20.json is not a task. It has no request counts, quality gates, or GPU time, so expressing it as optimization evidence would mean inventing those values.

Add a task

1

Start from recorded evidence

Use an infercrane.dev/optimization-evidence/v1 file from a real run. Recorded receipts live in docs/testing/evidence/. Put new laptop or fixture evidence in internal/optimizationevidence/testdata/golden/. Never edit measured values or build evidence to fit a case; boundary cases such as a gain of exactly 5% belong in unit tests.
2

Decide before you run the evaluator

Check every candidate’s quality gates and every lane against the SLO. Then compute the objective for each lane and its mean, and compare the best challenger with the baseline against the noise floor. Write the decision and the numbers behind it down first, so the evaluator’s output cannot shape the expectation.
3

Add the task to the manifest

Each task needs id, evidence (relative to the manifest), baseline_candidate_id, expected_decision, categories, and why. A promote task also needs expected_candidate_id and may list must_not_win candidates, such as one that breaks the SLO. Categories are clear_win, inside_noise, slo_gate, quality_gate, baseline_fails_slo, laptop_rehearsal, and recorded_h100. The test fails if any category except baseline_fails_slo loses its last task.
4

Run the suite

Run make test-optimizer-golden. If the evaluator disagrees with a new task, do not change the expectation to match. Find out which one is wrong. A pull request that changes an existing expectation must explain the new reasoning in that task’s why.
To record a new laptop task, run the rehearsal on a quiet machine:
Keep a run only when the drift between its two control runs is below 10%. The checked-in run drifted 1.7%. An earlier attempt on a busy machine drifted 30% and was discarded.

Planning tasks

The evaluator decides which measured setup wins. The planner decides which setups get measured at all. planning.json (schema infercrane.dev/optimizer-planning-golden/v1) fixes the planner’s choices: each task is a proposal request with a recorded endpoint diagnosis and the order a reviewer would measure setups in. The same command detects the schema and runs the planner instead of the evaluator. TestPlanningGoldenSuitePasses runs it under go test. A task fails when any of these differ from the expectation: Current result: 31 of 31 plans as reviewed, rule set diagnosis-technique-rules-v4. Version 2 widened the FP8 gate to the reviewed AMD Instinct GPUs; two MI300X tasks check that the same diagnosis chooses the same plan there and that RunPod’s MI300X name resolves to its 192 GB memory. Version 3 reads the trace analysis of a GPU profile capture; six gpu_trace tasks check it. Version 4 adds AMD’s AITER kernels for vLLM on reviewed AMD Instinct GPUs (accelerator.kernels_off_by_default); five accelerator_kernels tasks check where it is chosen and where it must not be, and the two earlier MI300X tasks now also forbid it. The gpu_trace tasks are written by hand from the trace analysis rules. They are not recorded captures: no real GPU trace has been planned from yet. The two RTX 4090 tasks in recorded_gpu_run come from the first real GPU run (Qwen3-0.6B, Vast through SkyPilot, a replay at one request at a time). vLLM defaults served 246 tok/s; n-gram speculation served 188 tok/s, and an FP8 KV cache with 2,048-token chunks served 193 tok/s. 2,048 is already vLLM’s default under 70 GB, so that arm changed only the KV cache. Both were hand-picked before the planner read a diagnosis. The tasks plan on RunPod Pods, because operator-declared clouds such as Vast are registered only when a control plane starts. To add a planning task, write the diagnosis from recorded readiness output, decide the order from the rule table before you run the planner, and record the reason in why. Categories are kv_pressure, queueing, host_bound, prefill, prefix_reuse, decode_low_concurrency, decode_high_concurrency, gpu_profile, gpu_trace, not_ready, reliability, recorded_gpu_run, publisher_checkpoint, pinned_image_setups, and accelerator_kernels, and the test fails if any of them loses its last task. A change to a rule, priority, or flag value must bump the rule set version (now diagnosis-technique-rules-v4) and explain each changed expectation in its task’s why.

Limits

  • The tasks replay screening evidence, mostly from one model and one workload family on H200.
  • The win rate rests on three expected promotions. Add tasks before reading much into it.
  • The suite checks decisions on recorded evidence. It does not check that a new campaign measures correctly; the campaign’s own control runs and noise-floor checks do that.
  • Planning tasks check which setups are measured and in what order, not whether they win. Only the two RTX 4090 tasks rest on a measured outcome; the rest encode the rule table.