> ## Documentation Index
> Fetch the complete documentation index at: https://docs.infercrane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimizer golden suite

> A fixed set of recorded decisions that every change to the optimization evaluator must keep getting right.

# Optimizer golden suite

The golden suite is a fixed evaluation of InferCrane's optimizer. Each task replays recorded
evidence, names the customer's current setup as the baseline, and states the decision a careful
reviewer would make from the raw numbers. A change to qualification, scoring, or the baseline rule
that turns a known loss into a promotion fails the suite.

The suite replays files. It starts no servers and allocates no GPUs, so it costs nothing to run.

## Run it

```bash title="Run the golden suite" theme={"theme":"css-variables"}
make test-optimizer-golden
```

The target runs the race-enabled Go tests and then prints both reports, the evaluator's decisions
and the [planner's choices](#planning-tasks):

```bash theme={"theme":"css-variables"}
infercrane optimize evidence golden \
  --file internal/optimizationevidence/testdata/golden/manifest.json
infercrane optimize evidence golden \
  --file internal/optimizationevidence/testdata/golden/planning.json
```

Add `--output json` for the full report, including each task's evaluator reason. The command exits
non-zero on any wrong decision or false win. `TestOptimizerGoldenSuite` runs the same tasks as part of
`go test`, so `make dev-check` fails on the same regressions.

## Current result

| Metric | Value |
| - | - |
| Tasks | 9, all decided correctly |
| Win rate | 100% (3 of 3 expected promotions) |
| False wins | 0 |
| Median gain over the baseline | 44.3% (2 promotions with a known gain, 1 unknown) |
| Measured GPU-hours per answer | 0.090 |

These numbers describe the evaluator on this task set, not a performance claim. The 44.3% comes from
two H200 screening receipts in which NEXTN speculative decoding lowered cost per 1M successful output
tokens against an SGLang control on one decode-saturation workload. Screening evidence uses 12 to 32
requests per lane and one run.

## What the metrics mean

Every task ends in one of three decisions:

* `promote`: a candidate replaces the baseline.
* `keep_baseline`: the current setup stays.
* `reject_all`: nothing qualified, including the baseline, so no setup can be recommended.

| Metric | Definition |
| - | - |
| Win rate | Correct promotions divided by tasks that expect a promotion. A promotion is correct only when it names the expected candidate. |
| False win | The evaluator promoted a candidate in a task that expects `keep_baseline` or `reject_all`, or promoted a candidate in the task's `must_not_win` list. One false win fails the suite. |
| Missed win | A task expects a promotion and the evaluator kept the baseline or rejected everything. It is a wrong decision, so it also fails the suite. |
| Median gain over the baseline | The median relative improvement of the promoted candidate over the baseline, on the objective the evidence declares. Positive means better: more throughput, or lower cost or latency. |
| Measured GPU-hours per answer | GPU time inside the measured windows of every lane of every candidate in a task's evidence (`gpu_seconds_per_successful_request` × `successful_requests`), averaged over tasks. |

A gain is known only when the baseline qualified. When the baseline fails its own SLO, the promoted
candidate has no comparable baseline score, so the gain is reported as unknown and left out of the
median instead of being estimated.

Measured GPU-hours leave out provisioning, model loading, and warmup, so they are a lower bound on
what an answer cost. Two tasks that share an evidence file each count it in full. The laptop task
counts the laptop's GPU time. A lane with no successful requests or no recorded GPU seconds makes its
task's GPU time unknown; the report counts those tasks and leaves them out of the average.

## The baseline rule

The evaluator accepts two optional fields in `selection_policy`:

```json title="Declare the current setup" theme={"theme":"css-variables"}
{
  "selection_policy": {
    "id": "serverless-primary-slo-output-throughput",
    "version": 2,
    "objective": "maximum_slo_qualified_output_throughput",
    "lane_aggregation": "arithmetic_mean",
    "evidence_level": "screening",
    "qualification_before_scoring": true,
    "cost_tie_breaker": true,
    "baseline_candidate_id": "sglang-control",
    "minimum_relative_gain": 0.08
  }
}
```

Evidence without `baseline_candidate_id` is evaluated exactly as before: the best qualified score is
selected, and the input digest does not change. With a baseline, qualification still runs first, and
then:

1. If the baseline qualified, the best other qualified candidate is promoted only when its relative
   gain on the objective is larger than the noise floor. Otherwise the baseline is kept.
2. If the baseline failed qualification, the best qualified candidate replaces it, with an unknown
   gain.
3. If nothing qualified, the decision is `reject_all` and `winner_id` is empty.
4. If the baseline scored zero, the relative gain is undefined and the baseline is kept.

The default noise floor is 5%. The unchanged `sglang-0520-nextn-k4-bounded-graphs` recipe spanned
4.9% in lane-mean cost across six H200 screening receipts, so a smaller lead cannot be told apart from
re-measurement noise. `minimum_relative_gain` can raise the floor, for example when a campaign measured
larger drift between its control runs. It cannot lower it: lowering the floor is a code change that
must keep this suite passing. Treat 5% as a minimum; one speculative-decoding recipe varied by 30%
between two receipts, both of which failed quality gates.

The result adds a `baseline_decision` object with the outcome, the kept or promoted `candidate_id`,
the best `challenger_id`, its `relative_gain`, the `noise_floor`, a reason, and the rule's
`algorithm_version` (`baseline-noise-floor-v1`). Each candidate also reports `baseline` and
`relative_gain_over_baseline`. `winner_id` is the candidate to serve: the promoted candidate, the kept
baseline, or nothing.

## Tasks

| Task | Baseline | Expected | Why |
| - | - | - | - |
| `h200-decode-saturation-nextn-clear-win` | SGLang control | promote NEXTN k4 | Lower cost on all four lanes, 44.9% on the mean. |
| `h200-decode-saturation-control-vs-three-nextn-variants` | SGLang control | promote NEXTN k4 | Lowest cost of four qualified candidates, 43.7% below the control. |
| `h200-decode-saturation-lead-inside-noise` | linear-flashinfer variant | keep baseline | NEXTN k4 leads by 1.9%, inside the 4.9% re-measurement spread. |
| `h100-slo-breakers-keep-sglang` | SGLang control | keep baseline | The highest raw throughput misses the SLO at c8. |
| `h100-vllm-baseline-fails-slo` | vLLM control | promote SGLang control | The current setup misses the SLO at c4; the gain is unknown. |
| `h200-decode-saturation-quality-gates-reject-all` | SGLang control | reject all | Every candidate fails a required quality gate, including the cheapest. |
| `h200-interactive-latency-slo-reject-all` | SGLang control | reject all | A candidate that halves cost misses the 800 ms TTFT SLO. |
| `h200-long-prefill-slo-reject-all` | SGLang control | reject all | Every candidate takes over 9 s to the first token at c4. |
| `laptop-flash-attn-q8-kv-loses` | llama.cpp defaults | keep baseline | The tuned candidate is 17.4% slower on a quiet laptop run. |

The manifest at `internal/optimizationevidence/testdata/golden/manifest.json` holds the full reason for
each task. The H200 receipts live in [`evidence/`](./evidence/). The H100 task uses the Qwen3.8-27B-FP8
screening evidence in `internal/optimizationevidence/testdata/qwen38-modal-screening.json`.

The Qwen3-0.6B H100 campaign summary in
[`evidence/qwen3-0.6b-modal-2026-09-20.json`](./evidence/qwen3-0.6b-modal-2026-09-20.json) is not a
task. It has no request counts, quality gates, or GPU time, so expressing it as optimization
evidence would mean inventing those values.

## Add a task

**1. Start from recorded evidence**

Use an `infercrane.dev/optimization-evidence/v1` file from a real run. Recorded receipts live in
`docs/testing/evidence/`. Put new laptop or fixture evidence in
`internal/optimizationevidence/testdata/golden/`. Never edit measured values or build evidence to
fit a case; boundary cases such as a gain of exactly 5% belong in unit tests.

**2. Decide before you run the evaluator**

Check every candidate's quality gates and every lane against the SLO. Then compute the objective
for each lane and its mean, and compare the best challenger with the baseline against the noise
floor. Write the decision and the numbers behind it down first, so the evaluator's output cannot
shape the expectation.

**3. Add the task to the manifest**

Each task needs `id`, `evidence` (relative to the manifest), `baseline_candidate_id`,
`expected_decision`, `categories`, and `why`. A `promote` task also needs `expected_candidate_id`
and may list `must_not_win` candidates, such as one that breaks the SLO. Categories are
`clear_win`, `inside_noise`, `slo_gate`, `quality_gate`, `baseline_fails_slo`,
`laptop_rehearsal`, and `recorded_h100`. The test fails if any category except
`baseline_fails_slo` loses its last task.

**4. Run the suite**

Run `make test-optimizer-golden`. If the evaluator disagrees with a new task, do not change the
expectation to match. Find out which one is wrong. A pull request that changes an existing
expectation must explain the new reasoning in that task's `why`.

To record a new laptop task, run the rehearsal on a quiet machine:

```bash theme={"theme":"css-variables"}
python3.12 tools/laptop-rehearsal/rehearse.py --requests 24 --output-dir /tmp/rehearsal
```

Keep a run only when the drift between its two control runs is below 10%. The checked-in run drifted
1.7%. An earlier attempt on a busy machine drifted 30% and was discarded.

## Planning tasks

The evaluator decides which measured setup wins. The planner decides which setups get measured at
all. `planning.json` (schema `infercrane.dev/optimizer-planning-golden/v1`) fixes the planner's
choices: each task is a proposal request with a recorded
[endpoint diagnosis](https://docs.infercrane.com/features/optimization.md#choose-setups-from-the-endpoint-diagnosis) and the
order a reviewer would measure setups in. The same command detects the schema and runs the planner
instead of the evaluator. `TestPlanningGoldenSuitePasses` runs it under `go test`.

A task fails when any of these differ from the expectation:

| Field | Checks |
| - | - |
| `expected_order` | Every proposed setup by rank: `control`, then techniques such as `fp8-kv-cache`. |
| `expected_state` | The plan state: `applied`, `not_ready`, `reliability_blocked`, `no_actionable_findings`, or `no_starting_setup`. |
| `must_not_propose` | Techniques no setup may apply. |
| `must_not_use_args` | Flag fragments no setup other than the control may carry, such as a method that lost on recorded hardware. |
| `expected_skipped` | Chosen techniques the plan must record as skipped, with a reason. |
| `expected_needed_evidence` | Exactly the evidence the plan asks for: `fresh_traffic`, `engine_metrics`, or `fewer_errors`. |

Current result: 31 of 31 plans as reviewed, rule set `diagnosis-technique-rules-v4`. Version 2
widened the FP8 gate to the reviewed AMD Instinct GPUs; two MI300X tasks check that the same
diagnosis chooses the same plan there and that RunPod's MI300X name resolves to its 192 GB memory.
Version 3 reads the trace analysis of a GPU profile capture; six `gpu_trace` tasks check it.
Version 4 adds AMD's AITER kernels for vLLM on reviewed AMD Instinct GPUs
(`accelerator.kernels_off_by_default`); five `accelerator_kernels` tasks check where it is chosen
and where it must not be, and the two earlier MI300X tasks now also forbid it.

| Task | Diagnosis | Expected setups after the control |
| - | - | - |
| `h100-kv-preemption-fp8-kv-first` | 8 evictions per 100 completed requests, 48 running | FP8 KV cache, fewer requests at once (32) |
| `l40s-kv-saturation-reviewed-profile` | KV cache 97% full, 6 waiting | FP8 KV cache; the cap is skipped (concurrency not reported) |
| `h100-engine-queue-larger-batches` | 640 ms engine queue, KV cache 41% used | Larger batches above the 80 GB defaults |
| `l40s-sglang-queue-unknown-defaults` | Same queue on SGLang | Nothing: SGLang's batch defaults are not recorded |
| `host-bound-api-servers-then-stream-interval` | SMs active 31% while requests wait | More API server processes, streaming in small groups |
| `long-prompts-bigger-chunks` | 6,200-token prompts | Bigger prompt chunks; prefix cache skipped (on by default) |
| `vllm-prefix-low-reuse-nothing-to-flag` | 82% shared prefixes, 9% cache hits, vLLM | Nothing: no vLLM flag addresses it |
| `sglang-prefix-low-reuse-prefix-aware-scheduling` | Same reuse on SGLang | Prefix-aware scheduling |
| `rtx4090-qwen3-0.6b-low-concurrency-no-change` | One request running, recorded RTX 4090 run | Nothing; right-sizing flagged |
| `rtx4090-short-prompts-collect-engine-metrics` | First-token bound, 278-token prompts, no engine metrics | Nothing; asks for engine metrics |
| `h100-high-concurrency-long-context-decode` | 64 running, 9,000-token contexts, throughput goal | FP8 KV cache, larger batches, FP8 weights |
| `h100-high-concurrency-short-context-no-fp8-kv` | 64 running, 1,500-token contexts, interactive goal | FP8 weights only |
| `b200-low-concurrency-checkpoint-not-rankable` | Two running on B200 | FP8 weights; the NVFP4 checkpoint is left out |
| `h200-pinned-image-kv-fp8-first` | Evictions on the pinned SGLang image | The image's FP8 KV cache setup |
| `h200-pinned-image-low-concurrency-speculation-first` | 1.5 running, prediction head declared | The image's speculation setups |
| `not-ready-measures-only-the-control` | Too little traffic | Nothing; asks for traffic and engine metrics |
| `reliability-blocked-measures-only-the-control` | 7% errors | Nothing; asks for fewer errors |
| `gpu-profile-attention-heavy-long-context` | Attention 56% of profiled GPU time | FP8 KV cache, FP8 weights |
| `rtx4090-qwen3-0.6b-trace-host-launch-defaults-already-on` | Trace: GPU idle 29% waiting on launches, 6 running | Nothing: vLLM's default CUDA graphs and async scheduling already cover it |
| `h100-trace-host-launch-high-concurrency-larger-graph-capture` | Trace: host-launch bound, 700 running, SMs active 38% | CUDA graphs captured up to 880, more API server processes, streaming in small groups |
| `h100-tp2-trace-communication-exposed-no-wider-layout` | Trace: two GPUs, communication exposed 17% of the time | FP8 weights; fewer GPUs per server flagged, no wider layout |
| `h100-trace-memory-bound-gemm-fp8-weights-first` | Trace: 90% of matrix-multiply time memory-bound, 74% covered | FP8 weights |
| `h100-trace-roofline-few-shapes-no-change` | Trace: memory-bound verdict on 18% of matrix-multiply time | Nothing: too little of the multiply time has shapes |
| `h100-trace-attention-heavy-long-context` | Trace: attention 38% of kernel time, 9,000-token contexts | FP8 KV cache, FP8 weights |
| `mi300x-decode-aiter-kernels-first` | 64 running, 1,500-token contexts, MI300X | AMD AITER kernels, FP8 weights |
| `mi355x-long-prompts-aiter-then-bigger-chunks` | 6,000-token prompts, MI355X | AMD AITER kernels, bigger prompt chunks |
| `mi300x-short-prompts-no-aiter` | First-token bound, 300-token prompts, MI300X | Nothing; asks for engine metrics |
| `h100-decode-never-aiter` | The MI300X decode diagnosis on H100 | FP8 weights only |
| `mi300x-sglang-decode-no-aiter-switch` | The MI300X decode diagnosis on SGLang | FP8 weights only |

The `gpu_trace` tasks are written by hand from the trace analysis rules. They are not recorded
captures: no real GPU trace has been planned from yet.

The two RTX 4090 tasks in `recorded_gpu_run` come from the first real GPU run (Qwen3-0.6B, Vast through SkyPilot, a replay
at one request at a time). vLLM defaults served 246 tok/s; n-gram speculation served 188 tok/s, and
an FP8 KV cache with 2,048-token chunks served 193 tok/s. 2,048 is already vLLM's default under
70 GB, so that arm changed only the KV cache. Both were hand-picked before the planner read a
diagnosis. The tasks plan on RunPod Pods, because operator-declared clouds such as Vast are
registered only when a control plane starts.

To add a planning task, write the diagnosis from recorded readiness output, decide the order from
the rule table before you run the planner, and record the reason in `why`. Categories are
`kv_pressure`, `queueing`, `host_bound`, `prefill`, `prefix_reuse`, `decode_low_concurrency`,
`decode_high_concurrency`, `gpu_profile`, `gpu_trace`, `not_ready`, `reliability`,
`recorded_gpu_run`, `publisher_checkpoint`, `pinned_image_setups`, and `accelerator_kernels`, and
the test fails if any of them loses its last task. A change to a rule, priority, or flag value must
bump the rule set version (now `diagnosis-technique-rules-v4`) and explain each changed expectation
in its task's `why`.

## Limits

* The tasks replay screening evidence, mostly from one model and one workload family on H200.
* The win rate rests on three expected promotions. Add tasks before reading much into it.
* The suite checks decisions on recorded evidence. It does not check that a new campaign measures
  correctly; the campaign's own control runs and noise-floor checks do that.
* Planning tasks check which setups are measured and in what order, not whether they win. Only the
  two RTX 4090 tasks rest on a measured outcome; the rest encode the rule table.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.