Skip to main content

Healthy infrastructure is not enough

A candidate may be ready, fast, and error-free while producing worse answers. InferCrane accepts aggregate evidence from a customer-owned evaluation suite, verifies its signature and revision identity, and lets Release Guard enforce explicit score thresholds. Optimization runs also produce their own evidence with a built-in answer check, so a run reaches a decision without you signing anything. InferCrane does not store prompts or generated outputs in this evidence format and does not choose an LLM judge for you.

Built-in answer check

When an optimization campaign measures a candidate and finds no evidence you attached for its exact revision, the worker checks the candidate’s answers itself. It sends the same prompts to the candidate and to a reference, the campaign control or, for an evolve-endpoint campaign, the target’s active revision, and scores how closely each candidate answer matches the reference answer. How a campaign checks answers describes the steps. Answers are compared in memory and discarded. Prompts of the built-in suite live in the product; nothing about your traffic is sent. Evidence you attach for the candidate’s exact revision always wins over the built-in check. A campaign created with --quality-check customer_evidence skips the built-in check and waits for your evidence. The run page and infercrane optimize inspect then show the exact evaluation ingest command for each waiting candidate. The answer check measures agreement with your current setup on general prompts. It is not a task accuracy benchmark. Use an eval set of your own prompts, or your own evaluator, for that.

Use your own prompts

An eval set replaces the built-in prompts. It has the same format as the built-in suite:
eval-set.json
Up to 200 prompts and 512 KiB. Prompt IDs are 1 to 64 letters, digits, dots, dashes, or underscores and must be unique. Unknown fields are rejected, so expected answers are not accepted: candidates are scored against the reference’s answers, not against stored answers.
The digest is the SHA-256 of the file’s exact bytes. The campaign stores it in its reviewed proposal, so the eval set cannot change under a running campaign. A campaign that names an eval set that is not uploaded waits for it, and the run page shows the upload command. evaluation set inspect shows an uploaded eval set without its prompts, and evaluation set delete removes it. Uploaded eval sets belong to your organization and are deleted with it.

Signing key

Built-in evidence is signed with the passport signing key from INFERCRANE_PASSPORT_SIGNING_KEY_FILE when the control plane has one. Without it, each control-plane process generates a key when it starts. Every row stays verifiable either way, because the envelope carries its public key, but the key ID then changes on restart. Configure the passport key for a stable key ID.

Create an evaluator key

Keep the private Ed25519 key outside the control plane. The operator authorized to attach evidence is the trust boundary.

Ingest a result from any evaluator

An artifact digest proves which private result bundle was evaluated, not that the evaluator was well-designed. Before generating or attaching evidence, approve the representative dataset, privacy boundary, evaluator and version, calibration procedure, sampling method, and required human review for high-risk changes. The commands below persist signed release evidence and --attach mutates the candidate record; they do not upload prompt/output bodies or authorize promotion.
Have Ragas, DeepEval, a custom CI evaluator, or another customer-controlled system emit the strict content-free evaluator result contract at schemas/evaluator-result-v2.schema.json:
evaluator-result.json
The abbreviated score array above is illustrative; in a valid file scores contains exactly sample_count finite values from 0 through 1 in the evaluator’s canonical pair order. Active and candidate files must use the same pairing digest. The digest commits the private sample identities without storing those identities, prompts, or responses in InferCrane. Bind that aggregate result to the exact immutable candidate revision, sign it, and explicitly attach it through the control-plane API:
Unknown fields are rejected, so prompt and generated-output bodies cannot accidentally enter the evidence envelope. If API attachment fails, the signed local file remains available for a safe evaluation attach retry.

Sign aggregate evidence directly

The signed payload records deployment, immutable revision, suite and evaluator versions, normalized score, pass/fail result, sample count, result-artifact digest, and evaluation time. Attaching the same evidence twice is idempotent.

Fail closed in Release Guard

Active and candidate evidence must use the same suite version and evaluator version to be comparable. Missing or incomparable required evidence leaves the decision waiting. A failed suite, score below the minimum, or regression beyond policy rejects the candidate. The evaluator cannot activate a revision; promotion remains a separate, authorized control-plane decision. Release Guard checks every eval the candidate has evidence for, not only the newest one, including the built-in answer check. For an evolve-endpoint campaign the built-in check records the active revision as its reference, so the two rows are comparable. An eval is one suite scored by one evaluator. For each eval, InferCrane takes the candidate’s newest evidence and compares it with the active revision’s newest evidence from the same suite version and evaluator version. threshold remains the default comparison mode. In bootstrap mode InferCrane performs 10,000 deterministic paired resamples. A confidence interval wholly below the allowed regression passes; one wholly beyond it rejects; an overlapping interval or too few pairs waits. The persisted decision records the seed, alpha, active/candidate counts, observed regression, and interval so a retry is reproducible. The three-score example demonstrates the contract, not a recommended production sample size. Choose and review a representative minimum for the evaluator and risk class; insufficient pairs must remain waiting.

PASS, WATCH, and BLOCK per eval

A small drop is not always a reason to stop a rollout, but it should not pass silently either. Add a watch band below the regression limit, and every eval gets one of four verdicts: The drop is relative to the active revision’s score: (active - candidate) / active × 100. In bootstrap mode, the interval decides BLOCK and UNKNOWN as described above. Once the interval clears the limit, the observed paired drop decides between WATCH and PASS. The model verdict is the most severe eval verdict, in the order BLOCK, UNKNOWN, WATCH, PASS. A candidate with no evidence at all is UNKNOWN.

Set the watch band

quality_watch_regression_percent sits beside max_quality_regression_percent in the persisted policy. It needs a regression limit and cannot exceed it. The example below keeps the 3% limit from the policy above and opens a watch band from 1% to 3%:
infercrane rollout policy set has no flag for the watch band yet. It keeps the stored value when you change other settings. Lowering --max-quality-regression below the watch threshold is rejected. Remove the band first by setting the field to null.

Read the verdicts

GET /api/v1/deployments/{name}/quality-evidence returns the evidence list and a verdicts object for the current candidate. It applies the persisted policy to the stored evidence and records nothing:
The example is abbreviated: real responses also carry data, content_recorded, evidence IDs, and a message per eval, and numbers are not rounded. delta is the candidate score minus the active score, so a drop is negative. model is the resolved immutable model identity, or the model the revision requested when no identity was resolved, and empty when neither is recorded. Without a compared state there is no verdict. The other states are no_candidate, not_gated (the policy does not gate on quality, or Release Guard is disabled), and unavailable (no running deployment or policy). The stored Release Guard evaluation stays the decision of record. Each evaluation persists the same report under metrics.quality. Its reasons name the eval (suite@suite_version) for every BLOCK or UNKNOWN eval, and for every WATCH eval when the decision is ACCEPT.

Follow up on WATCH

When the decision is ACCEPT and at least one eval is WATCH, the evaluation starts its reasons with one quality_watch reason per watched eval. infercrane rollout inspect prints each one with its follow-up. Release Guard also writes a release_guard.quality_watch audit event with outcome follow_up_required. That event lists the watched evals, their scores and drops, and the follow-up, so it stays visible after promotion. Listing audit events needs tenant-admin access:
A WATCH does not lower any other gate. If another check rejects or waits, that decision stands and no follow-up is recorded.