Healthy infrastructure is not enough
A candidate may be ready, fast, and error-free while producing worse answers. InferCrane accepts aggregate evidence from a customer-owned evaluation suite, verifies its signature and revision identity, and lets Release Guard enforce explicit score thresholds. Optimization runs also produce their own evidence with a built-in answer check, so a run reaches a decision without you signing anything. InferCrane does not store prompts or generated outputs in this evidence format and does not choose an LLM judge for you.Built-in answer check
When an optimization campaign measures a candidate and finds no evidence you attached for its exact revision, the worker checks the candidate’s answers itself. It sends the same prompts to the candidate and to a reference, the campaign control or, for an evolve-endpoint campaign, the target’s active revision, and scores how closely each candidate answer matches the reference answer. How a campaign checks answers describes the steps.
Answers are compared in memory and discarded. Prompts of the built-in suite live in the product;
nothing about your traffic is sent.
Evidence you attach for the candidate’s exact revision always wins over the built-in check. A
campaign created with
--quality-check customer_evidence skips the built-in check and waits for your
evidence. The run page and infercrane optimize inspect then show the exact evaluation ingest
command for each waiting candidate.
The answer check measures agreement with your current setup on general prompts. It is not a task
accuracy benchmark. Use an eval set of your own prompts, or your own evaluator, for that.
Use your own prompts
An eval set replaces the built-in prompts. It has the same format as the built-in suite:eval-set.json
evaluation set inspect shows
an uploaded eval set without its prompts, and evaluation set delete removes it. Uploaded eval sets
belong to your organization and are deleted with it.
Signing key
Built-in evidence is signed with the passport signing key fromINFERCRANE_PASSPORT_SIGNING_KEY_FILE when the control plane has one. Without it, each control-plane
process generates a key when it starts. Every row stays verifiable either way, because the envelope
carries its public key, but the key ID then changes on restart. Configure the passport key for a
stable key ID.
Create an evaluator key
Ingest a result from any evaluator
Have Ragas, DeepEval, a custom CI evaluator, or another customer-controlled system emit the strict content-free evaluator result contract atschemas/evaluator-result-v2.schema.json:
evaluator-result.json
scores contains exactly
sample_count finite values from 0 through 1 in the evaluator’s canonical pair order. Active and
candidate files must use the same pairing digest. The digest commits the private sample identities
without storing those identities, prompts, or responses in InferCrane.
Bind that aggregate result to the exact immutable candidate revision, sign it, and explicitly attach
it through the control-plane API:
evaluation attach retry.
Sign aggregate evidence directly
Fail closed in Release Guard
threshold remains the default comparison mode. In bootstrap mode InferCrane performs 10,000
deterministic paired resamples. A confidence interval wholly below the allowed regression passes;
one wholly beyond it rejects; an overlapping interval or too few pairs waits. The persisted decision
records the seed, alpha, active/candidate counts, observed regression, and interval so a retry is
reproducible.
The three-score example demonstrates the contract, not a recommended production sample size.
Choose and review a representative minimum for the evaluator and risk class; insufficient pairs
must remain waiting.
PASS, WATCH, and BLOCK per eval
A small drop is not always a reason to stop a rollout, but it should not pass silently either. Add a watch band below the regression limit, and every eval gets one of four verdicts:
The drop is relative to the active revision’s score:
(active - candidate) / active × 100. In
bootstrap mode, the interval decides BLOCK and UNKNOWN as described above. Once the interval
clears the limit, the observed paired drop decides between WATCH and PASS.
The model verdict is the most severe eval verdict, in the order BLOCK, UNKNOWN, WATCH,
PASS. A candidate with no evidence at all is UNKNOWN.
Set the watch band
quality_watch_regression_percent sits beside max_quality_regression_percent in the persisted
policy. It needs a regression limit and cannot exceed it. The example below keeps the 3% limit from
the policy above and opens a watch band from 1% to 3%:
infercrane rollout policy set has no flag for the watch band yet. It keeps the stored value when
you change other settings. Lowering --max-quality-regression below the watch threshold is
rejected. Remove the band first by setting the field to null.
Read the verdicts
GET /api/v1/deployments/{name}/quality-evidence returns the evidence list and a verdicts
object for the current candidate. It applies the persisted policy to the stored evidence and records
nothing:
data, content_recorded, evidence IDs, and
a message per eval, and numbers are not rounded. delta is the candidate score minus the active
score, so a drop is negative. model is the resolved immutable model identity, or the model the
revision requested when no identity was resolved, and empty when neither is recorded.
Without a compared state there is no verdict. The other states are no_candidate, not_gated
(the policy does not gate on quality, or Release Guard is disabled), and unavailable (no running
deployment or policy).
The stored Release Guard evaluation stays the decision of record. Each evaluation persists the same
report under metrics.quality. Its reasons name the eval (suite@suite_version) for every BLOCK
or UNKNOWN eval, and for every WATCH eval when the decision is ACCEPT.
Follow up on WATCH
When the decision isACCEPT and at least one eval is WATCH, the evaluation starts its reasons
with one quality_watch reason per watched eval. infercrane rollout inspect prints each one with
its follow-up. Release Guard also writes a release_guard.quality_watch audit event with outcome
follow_up_required. That event lists the watched evals, their scores and drops, and the
follow-up, so it stays visible after promotion. Listing audit events needs tenant-admin access:
WATCH does not lower any other gate. If another check rejects or waits, that decision stands and
no follow-up is recorded.