Skip to main content

​
Build a production service safely

This recipe defines one logical model, separate staging and production endpoints, one version-controlled deployment, endpoint admission, a distributed tenant request ceiling, bounded autoscaling, and a fail-closed candidate test. It does not make an experimental provider/runtime combination production-qualified.
deploy, apply, candidate provisioning, validation, and autoscaling can create billable provider capacity; validation sends the approved fixture to runtimes; promotion changes traffic; rejection and deletion can remove capacity. Complete the read-only plan, provider identity/inventory review, data approval, maximum-replica exposure, and exact real-environment qualification before mutation. Use a dedicated staging deployment for failure tests. Never send customer traffic to an unqualified candidate.

​
1. Keep the serving intent in version control

Create coder-runtime.yaml with the exact reviewed model, runtime, provider, region, GPU, and min=1,max=2. Do not depend on implicit CLI defaults:
The schema does not accept placeholders at apply time; replace every uppercase value. First run only read-only checks:
The outputs must agree on the exact immutable combination. max_replicas: 2 is a capacity ceiling, not a monetary budget. Stop if provider price, stock, quota, ownership, or required qualification is unknown.

​
2. Create stable environment identities

Create the domain objects before provider mutation:
Applications will call model="coder-production"; they never need a provider address or revision ID.

​
3. Deploy and bind one reviewed runtime

The first serving plan becomes active because the endpoint has no prior plan. A later plan is a candidate and requires endpoint Release Guard. Reattach to the returned durable operation after a CLI disconnect; never submit a second deployment to recover an uncertain provider response.

​
4. Bound new requests and tenant volume

Persist endpoint-wide admission first:
Tenant quota updates require an admin credential, affect the whole active tenant, and have no read endpoint. Record the current reviewed policy externally, set every field, and use a dedicated tenant while qualifying:
Omitted fields decode as zero. Endpoint admission is shared by its authorized tenants and does not provide per-tenant queue/concurrency isolation; use separate backend pools when strict noisy-neighbor isolation is required. Save the reviewed previous three-field quota document before mutation and restore it with the same PUT --data-binary @previous-quota.json; never reconstruct it from memory. Qualify in a dedicated tenant just after a UTC-minute boundary: send exactly the approved nonzero number of requests, require the next request to return 429 before upstream transmission, and confirm the audited quota.update. Missing 429 or audit evidence blocks opening the endpoint.

​
5. Prove failure handling away from production

The GPU-free local proof first verifies control flow without cost:
It stages an intentionally unready candidate, records a deterministic Guard rejection, verifies the active revision remains unchanged, and cleans local fixture state. It does not prove GPU/runtime or provider behavior. For real-environment evidence, create a separate coder-staging-runtime deployment and bind it only to coder-staging. Use a deliberately invalid candidate revision there, never invalid production configuration and never a second deployment after an uncertain response:
The next apply may provision billable staging capacity before the candidate fails. Before running it, approve the selected provider’s maximum cost and timeout, capture direct inventory, verify the candidate-cleanup procedure and ownership identities, and reserve enough time to watch deletion back to the recorded baseline. If any of those conditions is missing, stop after plan.
The expected outcome is candidate readiness failure or Guard REJECT; the staging active revision must remain routed. Preserve evidence, then remove only the exact candidate:
The same paid-resource and cleanup boundary applies to every retry. Reattach to the original durable operation and idempotency key; do not submit another candidate because a terminal or client wait ended.

​
6. Qualify autoscaling and active-stream drain

Against the exact valid staging combination, start one long SSE request and a bounded AIPerf load in separate terminals:
While load is active and after it stops, capture:
Acceptance requires observed 1 → 2 → 1, fresh vLLM signals, no capacity above max=2, one persisted provider identity per replica intent, an intact stream with terminal data: [DONE], one request ID with no replay, generation-safe drain, full cooldown/recovery, and canonical direct provider inventory returning to baseline after staging cleanup. Missing metrics or real capacity is INCONCLUSIVE, not permission to promote. The stream/list behavior and provider timing must be proven in the exact environment; local fake workers do not qualify it.

​
7. Open production only after real qualification

Run the selected provider’s credentialed product gate with an isolated account/project/namespace, least-privilege credentials, a fixed run ID, explicit paid-resource approval, and a recorded canonical inventory baseline:
Replace PROVIDER only with the selected maintained gate (runpod, aws, gcp, or kubernetes) and satisfy that gate’s documented credential/network prerequisites first. The exact model commit, runtime/container, provider adapter, region, accelerator, network, replica bounds, and InferCrane release must match this service. Rerun a disconnected gate with the same run ID; a new ID risks duplicate resources. The report must prove readiness, buffered/streaming protocols, cancellation, benchmarking, durable recovery, deletion, empty InferCrane orphan state, and provider inventory equal to baseline. Local fixtures, Kind, and provider HTTP simulations prove control logic but cannot prove GPU readiness, IAM/networking, capacity, billing, or deletion semantics. Before application traffic:
Confirm direct provider inventory and retain the qualification report. If real qualification is unavailable, keep the endpoint private and describe the combination as experimental; do not market it as production-qualified.