Choose a serving plan with evidence

Model APIs are often the fastest way to start. Self-hosted inference can offer more control for a stable workload. Hybrid routing can support migration or bounded overflow. InferCrane keeps the application endpoint stable while it records the evidence needed to compare those choices.

1. Establish the current baseline

Baseline the endpoint

infercrane observe coder-production
infercrane finops coder-runtime --window 168h

The request view can report gateway-observed TTFT, latency, queueing, errors, fallback, and token usage when the upstream supplies it. FinOps accepts sourced cost observations only. Unknown price, currency, or coverage stays unavailable.

What can be compared today

InferCrane can persist attributed self-hosted Kubernetes cost from OpenCost. It does not yet import provider invoices or fetch a model API price catalog. External-binding cost limits are worst-case authorization reservations, not observed spend. Therefore the current release cannot produce a trustworthy API-versus-self-hosted savings number by itself. For a defensible manual comparison, collect the same time window and currency from both sides:

Required input	Model API	Self-hosted
Cost source	Settled provider invoice or billing export	Exact OpenCost allocation keys
Coverage	Requests and tokens covered by that invoice window	Requests served by the selected revision in the same window
Identity	Provider connection, model, and endpoint	Deployment, immutable revision, GPU/runtime plan
Currency	Explicit ISO currency; no silent conversion	The same explicit ISO currency
Workload	Request count, input/output tokens, latency/SLO window	The same request shape and SLO window

Import the self-hosted side when it is Kubernetes-backed:

infercrane finops collect opencost coder-production \
  --url http://127.0.0.1:9003/allocation \
  --window 168h \
  --aggregate controller \
  --allocation coder-production \
  --currency USD

infercrane finops coder-production --window 168h

Keep the API invoice comparison external until InferCrane has a qualified billing importer. If any required input, request coverage, or currency alignment is missing, the only correct answer is unavailable, not zero cost or estimated savings.

Bound a hybrid fallback before it can receive data

Start with a self-hosted-only plan and follow the complete privacy-preserving binding procedure.

Do not run the provider connection command unless your organization has approved the provider, data-processing terms, residency, credential-reference policy, current pricing/currency, and a worst-case spend envelope. Registration does not enable traffic or bill inference, but it is still an external integration and must not be created by an unapproved operator. If approval or a settled invoice/billing export is missing, keep the self-hosted-only plan and report cost comparison as unavailable.

After approval, register the provider without enabling traffic:

export MANAGED_MODEL_API_KEY='injected-by-your-secret-manager'
infercrane provider connect emergency-model-api \
  --adapter openai-compatible-external \
  --url https://models.example.com/v1 \
  --model provider/emergency-model \
  --from-env MANAGED_MODEL_API_KEY

infercrane provider list --output json

After the approved provider connection exists and a worst-case request reservation is grounded in its current contract, create a disabled-by-default bounded binding. --enable-external is the explicit traffic and data boundary:

infercrane endpoint bind coder-production \
  --name emergency-fallback \
  --connection emergency-model-api \
  --request-limit 1000 \
  --cost-limit-usd 25.00 \
  --max-request-cost-usd 0.10 \
  --acknowledge-external-data \
  --enable-external

infercrane endpoint plan coder-production \
  --policy primary-fallback \
  --bindings self-hosted,emergency-fallback
infercrane endpoint inspect coder-production

The plan is staged as a candidate and does not move traffic until its Release Guard evidence passes and an operator promotes it. If price, currency, request coverage, residency, or provider terms are unknown, stop before endpoint bind and retain the self-hosted-only plan. The two USD flags reserve a hard worst-case authorization amount; they do not prove actual spend or savings.

2. Create a comparable candidate

Evaluate a candidate

infercrane benchmark coder-runtime --revision REVISION_ID \
  --requests 100 --concurrency 4

infercrane recommend coder-runtime

A benchmark is measured evidence for its exact model artifact, runtime, configuration, provider, accelerator, and workload. It is not a universal price or performance claim.

Qualify a provider-backed scaling plan

Provider-backed deploy, apply, benchmark load, autoscaling, and async execution may create capacity or paid inference. Use an isolated staging deployment, a reviewed min/max, and a hard request bound. Follow:

exact-combination compatibility;
autoscaling 1 → N → 1, streaming, and cleanup;
async deadlines, idempotency, encryption, and cancellation; and
the exact provider, runtime, model, and accelerator boundary in compatibility and qualification.

Approve the maximum provider cost and non-sensitive workload before the first mutation. Keep async request count, retention, retry, and deadline bounded; async protocol support still depends on the exact runtime/model profile. Missing provider inventory, price, quota, request accounting, stream integrity, or zero-resource cleanup makes the qualification failed or inconclusive.

3. Protect production

Inspect the release decision

infercrane rollout inspect coder-runtime
infercrane explain rollout coder-runtime

Release Guard compares only compatible active and candidate evidence. A missing cost or task-quality signal can produce INCONCLUSIVE; a measured regression can produce REJECT. Neither recommendation nor approval silently moves traffic.

Question	Evidence InferCrane uses
Is the API path meeting latency expectations?	Request telemetry attributed to the external binding
Can self-hosted capacity meet the workload?	Immutable benchmark or replay evidence
Is hybrid overflow bounded?	Privacy acknowledgement, request reservation, and hard cost reservation
Did the change actually help?	Comparable post-promotion monitoring and sourced cost windows

InferCrane does not fabricate provider pricing or advertise an estimated savings percentage as a realized result. Verify provider billing, workload comparability, and production performance before using the result for financial planning.

Start here

Deploy and connect

Operate

Evidence and decisions

Compare cost and performance

Choose a serving plan with evidence

1. Establish the current baseline

What can be compared today

Bound a hybrid fallback before it can receive data

2. Create a comparable candidate

Qualify a provider-backed scaling plan

3. Protect production

​Choose a serving plan with evidence

​1. Establish the current baseline

​What can be compared today

​Bound a hybrid fallback before it can receive data

​2. Create a comparable candidate

​Qualify a provider-backed scaling plan

​3. Protect production

Choose a serving plan with evidence

1. Establish the current baseline

What can be compared today

Bound a hybrid fallback before it can receive data

2. Create a comparable candidate

Qualify a provider-backed scaling plan

3. Protect production