Choose a serving plan with evidence
Model APIs are often the fastest way to start. Self-hosted inference can offer more control for a
stable workload. Hybrid routing can support migration or bounded overflow. InferCrane keeps the
application endpoint stable while it records the evidence needed to compare those choices.
1. Establish the current baseline
The request view can report gateway-observed TTFT, latency, queueing, errors, fallback, and token
usage when the upstream supplies it. FinOps accepts sourced cost observations only. Unknown price,
currency, or coverage stays unavailable.
What can be compared today
InferCrane can persist attributed self-hosted Kubernetes cost from OpenCost. It does not yet
import provider invoices or fetch a model API price catalog. External-binding cost limits are
worst-case authorization reservations, not observed spend. Therefore the current release cannot
produce a trustworthy API-versus-self-hosted savings number by itself.
For a defensible manual comparison, collect the same time window and currency from both sides:
Import the self-hosted side when it is Kubernetes-backed:
Keep the API invoice comparison external until InferCrane has a qualified billing importer. If any
required input, request coverage, or currency alignment is missing, the only correct answer is
unavailable—not zero cost or estimated savings.
Bound a hybrid fallback before it can receive data
Start with a self-hosted-only plan and follow the complete
privacy-preserving binding procedure.
Do not run the provider connection command unless your organization has approved the provider,
data-processing terms, residency, credential-reference policy, current pricing/currency, and a
worst-case spend envelope. Registration does not enable traffic or bill inference, but it is still
an external integration and must not be created by an unapproved operator. If approval or a settled
invoice/billing export is missing, keep the self-hosted-only plan and report cost comparison as
unavailable.
--enable-external is the
explicit traffic and data boundary:
The plan is staged as a candidate and does not move traffic until its Release Guard evidence passes
and an operator promotes it. If price, currency, request coverage, residency, or provider terms are
unknown, stop before endpoint bind and retain the self-hosted-only plan. The two USD flags reserve
a hard worst-case authorization amount; they do not prove actual spend or savings.
2. Create a comparable candidate
A benchmark is measured evidence for its exact model artifact, runtime, configuration, provider,
accelerator, and workload. It is not a universal price or performance claim.
Qualify a provider-backed scaling plan
Provider-backed deploy, apply, benchmark load, autoscaling, and async execution may create
capacity or paid inference. Use an isolated staging deployment, a reviewed min/max, and a hard
request bound. Follow:
- exact-combination compatibility;
- autoscaling
1 → N → 1, streaming, and cleanup;
- async deadlines, idempotency, encryption, and cancellation; and
- the selected provider acceptance runbook.
Approve the maximum provider cost and non-sensitive workload before the first mutation. Keep async
request count, retention, retry, and deadline bounded; async protocol support still depends on the
exact runtime/model profile. Missing provider inventory, price, quota, request accounting, stream
integrity, or zero-resource cleanup makes the qualification failed or inconclusive.
3. Protect production
Inspect the release decision
INCONCLUSIVE; a measured regression can produce REJECT. Neither recommendation
nor approval silently moves traffic.
InferCrane does not fabricate provider pricing or advertise an estimated savings percentage as a
realized result. Verify provider billing, workload comparability, and production performance before
using the result for financial planning.