Open-source inference control plane
Build or connect
Safe releases
Explainable operations
No Kubernetes required
requestOpenAI client
→
stable identityendpoint
→
policyroute generation
→
healthyruntime replicas
model=support-production] --> IC[InferCrane
endpoint · policy · evidence] IC --> LL[User-managed LiteLLM
translation · provider credentials] LL --> API1[Managed API] LL --> API2[Other model provider] IC --> SELF[Self-hosted vLLM or SGLang] ``` ## Connect in observe-only mode ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane connect https://litellm.internal.example/v1 \ --as support-production \ --type litellm \ --model company-coder infercrane observe support-production infercrane doctor support-production ``` InferCrane verifies the OpenAI-compatible discovery surface before persisting the connection. It does not install, configure, upgrade, or fork LiteLLM.
isolation · files · commands] SB -->|short-lived endpoint token| IC[InferCrane
coder-production only] IC --> MODEL[Qualified serving plan] SB -. provider lifecycle .-> OWNER[Sandbox provider] ``` ## Issue access Create the sandbox with its native API or control plane first. Then connect its external identity: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane sandbox connect \ --provider e2b \ --external-id sandbox-01JCGW \ --external-revision template-v3 \ --endpoint coder-production \ --ttl 30m ``` The command prints the credential once. Inject it using the sandbox provider's secret mechanism: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} export INFERCRANE_URL="https://inference.example.com" export INFERCRANE_API_KEY="ic_...shown-once..." ``` Inside the sandbox, use an ordinary OpenAI client: ```python theme={"theme":{"light":"github-light-default","dark":"vesper"}} import os from openai import OpenAI client = OpenAI( base_url=os.environ["INFERCRANE_URL"].rstrip("/") + "/v1", api_key=os.environ["INFERCRANE_API_KEY"], ) result = client.responses.create( model="coder-production", input="Explain the failing test without modifying files.", ) print(result.output_text) ``` That token cannot enumerate or invoke a different endpoint alias, and it cannot call the InferCrane control API. ## Inspect, rotate, and revoke ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane sandbox list infercrane sandbox rotate SANDBOX_REFERENCE_ID infercrane sandbox revoke SANDBOX_REFERENCE_ID --yes ``` Rotation invalidates the previous credential. Revocation disables InferCrane access but does not delete, pause, snapshot, or mutate the external sandbox. The issuing control-plane instance refreshes its database-independent credential snapshot before it returns. In a horizontally scaled control plane, other gateway replicas converge on the persisted change on their bounded credential refresh interval (one second by default). Keep a retry for an initial `401` from a newly issued credential. Rotation and revocation may take that same bounded interval to reach every replica.
and reservation available?} BUDGET -->|yes| EXT[Approved external target] BUDGET -->|no| DENY[Reject before sending bytes] ```
model: company-models] --> IC[InferCrane
identity · policy · evidence] IC --> LL[User-managed LiteLLM
protocol translation] LL --> OAI[Managed model API] LL --> OTHER[Other configured targets] ``` InferCrane does not bundle, fork, install, or license LiteLLM. The operator supplies and manages the gateway and its provider credentials. This keeps the core integration generic: the same connection path works with another compatible gateway when it passes discovery and health checks.
identity · lifecycle · evidence] IC --> DEP[InferCrane-managed deployment] IC --> LL[User-managed LiteLLM gateway] IC -. governed binding or overflow .-> OR[External model API] SB[User-managed sandbox
E2B · Daytona · Modal · Kubernetes] --> IC ``` ## Connect a user-managed LiteLLM gateway Keep LiteLLM's provider configuration and credentials in LiteLLM. Connect its OpenAI-compatible surface to InferCrane in observe-only mode first: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane connect https://litellm.internal.example/v1 \ --as company-models \ --type litellm \ --model coder infercrane doctor company-models ``` After qualification, transfer routing ownership explicitly: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane adopt promote company-models --ownership traffic-managed ``` InferCrane does not bundle, fork, install, or license LiteLLM. It owns the logical endpoint and operational evidence; LiteLLM remains responsible for protocol translation and its upstreams. ## Add managed APIs without changing application code An authenticated OpenRouter or OpenAI-compatible API can be a first-class endpoint binding. Stage it as a fallback while the application continues sending `model="coder-production"`: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane endpoint bind coder-production \ --name managed-fallback \ --target managed-coder \ --ownership traffic-managed \ --external-adapter openai-compatible-external \ --secret-reference SECRET_REFERENCE_ID \ --request-limit 1000 \ --cost-limit-usd 25.00 \ --max-request-cost-usd 0.10 \ --acknowledge-external-data \ --enable-external infercrane endpoint plan coder-production \ --policy primary-fallback \ --bindings self-hosted,managed-fallback ``` The plan remains a candidate until evaluated and promoted. InferCrane never forwards its public API key to the provider and never stores the provider key in binding configuration. ## Use OpenRouter as deployment overflow OpenRouter can also be optional deployment-level emergency capacity, not an implicit default. Configure it with a secret reference, explicit model mapping, privacy acknowledgement, and hard request and cost reservation ceilings. InferCrane selects it before sending bytes and never silently duplicates or shadows a request. ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} export OPENROUTER_API_KEY='...' infercrane secret create openrouter --from-env OPENROUTER_API_KEY --output json infercrane target add openrouter-qwen \ --provider openrouter \ --url https://openrouter.ai/api/v1 \ --upstream-model qwen/qwen3-8b ``` Complete the bounded policy using [Governed external capacity](/features/external-capacity). Real OpenRouter billing qualification is still deferred; InferCrane does not fabricate price evidence. ## Call InferCrane from an agent sandbox The sandbox is the application execution environment. Create it with the sandbox provider, then issue a short-lived credential restricted to exactly one InferCrane endpoint: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane sandbox connect \ --provider e2b \ --external-id sandbox-01JCGW \ --external-revision template-v3 \ --endpoint coder-production \ --ttl 30m ``` Inject the shown-once credential through the sandbox provider's secret mechanism, then use an ordinary OpenAI client inside the sandbox:
still serving] --> COMPARE[Persisted comparison] CANDIDATE[Candidate revision
not in logical route] --> COMPARE POLICY[Snapshotted policy] --> COMPARE COMPARE -->|REJECT| CLEAN[Keep active
clean candidate] COMPARE -->|ACCEPT| PROMOTE[Publish candidate
drain old generation] ``` Every evaluation records: * active and candidate revision identity; * the exact policy snapshot; * compatible benchmark IDs and workload parameters; * measurements that were available; * explicit unavailable measurements; * deterministic reason codes and timestamp. Use the persisted decision during an incident or approval review: ```bash theme={"theme":{"light":"github-light-default","dark":"vesper"}} infercrane explain rollout qwen-prod infercrane rollout inspect qwen-prod --output json ```
model or endpoint] --> IC[InferCrane
intent · lifecycle · evidence] IC --> P[Provider contract] IC --> R[Runtime contract] IC --> G[Stable OpenAI-compatible endpoint] P --> CLOUD[AWS · GCP · RunPod · Kubernetes] R --> ENGINE[vLLM · SGLang · custom OCI] ``` InferCrane replaces handwritten lifecycle glue, not the infrastructure ecosystem: * SkyPilot or provider APIs continue provisioning infrastructure. * vLLM, SGLang, or custom runtimes continue executing inference. * vLLM Router continues distributing requests among standalone replicas. * AIPerf continues generating benchmark load. * Hugging Face Hub/Xet continues resolving and transferring model artifacts. * OpenTelemetry conventions continue defining portable telemetry. InferCrane owns the stable endpoint, desired and observed state, durable operations, revisions, rollout policy, evidence, explanations, and adapter qualification. ## Choose InferCrane when * You want to operate open-weight or custom inference on infrastructure you control. * You already run vLLM or an OpenAI-compatible gateway and need safer operations without migration. * You need asynchronous deploy/update/delete operations that survive a disconnected CLI. * You need evidence before promoting a runtime, model, GPU, or provider change. * You want provider and runtime portability without adopting a universal lowest-common-denominator API. ## Do not choose InferCrane when * You want a hosted model API with no infrastructure account or control plane to operate. * You need training, fine-tuning orchestration, a generic workflow engine, or an agent framework. * You expect every model/runtime/provider combination to be supported without qualification. * You require InferCrane-managed sandbox execution; sandboxes are currently application-managed. * You need a production claim for an adapter marked experimental or locally qualified only.