Skip to main content

​
Architecture

InferCrane separates durable deployment decisions from latency-sensitive inference routing. PostgreSQL is authoritative for control-plane state. An atomic in-memory snapshot is authoritative for each gateway replica’s request path.

​
Whole-product topology

The public model ID and API endpoint stay stable. The private route may move from an upstream API to a qualified RunPod target and later to dedicated or BYOC capacity. That movement creates new immutable target bindings and route publications; it never rewrites historical evidence.
01 · applicationOpenAI client
02 · stable identitylogical endpoint
03 · memoryroute snapshot
04 · qualifiedruntime binding
The gateway authenticates the request, resolves the logical model alias from one atomic route snapshot, and streams to a ready binding. No PostgreSQL lookup occurs in this routing decision.

​
Control plane and data plane

The CLI, separately deployed web console, generated SDKs, Terraform provider, and GitHub delivery action use the same authenticated API. None reads PostgreSQL directly or owns provider resources. Each horizontally scaled gateway process owns its loopback router processes and deterministic ports. Gateway instances share PostgreSQL state, never another instance’s local router.

​
Replaceable contracts

Core lifecycle state does not depend on RunPod, AWS, Kubernetes, vLLM, SGLang, or a specific gateway. Users select a serving plan; adapters translate it into infrastructure-specific work and report capabilities and qualification evidence separately. A user-managed LiteLLM deployment, for example, is connected as an OpenAI-compatible external gateway. InferCrane does not bundle or fork LiteLLM: InferCrane retains endpoint identity and operational evidence while LiteLLM retains provider translation and configuration. See the gateway and sandbox showcase.

​
Request lifecycle

1

Authenticate and admit

The gateway authenticates the bearer token, validates the protocol request, applies budgets and quotas, and resolves the endpoint alias.
2

Read one route snapshot

The route directory returns a ready generation without network or database I/O.
3

Route to a binding

A standalone runtime, provider-native serverless endpoint, or governed external target remains an explicit route type.
4

Stream safely

The gateway propagates cancellation and does not constrain streaming responses with the ordinary server write timeout.
5

Record bounded telemetry

Request accounting and normalized measurements enter bounded buffers and persist outside the routing decision.

​
Safe route changes

The reconciler probes worker health and served-model identity, calculates membership, starts a candidate router generation, and only then publishes a new snapshot. Scale-down fences routing and drains the worker before provider termination. Failed candidates never replace the last healthy route.

System invariants

Read the rules every implementation change must preserve.

Provider contract

See how idempotency, adoption, inventory, and qualification keep providers replaceable.