> ## Documentation Index > Fetch the complete documentation index at: https://infercrane.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Architecture > InferCrane's persistent control plane, database-free inference path, and provider boundaries. # Architecture InferCrane separates durable deployment decisions from latency-sensitive inference routing. PostgreSQL is authoritative for the control plane; an atomic in-memory snapshot is authoritative for each gateway replica's request path. Animated architecture diagram separating InferCrane's durable control plane from its database-free inference data plane. ```mermaid theme={"theme":{"light":"github-light-default","dark":"vesper"}} flowchart LR subgraph clients[Clients] CLI[InferCrane CLI] DASH[Separate web console] DELIVERY[SDK · Terraform · GitHub] SDK[Inference client] end subgraph infercrane[InferCrane] API[Authenticated control API] OPS[Leased operation worker] DB[(PostgreSQL)] REC[Reconciler] SNAP[Atomic route snapshot] GW[Gateway] ROUTER[Replica router generation] COMPOSE[Composition registry] end subgraph external[External systems] ELASTIC[Elastic backend adapter] NATIVE[Serverless backend adapter] V1[Runtime replica] V2[Runtime replica] EXTGW[User-managed gateway] SANDBOX[External sandbox] TRAIN[Training pipeline / registry] end CLI --> API --> DB DASH --> API DELIVERY --> API DB --> OPS OPS --> ELASTIC OPS --> NATIVE DB --> REC REC --> V1 REC --> V2 REC --> ROUTER REC --> SNAP API --> COMPOSE COMPOSE --> EXTGW COMPOSE --> SANDBOX COMPOSE --> TRAIN SDK --> GW --> SNAP --> ROUTER SANDBOX -. endpoint-scoped inference token .-> GW TRAIN -. signed artifact handoff .-> API ROUTER --> V1 ROUTER --> V2 GW -. buffered accounting .-> DB ``` ## Control plane The CLI, separately deployed web console, generated SDKs, Terraform provider, and GitHub delivery action use the same authenticated API. None owns provider resources or reads PostgreSQL. PostgreSQL stores desired and observed deployment state, immutable revisions, replicas, operations, events, policy evaluations, and bounded measurements. A leased worker resumes pending provider work after process failure and reconciles results back into persisted state. External ownership stays explicit and replaceable behind narrow adapters: * An elastic backend provisions and deletes replica infrastructure. * A provider-native serverless backend owns worker allocation and scale-to-zero. * An artifact resolver identifies and transfers immutable model artifacts. * A benchmark runner generates load and returns reproducible evidence. * A composition registry describes replaceable gateway, sandbox, and training handoff contracts without claiming lifecycle ownership of those external systems. InferCrane does not replace any of them with a general scheduler or workflow engine. The process composes registered elastic, serverless, external-target, runtime, artifact, and benchmark adapters. RunPod, AWS EC2, and Kubernetes are current infrastructure adapters with different qualification states; vLLM, SGLang, and custom OCI are current runtime profiles. Durable algorithms do not select implementations with provider conditionals; [qualification is separate from registration](/adr/0009-qualified-support-and-backend-registration). ## A modular system, not a bundled stack Users select a serving plan; adapters translate it into infrastructure-specific work. Core lifecycle state does not depend on a RunPod, AWS, Kubernetes, vLLM, SGLang, or gateway-specific domain model. ```mermaid theme={"theme":{"light":"github-light-default","dark":"vesper"}} flowchart TB INTENT[Endpoint · environment · serving intent] CORE[InferCrane core
desired state · operations · revisions · evidence] INTENT --> CORE CORE --> INFRA{Infrastructure contract} CORE --> RUNTIME{Runtime contract} CORE --> ROUTE{Route type} CORE --> ARTIFACT{Artifact resolver} CORE --> BENCH{Benchmark runner} CORE --> COMPOSE{Composition contract} INFRA --> AWS[AWS EC2] INFRA --> RP[RunPod] INFRA --> K8S[Kubernetes] RUNTIME --> VLLM[vLLM] RUNTIME --> SGL[SGLang] RUNTIME --> OCI[Custom OCI] ROUTE --> NATIVE[InferCrane-managed replicas] ROUTE --> SL[Provider-native serverless] ROUTE --> EXT[Adopted or governed external target] ARTIFACT --> HF[Hugging Face / Xet] BENCH --> AIP[AIPerf] COMPOSE --> LLMGW[LiteLLM or another gateway] COMPOSE --> SBX[E2B · Modal · Kubernetes sandbox] COMPOSE --> TRN[MLflow · Kubeflow · external training] ``` | Layer | InferCrane owns | Integrated system owns | | -------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | Infrastructure | intent, durable identity, retries, reconciliation, cleanup evidence | compute allocation and provider API semantics | | Runtime | capability contract, immutable configuration, readiness qualification | inference engine and model execution | | Gateway or external target | stable endpoint identity, admission, policy, request evidence | optional protocol translation and its provider credentials | | Artifact | immutable model identity and cache observations | transfer and provider-native storage primitives | | Benchmark | reproduction record, comparison, policy input | load generation and raw measurement | | Sandbox composition | endpoint-scoped, expiring inference identity and content-free reference metadata | isolation, commands, files, networking, snapshots, and execution lifecycle | | Training composition | signature verification, immutable artifact lineage, and revision attachment | datasets, training jobs, registry availability, and checkpoint storage | A user-managed LiteLLM deployment, for example, can be connected as an OpenAI-compatible external gateway. InferCrane does not bundle or fork LiteLLM: it retains endpoint identity and operational evidence while LiteLLM retains provider translation and its own configuration. See the [gateway and sandbox showcase](/showcase/gateways-and-sandboxes). ## Data plane An OpenAI-compatible request resolves the logical model alias from an atomic route snapshot. For standalone replica runtimes, it then passes to an instance-owned vLLM Router generation and a healthy worker. Provider-native Serverless and governed external targets remain explicit route types. No PostgreSQL lookup occurs in that routing decision. Each horizontally scaled gateway process owns its loopback router processes and deterministic ports. Gateway instances share PostgreSQL state, never another instance's local router. ## Request lifecycle The gateway authenticates the bearer token, validates the OpenAI request, and resolves the requested deployment alias. The route directory returns a ready generation without network or database I/O. vLLM Router applies the persisted strategy to its healthy standalone vLLM endpoints. The gateway proxies the upstream response. Streaming responses are not constrained by the ordinary server write timeout. Request accounting and normalized measurements enter bounded buffers and persist outside the routing decision. ## Safe route changes The reconciler probes worker health and served-model identity, calculates membership, starts a candidate router generation, and only then publishes a new snapshot. Scale-down fences routing and drains the worker before provider termination. Failed candidates never replace the last healthy route. Read the rules every implementation change must preserve. Follow the reasoning behind persistence, tenancy, revisions, and operation execution.