Skip to main content

vLLM

vLLM is InferCrane’s default qualified runtime profile. InferCrane owns deployment state and lifecycle; vLLM owns model execution. A supervised vLLM Router process distributes requests across healthy standalone replicas.

Connect existing workers

Each worker must expose the OpenAI-compatible vLLM API and be reachable from the InferCrane gateway.
The reconciler verifies health and served-model identity before publishing a route. An unhealthy or mismatched worker does not enter the request path.

Responsibility boundary

InferCrane does not implement an inference engine or distributed KV cache. SGLang and custom OCI workloads use the same versioned runtime boundary and have independent qualification states.

Gateway and routing

Follow an OpenAI request from alias resolution to a healthy vLLM replica.