Skip to main content
InferCrane exposes one stable logical endpoint while preserving the selected runtime’s native OpenAI-compatible request and response shape. It does not translate every workload into a lossy universal schema.

​
Surfaces

The current vLLM protocol documentation lists these upstream surfaces, but availability depends on both runtime version and model task. InferCrane therefore treats each capability independently. An unknown or unsupported claim returns 422 unsupported_protocol before any request reaches the workload.

​
Model identity

Send the stable InferCrane endpoint name as model. The gateway selects a pinned route generation and rewrites only that field to the binding’s upstream model identity.
All other protocol fields pass through. Upstream status, content type, response body, cancellation, and streaming semantics are preserved. InferCrane never retries a partially transmitted stream.

​
Telemetry and privacy

Every accepted surface records the protocol operation, logical endpoint, selected binding, deployment/revision where known, provider/runtime dimensions, latency, status, token counts where the upstream reports them, and streaming errors. Prompt, response, embedding vectors, and tool arguments are not persisted by default.

​
Current boundary

The default vLLM image remains pinned to 0.8.5.post1; its profile does not claim Responses or the newer online chat-batch endpoint. This is deliberate capability honesty, not automatic version guessing. A later runtime update must pass protocol conformance and real GPU qualification before those claims become default.