Skip to main content

Inference telemetry

InferCrane records operational measurements for each request without storing prompts or generated content. Request bodies are used only to forward the OpenAI-compatible request, and response chunks are observed transiently to extract timing and usage metadata when the runtime supplies it. Persisted request dimensions are:
  • deployment and active revision
  • provider, runtime, and compute mode
  • GenAI operation name (chat)
  • requested logical model and runtime-reported response model
  • OpenTelemetry GenAI schema identity (https://opentelemetry.io/schemas/gen-ai/1.42.0)
  • HTTP status, error type, and whether the response streamed
  • response model, when returned by the runtime
Persisted measurements are request latency, time to first response byte/chunk at the InferCrane gateway boundary, and input/output token counts when vLLM returns OpenAI usage fields. For streaming requests this timing maps to gen_ai.response.time_to_first_chunk; InferCrane does not claim it is the model server’s internal gen_ai.server.time_to_first_token. Missing token usage remains unknown; InferCrane does not estimate it. Replica identity remains unset when the standalone router does not provide a trustworthy selected-worker identity. The corresponding OpenTelemetry GenAI concepts are gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.request.stream, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.server.request.duration, and gen_ai.response.time_to_first_chunk. InferCrane-specific deployment, revision, provider/runtime, compute-mode, and operation dimensions are retained with the durable request record so decisions can be reproduced later. runpod is a grounded custom provider value because the convention permits provider-specific names outside its well-known list. Aggregated deployment statistics expose request rate, error rate, latency p50/p95, TTFT p50/p95, and observed input/output tokens per second over the selected window. Token throughput is emitted only from runtime-reported usage. The endpoint monitoring API returns the same content-free evidence as a bounded time series, plus binding/deployment/revision attribution and persisted lifecycle overlays. The console uses this API; it never queries PostgreSQL or an OpenTelemetry collector from the browser. Missing buckets and unsupported runtime metrics stay null or explicitly unavailable rather than becoming zero. Each operator-visible metric also carries a normalized evidence envelope: value, unit, availability, evidence class, source, observation time, freshness boundary, sample count, and a machine-readable absence reason. Availability distinguishes available, stale, not_observed, and unsupported; evidence class distinguishes measured, provider-reported, modeled, and estimated values. This lets CLIs, SDKs, and consoles preserve the same truth boundary instead of inferring one from a nullable number.
The human view includes the latest traffic summary. Use the authenticated console for 1-hour to 30-day charts and correlated scaling or release events.

Import existing production telemetry

Customers can keep Baseten, Fireworks, Datadog, or an OpenTelemetry pipeline as their monitoring source and post a strict content-free observation batch:
The endpoint accepts at most 1,000 observations per batch and rejects unknown fields, including prompt or response content. source_request_id is deterministically scoped to the tenant, deployment, and source, so retrying an import updates the same request record. After import, POST /api/v1/deployments/{name}/replays captures an immutable workload-shape trace; GET /api/v1/deployments/{name}/replays lists its history. Replay evidence contains arrival, duration, token-count, session-hash, streaming, and concurrency shapes only. It does not duplicate live traffic. Real prompt shadowing remains a separate explicit data-policy and approval boundary. The Prometheus endpoint also exposes accounting queue depth and capacity, persisted and dropped request-record counters, and persistence failures. These distinguish inference-path health from telemetry backpressure without putting PostgreSQL on the request path. Serving-engine running/waiting gauges are persisted with autoscaling decisions, and the wider engine summary is persisted as operational measurements (see Serving-engine metrics); streaming cancellations and upstream disconnects are persisted as client_cancelled and upstream_disconnect error types. GPU metrics remain unavailable unless a qualified provider/runtime collector supplies a trustworthy fresh measurement.

Serving-engine metrics

Every 30 seconds the control plane reads the Prometheus endpoint ({target}/metrics) of each healthy, self-hosted replica of a running deployment and persists one bounded, content-free summary per scrape. The engine is recognized from its metric namespace (vllm:, sglang:, or llamacpp:), not from the deployment’s runtime name. Enable the endpoint where it is off by default: SGLang needs --enable-metrics and llama.cpp server needs --metrics. The scrape presents the same credential the gateway route already presents to that replica. External provider targets and direct serverless provider targets are never scraped, because their URLs are provider APIs. A tenant compute replica without a derivable worker credential is skipped rather than scraped unauthenticated. A payload from an unsupported runtime is ignored and produces no evidence. A server observed through the InferCrane agent is scraped by the agent on its own machine every 30 seconds. The agent keeps only these families, then relays the current scrape with the oldest scrape of the same engine process from the last five minutes. The control plane derives the same measurements from that pair as from its own scrapes and keys them by the server’s target, so the endpoint’s monitoring and readiness read them the same way. A server connected directly with infercrane connect is not scraped in this release. Engine metric names omit their namespace prefix in the table. The three names that existed before engine collection (runtime_internal_itl, runtime_internal_tpot, and runtime_kv_cache_timeseries) keep their API spelling; they now carry a p95 latency or the latest KV-cache occupancy. The running, waiting, and KV-cache values come from the current scrape. Every other value is a window: the difference between the current scrape and the oldest retained scrape of the same engine process, up to five minutes back. The first scrape after the control plane starts, after a rollout changes the active revision, after the replica URL changes, or after the engine restarts (any cumulative counter moving backwards) therefore records gauges only. A window with no completed requests records latency as unavailable, while preemption and completion counts are a measured zero. Speculative acceptance needs at least one draft token in the window, so an engine with speculation off reports it as unavailable rather than 0%. SGLang reports its acceptance as an engine-computed gauge, used only while spec_num_draft_tokens is positive. Latency p95 values are estimated from the engine histogram’s bucket deltas with the same linear interpolation Prometheus histogram_quantile uses. A p95 that falls in the +Inf bucket has no finite bound and is reported as unavailable. Series that differ only by tensor-, pipeline-, or expert-parallel rank labels are counted once. Data-parallel series are added. SGLang per-priority queue breakdowns are ignored in favor of its priority="" total. runtime_tpot_mean is request-weighted for vLLM and SGLang and token-weighted for llama.cpp server. Each summary is valid for 90 seconds (three scrapes) and is attached to the deployment’s active revision at ingestion, like other collector snapshots. Monitoring shows the latest summary per replica. Without one, every engine measurement is not_observed with the reason no serving-engine metrics scrape has reported for the active revision. When the engine is reachable but does not export a signal, the reason says so instead. After the validity window the envelope becomes stale with no value. Engine values are never shown as zero because they are missing. The monitoring API also reports shared_prefix_request_ratio. It is the share of chat requests whose leading system messages repeat those of another request in the window, computed from the gateway’s keyed digest of those messages. Non-chat operations and imported observations without a digest are left out of the denominator, because their prefix shape is unknown. Metric names were checked against the vLLM, SGLang, and llama.cpp server sources in November 2026. The tests use synthetic expositions in internal/metrics/testdata. Collection against a live engine of each kind has not yet been qualified.

Diagnose a bottleneck from engine signals

GET /api/v1/endpoints/{name}/optimization-readiness (algorithm monitoring-stage-triage-v2) adds engine signals to the gateway’s stage triage. Engine rules run only after the existing gates: fresh request evidence, at least 20 requests, and an error rate at or below 2%. Each rule that fires is returned in findings with the observed value, the threshold, and the technique families worth measuring. Families are candidates for a measured comparison, never changes InferCrane applies by itself, and the custom-kernel gate stays closed. A proposal can plan from this diagnosis: it turns the findings into setups to measure, ordered by expected benefit, with the reason recorded on each.
Read engine findings
KV-cache pressure takes precedence over the other stages, because a full cache causes queueing and preemption rather than following from them. A host-bound GPU comes next and is reported as host-and-scheduler: requests wait while the GPU has idle SMs, so adding GPUs would not help. Below it, the stage order is unchanged: queueing, then first-token, then generation. A memory-bandwidth finding leads a generation bottleneck, a compute-bound finding leads a first-token bottleneck, and GPU findings that do not explain the diagnosed stage are listed after its own findings. When the gateway has no first-token time (for example non-streaming traffic), the engine’s TTFT p95 is used. Prefix-reuse and speculative findings apply to any stage. Candidate families are ordered with families from findings about the diagnosed bottleneck first, then families from other findings, then the stage’s general families. A missing engine signal never satisfies a rule. When shared prefixes are common but the hit rate is unknown, the engine prefix-cache hit rate is listed in required_evidence, as is the running-requests gauge for a generation bottleneck without it. These defaults are policy choices, not measured optimums:
  • 90% KV occupancy leaves little headroom for growing sequences. High occupancy without waiting requests is a well-packed batch and does not fire.
  • One preemption per hundred completed requests means the engine regularly recomputes evicted work.
  • 2,048 mean input tokens is where the per-step batched-token budget starts to decide how prefill interleaves with decode.
  • Four or fewer running sequences per replica leaves decode memory-bandwidth bound, which is where fewer target-model steps or smaller weights help most.
  • A 25% token hit rate with half the requests sharing a system prefix suggests the cache is off, too small, or split across replicas. The rate is token-weighted, so long unique suffixes lower it.
  • Below 50% acceptance, speculative decoding commonly spends more on drafts than it saves.
  • NVIDIA documents about 80% DRAM active as the practical peak, so 60% (three quarters of it) with tensor cores mostly idle means bytes, not math, limit the work.
  • 50% tensor active means the tensor pipe issues work in most cycles, where faster math or skipped recomputation is the remaining lever.
  • NVIDIA documents SM active below 50% as likely ineffective GPU use. With requests waiting, the serving process rather than the GPU is the limit.
  • 70% of a declared link peak leaves little headroom for collective bursts on a multi-GPU replica.
The running-requests value is the latest scrape, not a window average, so treat the low-concurrency finding as a reason to measure rather than proof. GPU rules read the imported DCGM snapshot described below. InferCrane does not scrape DCGM by itself, and the GPU rules never fire without a fresh import. GPU utilization (gpu_utilization) is reported as a signal but never used by a rule.

NVIDIA GPU evidence

When an environment already runs NVIDIA DCGM Exporter, an operator can import a bounded snapshot without exposing the exporter to the browser:
Use --file dcgm.prom instead of --url for an offline snapshot. --selector accepts exact, comma-separated Prometheus label matches. Shared exporters should always be scoped to the intended workload. The legacy DCGM_FI_DEV_GPU_UTIL is percent by default; pass --utilization-unit ratio only when an upstream compatibility layer deliberately rewrites it to 0..1. dcgm-exporter 4.9 renamed most fields, for example DCGM_FI_DEV_GPU_UTIL to DCGM_FI_DEV_GPU_UTIL_RATIO and DCGM_FI_PROF_DRAM_ACTIVE to DCGM_FI_PROF_DRAM_UTIL_RATIO. InferCrane accepts both spellings and converts each to one unit. If an exporter emits both spellings of one field, only the first one seen is counted. Across replicas, monitoring sums memory, power, and XID counts, keeps the maximum of utilization, temperature, device count, throughput, and saturation, and averages the activity percentages weighted by GPU count. The profiling (DCP) fields are averages over DCGM’s sampling interval, and many exporter configurations leave them off. In dcgm-exporter 4.9 the default counters file enables tensor, DRAM, and PCIe but not SM active, SM occupancy, or NVLink. A field the exporter does not send stays not_observed, never zero. The 4.9 DCGM_FI_DEV_XID_ERROR field reports the last XID code rather than a count, so it is not read. DCGM reports link throughput but not link capacity. To get gpu_nvlink_saturation and gpu_pcie_saturation (percent of peak, busiest GPU and direction), declare the per-GPU, per-direction peak in GB/s that applies to your hardware:
Declare link peaks
Without a declared peak, saturation stays unknown. PCIe throughput includes protocol headers, so saturation is capped at 100%. To keep the counters fresh, add --interval 15s (5 seconds to 10 minutes): the command then reads the exporter and records a snapshot every interval until stopped, each fresh for three intervals unless --ttl says otherwise. A failed read is reported and retried at the next interval; the last snapshot goes stale on its own. GPU utilization is not a bottleneck signal. NVIDIA defines it as the share of the sample period in which at least one kernel was running. A GPU that runs one small kernel at a time reads 100%, so InferCrane shows it but never diagnoses from it. SM active, tensor active, and DRAM active are the fields that show where the work is limited.

AMD Instinct GPUs

On AMD Instinct GPUs, import a snapshot from the AMD Device Metrics Exporter instead:
InferCrane reads only the fields whose name, unit, and meaning were confirmed in the exporter’s v1.5.3 source: Pass --metric-prefix when the exporter’s MetricsFieldPrefix is set. Other fields stay unknown: gpu_used_vram is documented in MB without saying whether that is MiB, gpu_umc_activity (memory controller) is not shown to equal DCGM’s DRAM active, and the GPU_PROF_* counters need profiler mode and are not shown to equal SM, tensor, or DRAM active. So the GPU rules above do not apply to AMD hardware, and the exporter’s values have not been checked against a live AMD GPU. See AMD Instinct GPUs. The snapshot is attached to the active revision at ingestion time. It cannot silently move to a later rollout. Once its TTL expires, the monitoring API returns a stale envelope with no value. This integration proves the control-plane semantics locally; actual DCGM field availability varies by GPU and DCGM version and still requires real-hardware qualification. Collector snapshots follow the configured high-volume request-evidence retention period (INFERCRANE_REQUEST_RETENTION_HOURS).

GPU profiles

A GPU profile shows where GPU time goes, from Python and PyTorch down through the CUDA launch to each kernel, its SASS instructions, and the reason its warps were waiting. infercrane profile captures one through the serving engine’s own profiler, analyzes it, and records it: see GPU profiling. This section describes the call-stack recording format, which any collector can also post itself:
capture.json follows infercrane.gpu-profile/v1. Frames are listed once and each stack refers to them by index, root first:
  • kind is python, torch, native, cuda_api, kernel, sass, or stall. Key Python frames by the function’s definition line, not the call site, so one function stays one frame.
  • gpu_ns is GPU time attributed to exactly that stack. All stacks together cannot exceed the capture length times its GPU count.
  • Stall frames use the CUPTI PC sampling reason without its smsp__pcsamp_warps_issue_stalled_ prefix, for example long_scoreboard or math_pipe_throttle.
  • A bucket with no samples has gpu_seconds: null, never 0.
  • List the layers the tool could not record in unavailable_layers. A capture without PC sampling, for example, names sass and stall, and the dashboard then says stall reasons were not recorded instead of drawing an empty layer.
  • Only evidence_class: "measured" is accepted, and content_recorded must be false. Frames carry code identifiers only, never prompts or outputs.
A capture is limited to 8 MiB, 20,000 frames, 10,000 stacks, 128 frames per stack, 500 timeline buckets, and 24 hours, and must have ended within the last 30 days. Unknown fields are rejected. Recording requires the deploy permission. The capture is attached to the revision that is active when it arrives. A retry of the same capture (same tool, start, and end) returns the original receipt, and a retry that changes the samples is a conflict. The newest eight captures per deployment are kept. GET /api/v1/endpoints/{name}/gpu-profile returns the newest capture of the active revision of any deployment the endpoint routes to, with the endpoint, deployment, and revision number it belongs to. It needs only read permission. After a rollout, captures of the previous revision are no longer returned: they describe code that no longer serves. When nothing is recorded the API answers 404 gpu_profile_not_found, and the dashboard shows Not recorded under Observe › Profiling with Advanced on, rather than an empty graph. Recording and reading a profile never touch the inference path. A infercrane.gpu-profile/v2 recording adds the trace analysis and a timeline excerpt; its fields, the capture history, and before-and-after comparison are on GPU profiling. A capture that ended more than 30 days ago is no longer returned.