Inference telemetry
InferCrane records operational measurements for each request without storing prompts or generated content. Request bodies are used only to forward the OpenAI-compatible request, and response chunks are observed transiently to extract timing and usage metadata when the runtime supplies it. Persisted request dimensions are:- deployment and active revision
- provider, runtime, and compute mode
- GenAI operation name (
chat) - requested logical model and runtime-reported response model
- OpenTelemetry GenAI schema identity (
https://opentelemetry.io/schemas/gen-ai/1.42.0) - HTTP status, error type, and whether the response streamed
- response model, when returned by the runtime
gen_ai.response.time_to_first_chunk; InferCrane does not claim it is
the model server’s internal gen_ai.server.time_to_first_token. Missing token usage remains unknown;
InferCrane does not estimate it. Replica identity remains unset when the standalone router does not
provide a trustworthy selected-worker identity.
The corresponding OpenTelemetry GenAI concepts are gen_ai.operation.name,
gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model,
gen_ai.request.stream, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens,
gen_ai.server.request.duration, and gen_ai.response.time_to_first_chunk. InferCrane-specific
deployment, revision, provider/runtime, compute-mode, and operation dimensions are retained with
the durable request record so decisions can be reproduced later. runpod is a grounded custom
provider value because the convention permits provider-specific names outside its well-known list.
Aggregated deployment statistics expose request rate, error rate, latency p50/p95, TTFT p50/p95, and observed input/output tokens per second over the selected window. Token throughput is emitted only from runtime-reported usage.
The endpoint monitoring API returns the same content-free evidence as a bounded time series, plus
binding/deployment/revision attribution and persisted lifecycle overlays. The console uses this API;
it never queries PostgreSQL or an OpenTelemetry collector from the browser. Missing buckets and
unsupported runtime metrics stay null or explicitly unavailable rather than becoming zero.
Each operator-visible metric also carries a normalized evidence envelope: value, unit,
availability, evidence class, source, observation time, freshness boundary, sample count, and a
machine-readable absence reason. Availability distinguishes available, stale, not_observed,
and unsupported; evidence class distinguishes measured, provider-reported, modeled, and estimated
values. This lets CLIs, SDKs, and consoles preserve the same truth boundary instead of inferring one
from a nullable number.
Import existing production telemetry
Customers can keep Baseten, Fireworks, Datadog, or an OpenTelemetry pipeline as their monitoring source and post a strict content-free observation batch:source_request_id is deterministically scoped to the tenant,
deployment, and source, so retrying an import updates the same request record.
After import, POST /api/v1/deployments/{name}/replays captures an immutable workload-shape trace;
GET /api/v1/deployments/{name}/replays lists its history. Replay evidence contains arrival,
duration, token-count, session-hash, streaming, and concurrency shapes only. It does not duplicate
live traffic. Real prompt shadowing remains a separate explicit data-policy and approval boundary.
The Prometheus endpoint also exposes accounting queue depth and capacity, persisted and dropped
request-record counters, and persistence failures. These distinguish inference-path health from
telemetry backpressure without putting PostgreSQL on the request path. Serving-engine
running/waiting gauges are persisted with autoscaling decisions, and the wider engine summary is
persisted as operational measurements (see Serving-engine metrics);
streaming cancellations and upstream disconnects are persisted as client_cancelled and
upstream_disconnect error types. GPU metrics remain unavailable unless a qualified
provider/runtime collector supplies a trustworthy fresh measurement.
Serving-engine metrics
Every 30 seconds the control plane reads the Prometheus endpoint ({target}/metrics) of each
healthy, self-hosted replica of a running deployment and persists one bounded, content-free
summary per scrape. The engine is recognized from its metric namespace (vllm:, sglang:, or
llamacpp:), not from the deployment’s runtime name. Enable the endpoint where it is off by
default: SGLang needs --enable-metrics and llama.cpp server needs --metrics.
The scrape presents the same credential the gateway route already presents to that replica.
External provider targets and direct serverless provider targets are never scraped, because their
URLs are provider APIs. A tenant compute replica without a derivable worker credential is
skipped rather than scraped unauthenticated. A payload from an unsupported runtime is ignored and
produces no evidence.
A server observed through the InferCrane agent is scraped by the agent on
its own machine every 30 seconds. The agent keeps only these families, then relays the current
scrape with the oldest scrape of the same engine process from the last five minutes. The control
plane derives the same measurements from that pair as from its own scrapes and keys them by the
server’s target, so the endpoint’s monitoring and readiness read them the same way. A server
connected directly with infercrane connect is not scraped in this release.
Engine metric names omit their namespace prefix in the table. The three names that existed before
engine collection (
runtime_internal_itl, runtime_internal_tpot, and
runtime_kv_cache_timeseries) keep their API spelling; they now carry a p95 latency or the latest
KV-cache occupancy.
The running, waiting, and KV-cache values come from the current scrape. Every other value is a
window: the difference between the current scrape and the oldest retained scrape of the same
engine process, up to five minutes back. The first scrape after the control plane starts, after a
rollout changes the active revision, after the replica URL changes, or after the engine restarts
(any cumulative counter moving backwards) therefore records gauges only. A window with no
completed requests records latency as unavailable, while preemption and completion counts are a
measured zero. Speculative acceptance needs at least one draft token in the window, so an engine
with speculation off reports it as unavailable rather than 0%. SGLang reports its acceptance as an
engine-computed gauge, used only while spec_num_draft_tokens is positive.
Latency p95 values are estimated from the engine histogram’s bucket deltas with the same linear
interpolation Prometheus histogram_quantile uses. A p95 that falls in the +Inf bucket has no
finite bound and is reported as unavailable. Series that differ only by tensor-, pipeline-, or
expert-parallel rank labels are counted once. Data-parallel series are added. SGLang per-priority
queue breakdowns are ignored in favor of its priority="" total. runtime_tpot_mean is
request-weighted for vLLM and SGLang and token-weighted for llama.cpp server.
Each summary is valid for 90 seconds (three scrapes) and is attached to the deployment’s active
revision at ingestion, like other collector snapshots. Monitoring shows the latest summary per
replica. Without one, every engine measurement is not_observed with the reason
no serving-engine metrics scrape has reported for the active revision. When the engine is
reachable but does not export a signal, the reason says so instead. After the validity window the
envelope becomes stale with no value. Engine values are never shown as zero because they are
missing.
The monitoring API also reports shared_prefix_request_ratio. It is the share of chat requests
whose leading system messages repeat those of another request in the window, computed from the
gateway’s keyed digest of those messages. Non-chat operations and imported observations without
a digest are left out of the denominator, because their prefix shape is unknown.
Metric names were checked against the vLLM, SGLang, and llama.cpp server sources in November
2026. The tests use synthetic expositions in internal/metrics/testdata. Collection against a
live engine of each kind has not yet been qualified.
Diagnose a bottleneck from engine signals
GET /api/v1/endpoints/{name}/optimization-readiness (algorithm monitoring-stage-triage-v2)
adds engine signals to the gateway’s stage triage. Engine rules run only after the existing gates:
fresh request evidence, at least 20 requests, and an error rate at or below 2%. Each rule that
fires is returned in findings with the observed value, the threshold, and the technique families
worth measuring. Families are candidates for a measured comparison, never changes InferCrane
applies by itself, and the custom-kernel gate stays closed. A proposal can
plan from this diagnosis:
it turns the findings into setups to measure, ordered by expected benefit, with the reason recorded
on each.
Read engine findings
KV-cache pressure takes precedence over the other stages, because a full cache causes queueing
and preemption rather than following from them. A host-bound GPU comes next and is reported as
host-and-scheduler: requests wait while the GPU has idle SMs, so adding GPUs would not help. Below
it, the stage order is unchanged: queueing, then first-token, then generation. A memory-bandwidth
finding leads a generation bottleneck, a compute-bound finding leads a first-token bottleneck, and
GPU findings that do not explain the diagnosed stage are listed after its own findings. When the gateway has no first-token time (for example
non-streaming traffic), the engine’s TTFT p95 is used. Prefix-reuse and speculative findings apply
to any stage. Candidate families are ordered with families from findings about the diagnosed
bottleneck first, then families from other findings, then the stage’s general families. A missing
engine signal never satisfies a rule. When shared prefixes are common but the hit rate is unknown,
the engine prefix-cache hit rate is listed in required_evidence, as is the running-requests
gauge for a generation bottleneck without it.
These defaults are policy choices, not measured optimums:
- 90% KV occupancy leaves little headroom for growing sequences. High occupancy without waiting requests is a well-packed batch and does not fire.
- One preemption per hundred completed requests means the engine regularly recomputes evicted work.
- 2,048 mean input tokens is where the per-step batched-token budget starts to decide how prefill interleaves with decode.
- Four or fewer running sequences per replica leaves decode memory-bandwidth bound, which is where fewer target-model steps or smaller weights help most.
- A 25% token hit rate with half the requests sharing a system prefix suggests the cache is off, too small, or split across replicas. The rate is token-weighted, so long unique suffixes lower it.
- Below 50% acceptance, speculative decoding commonly spends more on drafts than it saves.
- NVIDIA documents about 80% DRAM active as the practical peak, so 60% (three quarters of it) with tensor cores mostly idle means bytes, not math, limit the work.
- 50% tensor active means the tensor pipe issues work in most cycles, where faster math or skipped recomputation is the remaining lever.
- NVIDIA documents SM active below 50% as likely ineffective GPU use. With requests waiting, the serving process rather than the GPU is the limit.
- 70% of a declared link peak leaves little headroom for collective bursts on a multi-GPU replica.
gpu_utilization) is reported as a signal but never used by a rule.
NVIDIA GPU evidence
When an environment already runs NVIDIA DCGM Exporter, an operator can import a bounded snapshot without exposing the exporter to the browser:--file dcgm.prom instead of --url for an offline snapshot. --selector accepts exact,
comma-separated Prometheus label matches. Shared exporters should always be scoped to the intended
workload. The legacy DCGM_FI_DEV_GPU_UTIL is percent by default; pass --utilization-unit ratio
only when an upstream compatibility layer deliberately rewrites it to 0..1.
dcgm-exporter 4.9 renamed most fields, for example DCGM_FI_DEV_GPU_UTIL to
DCGM_FI_DEV_GPU_UTIL_RATIO and DCGM_FI_PROF_DRAM_ACTIVE to DCGM_FI_PROF_DRAM_UTIL_RATIO.
InferCrane accepts both spellings and converts each to one unit. If an exporter emits both
spellings of one field, only the first one seen is counted.
Across replicas, monitoring sums memory, power, and XID counts, keeps the maximum of utilization,
temperature, device count, throughput, and saturation, and averages the activity percentages
weighted by GPU count.
The profiling (DCP) fields are averages over DCGM’s sampling interval, and many exporter
configurations leave them off. In dcgm-exporter 4.9 the default counters file enables tensor,
DRAM, and PCIe but not SM active, SM occupancy, or NVLink. A field the exporter does not send stays
not_observed, never zero. The 4.9 DCGM_FI_DEV_XID_ERROR field reports the last XID code rather
than a count, so it is not read.
DCGM reports link throughput but not link capacity. To get gpu_nvlink_saturation and
gpu_pcie_saturation (percent of peak, busiest GPU and direction), declare the per-GPU,
per-direction peak in GB/s that applies to your hardware:
Declare link peaks
--interval 15s (5 seconds to 10 minutes): the command then reads
the exporter and records a snapshot every interval until stopped, each fresh for three intervals unless
--ttl says otherwise. A failed read is reported and retried at the next interval; the last snapshot
goes stale on its own.
GPU utilization is not a bottleneck signal. NVIDIA defines it as the share of the sample period in
which at least one kernel was running. A GPU that runs one small kernel at a time reads 100%, so
InferCrane shows it but never diagnoses from it. SM active, tensor active, and DRAM active are the
fields that show where the work is limited.
AMD Instinct GPUs
On AMD Instinct GPUs, import a snapshot from the AMD Device Metrics Exporter instead:
Pass
--metric-prefix when the exporter’s MetricsFieldPrefix is set. Other fields stay unknown:
gpu_used_vram is documented in MB without saying whether that is MiB, gpu_umc_activity
(memory controller) is not shown to equal DCGM’s DRAM active, and the GPU_PROF_* counters need
profiler mode and are not shown to equal SM, tensor, or DRAM active. So the GPU rules above do not
apply to AMD hardware, and the exporter’s values have not been checked against a live AMD GPU. See
AMD Instinct GPUs.
The snapshot is attached to the active revision at ingestion time. It cannot silently move to a
later rollout. Once its TTL expires, the monitoring API returns a stale envelope with no value.
This integration proves the control-plane semantics locally; actual DCGM field availability varies
by GPU and DCGM version and still requires real-hardware qualification. Collector snapshots follow
the configured high-volume request-evidence retention period (INFERCRANE_REQUEST_RETENTION_HOURS).
GPU profiles
A GPU profile shows where GPU time goes, from Python and PyTorch down through the CUDA launch to each kernel, its SASS instructions, and the reason its warps were waiting.infercrane profile captures one
through the serving engine’s own profiler, analyzes it, and records it: see
GPU profiling. This section describes the call-stack recording format, which
any collector can also post itself:
capture.json follows infercrane.gpu-profile/v1. Frames are listed once and each stack refers to
them by index, root first:
kindispython,torch,native,cuda_api,kernel,sass, orstall. Key Python frames by the function’s definition line, not the call site, so one function stays one frame.gpu_nsis GPU time attributed to exactly that stack. All stacks together cannot exceed the capture length times its GPU count.- Stall frames use the CUPTI PC sampling reason without its
smsp__pcsamp_warps_issue_stalled_prefix, for examplelong_scoreboardormath_pipe_throttle. - A bucket with no samples has
gpu_seconds: null, never0. - List the layers the tool could not record in
unavailable_layers. A capture without PC sampling, for example, namessassandstall, and the dashboard then says stall reasons were not recorded instead of drawing an empty layer. - Only
evidence_class: "measured"is accepted, andcontent_recordedmust befalse. Frames carry code identifiers only, never prompts or outputs.
GET /api/v1/endpoints/{name}/gpu-profile returns the newest capture of the active revision of any
deployment the endpoint routes to, with the endpoint, deployment, and revision number it belongs
to. It needs only read permission. After a rollout, captures of the previous revision are no longer
returned: they describe code that no longer serves. When nothing is recorded the API answers 404
gpu_profile_not_found, and the dashboard shows Not recorded under Observe › Profiling with
Advanced on, rather than an empty graph. Recording and reading a profile never touch the inference
path.
A infercrane.gpu-profile/v2 recording adds the trace analysis and a timeline excerpt; its fields,
the capture history, and before-and-after comparison are on GPU profiling.
A capture that ended more than 30 days ago is no longer returned.