AMD Instinct GPUs
InferCrane can deploy and optimize open models on AMD Instinct GPUs with the same engine versions it runs on NVIDIA: vLLM 0.22.1 and SGLang 0.5.12, built for ROCm and pinned by digest. The deployment spec, rollout, routing, AIPerf measurement, engine metrics, and the optimizer’s rules are the same on both vendors. What differs, the runtime image and where the GPU can be rented, is data, not code.What is supported
Architecture, target, and memory come from the
ROCm GPU specifications.
InferCrane recognizes each GPU under its short name (
MI300X), its product name
(AMD Instinct MI300X), RunPod’s GPU type (AMD Instinct MI300X OAM), and the AMD node labeller’s
label value (AMD_Instinct_MI300X_OAM). Any other name, including other AMD GPUs such as MI250X or
MI300A, is not recognized and keeps the default CUDA images.
GET /api/v1/catalog/accelerators returns this list with sources, the pinned images, and where each
platform may run.
Pinned runtime images
Each digest was read from the registry manifest of the named tag on 2026-11-17. The images are the
engines’ own release builds; InferCrane does not build or modify them. The command, port,
/health,
/v1/models, and /metrics paths are the same as on NVIDIA.
Where AMD GPUs can run
A spec that asks for an AMD GPU anywhere else is refused before anything is provisioned, with the
places where its platform is qualified. vLLM and custom OCI images run on RunPod and Kubernetes;
SGLang runs on Kubernetes, as on NVIDIA. A custom OCI image must itself be a ROCm build.
Deploy on RunPod
examples/runpod-mi300x.yaml
AMD Instinct MI300X OAM and starts the vLLM ROCm image. Prices
come from RunPod’s API under that exact GPU type, the same way as NVIDIA prices. When RunPod has no
current secure-cloud offer, the price stays unknown; it is never filled in from a list price. On
2026-11-17 RunPod listed the MI300X but returned no current offer.
Deploy on Kubernetes
Install AMD’s GPU device plugin and node labeller (Apache-2.0), or the AMD GPU Operator that ships both. Then point InferCrane at them:amd.com/gpu.product-name label value as the GPU in the spec, for example
resources: {gpu: AMD_Instinct_MI300X_OAM}. The labeller writes the product name with spaces
replaced by underscores; check yours with kubectl get nodes -L amd.com/gpu.product-name.
One control plane schedules one device resource. A GPU whose device plugin does not match the
configured resource, such as an MI300X on a cluster configured for nvidia.com/gpu, is refused
before anything is applied. Other resource names, such as NVIDIA MIG slices, are not checked.
How the optimizer plans on AMD
The optimizer’s rules read the endpoint’s diagnosis, not the vendor, so the same bottleneck chooses the same changes on an MI300X as on an H100. Three things depend on the GPU:- Engine defaults. The ROCm images run the same engine versions as the CUDA ones, so the
recorded defaults hold. vLLM 0.22.1 sizes its batch defaults from device memory on every platform
(
EngineArgs.get_batch_defaults); every reviewed AMD GPU has at least 70 GB, so it starts with 1,024 sequences and 8,192 batched tokens and a setup that restates those values is dropped as a no-op. vLLM’s ROCm platform does not change scheduler or cache defaults, and SGLang 0.5.12’s recorded defaults (KV cache dtype, quantization, schedule policy, speculation) are the same on ROCm. - FP8. CDNA 3 and CDNA 4 run FP8 matrix math in hardware (AMD’s product pages list FP8 throughput for all four). vLLM 0.22.1 lists FP8 W8A8 as supported on AMD GPUs and an FP8 (e4m3) KV cache on ROCm (quantization, cache config), and SGLang 0.5.12 runs FP8 on AMD through AITER or Triton (AMD GPU guide). So FP8 weights and an FP8 KV cache can be chosen on all four GPUs.
- AITER kernels. SGLang’s ROCm images turn AMD’s AITER kernels on by default
(
SGLANG_USE_AITER=1in the image), so there is nothing to switch. vLLM 0.22.1 leaves them off unlessVLLM_ROCM_USE_AITER=1is set, and the pinned image does not set it. When the diagnosis says the GPU’s work is the bottleneck, the optimizer adds a setup that is the starting setup with only that switch on. See Turn on AITER kernels for vLLM.
Publisher checkpoints for AMD
AMD publishes quantized checkpoints on Hugging Face, and the reviewed checkpoint catalog tracks them. The catalog records each build against the compiler targets it runs on: a format the policy reviews for CDNA 3 isgfx942 (MI300X, MI325X), and one for CDNA 4 is gfx950 (MI350X,
MI355X). A build is offered only on a GPU with one of its targets, so an MXFP4 build for CDNA 4 is
never proposed on an MI300X.
A build can seed a setup when every catalog gate passes and the pinned ROCm image of one of its
reviewed runtimes is at least the engine version its model card states. A build whose card needs a
newer engine stays in the catalog with that reason, for example vllm needs 0.28.0 or newer, pinned 0.22.1.
The catalog reviewed on 2026-11-17 lists 46 AMD builds, each recorded as runnable: 27 on gfx942
and gfx950, and 19 on gfx950 only. None seeds a setup yet:
- LLM Compressor builds (W4A16, W8A8) run on vLLM and can seed a setup on a matching GPU. All 18 AMD LLM Compressor builds state vLLM 0.26.0 to 0.31.0, newer than the pinned 0.22.1.
- Quark builds (MXFP4, PTPC-FP8, INT4) stay catalog-only: Quark is not yet a tool whose checkpoints can seed a setup. There are 27.
- DFlash speculators stay catalog-only, as on NVIDIA. There is one.
Turn on AITER kernels for vLLM
AITER is AMD’s library of fused attention, matrix-multiply, MoE, and RMSNorm kernels for MI300X, MI325X, and the MI350 Series. vLLM uses it only whenVLLM_ROCM_USE_AITER is 1 or true; the default is
off (vllm/envs.py at v0.22.1),
and the pinned image sets AITER_ROCM_ARCH=gfx942;gfx950 but not the switch. Once it is on, vLLM’s
AITER sub-switches for linear layers, MoE, RMSNorm, MLA, and MHA default to on. AMD’s vLLM
optimization guide and the
vLLM recipes’ AMD preset
both set it.
Set it in the deployment spec’s runtime.env:
runtime.env accepts only reviewed engine switches with reviewed values; see
Engine environment switches. This one is reviewed
for vLLM 0.22.1’s pinned ROCm image on rocm-gfx942 and rocm-gfx950, with the values 0 and 1.
On an NVIDIA GPU, on SGLang, or on another vLLM version or image it is refused. RunPod Pods,
Kubernetes, AWS EC2, and GCP Compute pass it into the runtime container; SkyPilot, RunPod Serverless,
and Kubernetes Dynamo cannot, so a deployment that sets it there is refused before anything is
provisioned.
When the optimizer chooses it
The ruleaccelerator.kernels_off_by_default fires only on a reviewed AMD Instinct GPU whose
starting setup runs vLLM 0.22.1 with the switch off, and only when the diagnosis points at kernel
time:
When the profile capture has a trace analysis, its attention finding replaces the attention share,
as it does for the FP8 KV cache rule.
A full KV cache, a queue, or short prompts are not kernel time, so the rule does not fire for
them. The setup changes only the switch: same flags, model, image, and GPU. Because it swaps the
kernels that compute each layer, it needs the answer-quality check against the starting setup, like
a precision change. A starting setup that already sets the switch is not changed again.
AMD’s guide reports speedups for AITER on its own hardware and models. InferCrane has not measured
AITER on any GPU; the campaign decides whether it wins on your traffic.
Measure on AMD
- AIPerf load and answer-quality checks call the OpenAI-compatible endpoint, so they run unchanged.
-
Engine metrics are read from the same
/metricspath with the same names on vLLM and SGLang. -
GPU telemetry comes from the
AMD Device Metrics Exporter (Apache-2.0):
Use
--metric-prefixif the exporter sets aMetricsFieldPrefix. Memory used, memory-controller activity, and theGPU_PROF_*profiler counters are not read: their units or their equivalence to NVIDIA’s SM, tensor, and DRAM activity are not confirmed. So the GPU bottleneck rules (memory bandwidth, compute, host, interconnect) do not fire on AMD, and GPU utilization is shown but never diagnosed from, as on NVIDIA. -
GPU profiles. A recording accepts any capture tool. On AMD, capture with
rocprofv3 and list
sassandstallas unavailable layers; those are NVIDIA PC-sampling layers.
Not yet measured
- No AMD run exists. The smallest qualification run (one MI300X, Qwen3-8B, about an hour) is planned and needs approval before any GPU is rented.
- Other AMD GPUs (MI250X, MI300A, Radeon) are not recognized.
- No publisher checkpoint has been loaded on an AMD GPU. Quark checkpoints cannot seed a setup yet (see Publisher checkpoints for AMD).
- AITER for vLLM is planned but not measured. ROCm attention backend flags
(
--attention-backend) and AITER sub-switches are not reviewed. - AMD memory, memory-controller, and profiler counters are not mapped.