Skip to main content

AMD Instinct GPUs

InferCrane can deploy and optimize open models on AMD Instinct GPUs with the same engine versions it runs on NVIDIA: vLLM 0.22.1 and SGLang 0.5.12, built for ROCm and pinned by digest. The deployment spec, rollout, routing, AIPerf measurement, engine metrics, and the optimizer’s rules are the same on both vendors. What differs, the runtime image and where the GPU can be rented, is data, not code.
No model has been served or measured on an AMD GPU through InferCrane yet. Everything on this page is checked by hermetic tests against fake providers and recorded source code. Speed, cost, and answer quality on AMD stay unknown until a recorded run exists.

What is supported

Architecture, target, and memory come from the ROCm GPU specifications. InferCrane recognizes each GPU under its short name (MI300X), its product name (AMD Instinct MI300X), RunPod’s GPU type (AMD Instinct MI300X OAM), and the AMD node labeller’s label value (AMD_Instinct_MI300X_OAM). Any other name, including other AMD GPUs such as MI250X or MI300A, is not recognized and keeps the default CUDA images. GET /api/v1/catalog/accelerators returns this list with sources, the pinned images, and where each platform may run.

Pinned runtime images

Each digest was read from the registry manifest of the named tag on 2026-11-17. The images are the engines’ own release builds; InferCrane does not build or modify them. The command, port, /health, /v1/models, and /metrics paths are the same as on NVIDIA.

Where AMD GPUs can run

A spec that asks for an AMD GPU anywhere else is refused before anything is provisioned, with the places where its platform is qualified. vLLM and custom OCI images run on RunPod and Kubernetes; SGLang runs on Kubernetes, as on NVIDIA. A custom OCI image must itself be a ROCm build.

Deploy on RunPod

examples/runpod-mi300x.yaml
InferCrane asks RunPod for GPU type AMD Instinct MI300X OAM and starts the vLLM ROCm image. Prices come from RunPod’s API under that exact GPU type, the same way as NVIDIA prices. When RunPod has no current secure-cloud offer, the price stays unknown; it is never filled in from a list price. On 2026-11-17 RunPod listed the MI300X but returned no current offer.

Deploy on Kubernetes

Install AMD’s GPU device plugin and node labeller (Apache-2.0), or the AMD GPU Operator that ships both. Then point InferCrane at them:
Use the node’s amd.com/gpu.product-name label value as the GPU in the spec, for example resources: {gpu: AMD_Instinct_MI300X_OAM}. The labeller writes the product name with spaces replaced by underscores; check yours with kubectl get nodes -L amd.com/gpu.product-name. One control plane schedules one device resource. A GPU whose device plugin does not match the configured resource, such as an MI300X on a cluster configured for nvidia.com/gpu, is refused before anything is applied. Other resource names, such as NVIDIA MIG slices, are not checked.

How the optimizer plans on AMD

The optimizer’s rules read the endpoint’s diagnosis, not the vendor, so the same bottleneck chooses the same changes on an MI300X as on an H100. Three things depend on the GPU:
  • Engine defaults. The ROCm images run the same engine versions as the CUDA ones, so the recorded defaults hold. vLLM 0.22.1 sizes its batch defaults from device memory on every platform (EngineArgs.get_batch_defaults); every reviewed AMD GPU has at least 70 GB, so it starts with 1,024 sequences and 8,192 batched tokens and a setup that restates those values is dropped as a no-op. vLLM’s ROCm platform does not change scheduler or cache defaults, and SGLang 0.5.12’s recorded defaults (KV cache dtype, quantization, schedule policy, speculation) are the same on ROCm.
  • FP8. CDNA 3 and CDNA 4 run FP8 matrix math in hardware (AMD’s product pages list FP8 throughput for all four). vLLM 0.22.1 lists FP8 W8A8 as supported on AMD GPUs and an FP8 (e4m3) KV cache on ROCm (quantization, cache config), and SGLang 0.5.12 runs FP8 on AMD through AITER or Triton (AMD GPU guide). So FP8 weights and an FP8 KV cache can be chosen on all four GPUs.
  • AITER kernels. SGLang’s ROCm images turn AMD’s AITER kernels on by default (SGLANG_USE_AITER=1 in the image), so there is nothing to switch. vLLM 0.22.1 leaves them off unless VLLM_ROCM_USE_AITER=1 is set, and the pinned image does not set it. When the diagnosis says the GPU’s work is the bottleneck, the optimizer adds a setup that is the starting setup with only that switch on. See Turn on AITER kernels for vLLM.
The custom kernel lab does not build HIP kernels yet; it still targets NVIDIA only.

Publisher checkpoints for AMD

AMD publishes quantized checkpoints on Hugging Face, and the reviewed checkpoint catalog tracks them. The catalog records each build against the compiler targets it runs on: a format the policy reviews for CDNA 3 is gfx942 (MI300X, MI325X), and one for CDNA 4 is gfx950 (MI350X, MI355X). A build is offered only on a GPU with one of its targets, so an MXFP4 build for CDNA 4 is never proposed on an MI300X. A build can seed a setup when every catalog gate passes and the pinned ROCm image of one of its reviewed runtimes is at least the engine version its model card states. A build whose card needs a newer engine stays in the catalog with that reason, for example vllm needs 0.28.0 or newer, pinned 0.22.1. The catalog reviewed on 2026-11-17 lists 46 AMD builds, each recorded as runnable: 27 on gfx942 and gfx950, and 19 on gfx950 only. None seeds a setup yet:
  • LLM Compressor builds (W4A16, W8A8) run on vLLM and can seed a setup on a matching GPU. All 18 AMD LLM Compressor builds state vLLM 0.26.0 to 0.31.0, newer than the pinned 0.22.1.
  • Quark builds (MXFP4, PTPC-FP8, INT4) stay catalog-only: Quark is not yet a tool whose checkpoints can seed a setup. There are 27.
  • DFlash speculators stay catalog-only, as on NVIDIA. There is one.
A seeded build is unmeasured like every other candidate. It is compared with the unmodified model on the same GPU before it can win.

Turn on AITER kernels for vLLM

AITER is AMD’s library of fused attention, matrix-multiply, MoE, and RMSNorm kernels for MI300X, MI325X, and the MI350 Series. vLLM uses it only when VLLM_ROCM_USE_AITER is 1 or true; the default is off (vllm/envs.py at v0.22.1), and the pinned image sets AITER_ROCM_ARCH=gfx942;gfx950 but not the switch. Once it is on, vLLM’s AITER sub-switches for linear layers, MoE, RMSNorm, MLA, and MHA default to on. AMD’s vLLM optimization guide and the vLLM recipes’ AMD preset both set it. Set it in the deployment spec’s runtime.env:
runtime.env accepts only reviewed engine switches with reviewed values; see Engine environment switches. This one is reviewed for vLLM 0.22.1’s pinned ROCm image on rocm-gfx942 and rocm-gfx950, with the values 0 and 1. On an NVIDIA GPU, on SGLang, or on another vLLM version or image it is refused. RunPod Pods, Kubernetes, AWS EC2, and GCP Compute pass it into the runtime container; SkyPilot, RunPod Serverless, and Kubernetes Dynamo cannot, so a deployment that sets it there is refused before anything is provisioned.

When the optimizer chooses it

The rule accelerator.kernels_off_by_default fires only on a reviewed AMD Instinct GPU whose starting setup runs vLLM 0.22.1 with the switch off, and only when the diagnosis points at kernel time: When the profile capture has a trace analysis, its attention finding replaces the attention share, as it does for the FP8 KV cache rule. A full KV cache, a queue, or short prompts are not kernel time, so the rule does not fire for them. The setup changes only the switch: same flags, model, image, and GPU. Because it swaps the kernels that compute each layer, it needs the answer-quality check against the starting setup, like a precision change. A starting setup that already sets the switch is not changed again. AMD’s guide reports speedups for AITER on its own hardware and models. InferCrane has not measured AITER on any GPU; the campaign decides whether it wins on your traffic.

Measure on AMD

  • AIPerf load and answer-quality checks call the OpenAI-compatible endpoint, so they run unchanged.
  • Engine metrics are read from the same /metrics path with the same names on vLLM and SGLang.
  • GPU telemetry comes from the AMD Device Metrics Exporter (Apache-2.0):
    Use --metric-prefix if the exporter sets a MetricsFieldPrefix. Memory used, memory-controller activity, and the GPU_PROF_* profiler counters are not read: their units or their equivalence to NVIDIA’s SM, tensor, and DRAM activity are not confirmed. So the GPU bottleneck rules (memory bandwidth, compute, host, interconnect) do not fire on AMD, and GPU utilization is shown but never diagnosed from, as on NVIDIA.
  • GPU profiles. A recording accepts any capture tool. On AMD, capture with rocprofv3 and list sass and stall as unavailable layers; those are NVIDIA PC-sampling layers.

Not yet measured

  • No AMD run exists. The smallest qualification run (one MI300X, Qwen3-8B, about an hour) is planned and needs approval before any GPU is rented.
  • Other AMD GPUs (MI250X, MI300A, Radeon) are not recognized.
  • No publisher checkpoint has been loaded on an AMD GPU. Quark checkpoints cannot seed a setup yet (see Publisher checkpoints for AMD).
  • AITER for vLLM is planned but not measured. ROCm attention backend flags (--attention-backend) and AITER sub-switches are not reviewed.
  • AMD memory, memory-controller, and profiler counters are not mapped.