> ## Documentation Index
> Fetch the complete documentation index at: https://docs.infercrane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AMD Instinct GPUs

> Deploy and optimize open models on AMD Instinct MI300X, MI325X, MI350X, and MI355X with pinned ROCm builds of vLLM and SGLang.

# AMD Instinct GPUs

InferCrane can deploy and optimize open models on AMD Instinct GPUs with the same engine versions it
runs on NVIDIA: vLLM 0.22.1 and SGLang 0.5.12, built for ROCm and pinned by digest. The deployment
spec, rollout, routing, AIPerf measurement, engine metrics, and the optimizer's rules are the same on
both vendors. What differs, the runtime image and where the GPU can be rented, is data, not code.

> [!WARNING]
>
> No model has been served or measured on an AMD GPU through InferCrane yet. Everything on this page
> is checked by hermetic tests against fake providers and recorded source code. Speed, cost, and
> answer quality on AMD stay unknown until a recorded run exists.

## What is supported

| GPU | Architecture | Target | Memory | Platform | Where you can run it |
| - | - | - | - | - | - |
| MI300X | CDNA 3 | `gfx942` | 192 GB | `rocm-gfx942` | RunPod Pods, your Kubernetes cluster |
| MI325X | CDNA 3 | `gfx942` | 256 GB | `rocm-gfx942` | Your Kubernetes cluster |
| MI350X | CDNA 4 | `gfx950` | 288 GB | `rocm-gfx950` | RunPod Pods, your Kubernetes cluster |
| MI355X | CDNA 4 | `gfx950` | 288 GB | `rocm-gfx950` | Your Kubernetes cluster |

Architecture, target, and memory come from the
[ROCm GPU specifications](https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html).
InferCrane recognizes each GPU under its short name (`MI300X`), its product name
(`AMD Instinct MI300X`), RunPod's GPU type (`AMD Instinct MI300X OAM`), and the AMD node labeller's
label value (`AMD_Instinct_MI300X_OAM`). Any other name, including other AMD GPUs such as MI250X or
MI300A, is not recognized and keeps the default CUDA images.

`GET /api/v1/catalog/accelerators` returns this list with sources, the pinned images, and where each
platform may run.

### Pinned runtime images

| Runtime | GPUs | Image | Stack |
| - | - | - | - |
| vLLM 0.22.1 | all four | `vllm/vllm-openai-rocm:v0.22.1@sha256:368b2992c776ad653e40e9e00d38aa51cf1e9a16558b05d07c42818abd9ec031` | ROCm 7.2.2, AITER v0.1.13, built for `gfx942` and `gfx950` |
| SGLang 0.5.12 | MI300X, MI325X | `lmsysorg/sglang:v0.5.12-rocm720-mi30x@sha256:1f5c080be74269393bdc07d2cb30127dd2931c45b0b53e448235fb14d854da63` | ROCm 7.2, `gfx942` |
| SGLang 0.5.12 | MI350X, MI355X | `lmsysorg/sglang:v0.5.12-rocm720-mi35x@sha256:3be1556105cdfdede103dc053f002d4d0307bdb229d0e15bfef130d5c53af208` | ROCm 7.2, `gfx950` |

Each digest was read from the registry manifest of the named tag on 2026-11-17. The images are the
engines' own release builds; InferCrane does not build or modify them. The command, port, `/health`,
`/v1/models`, and `/metrics` paths are the same as on NVIDIA.

### Where AMD GPUs can run

| Provider | AMD GPUs | Why |
| - | - | - |
| RunPod Pods (`runpod` elastic) | Yes: MI300X, MI350X | RunPod rents both as secure-cloud pods. It lists no MI325X or MI355X. |
| Kubernetes (`kubernetes` elastic) | Yes: all four | Your cluster's AMD GPU device plugin exposes them as `amd.com/gpu`. |
| RunPod Serverless | No | The serverless worker image is a CUDA build. |
| AWS EC2, GCP Compute | No | Neither offers AMD Instinct instances. |
| Dynamo on Kubernetes | No | Its runtime images are CUDA builds. |
| Any SkyPilot adapter, including RunPod through SkyPilot | No | SkyPilot reaches its node over SSH, and only InferCrane's CUDA images carry it. On RunPod, AMD GPUs use the `runpod-pods` adapter. |

A spec that asks for an AMD GPU anywhere else is refused before anything is provisioned, with the
places where its platform is qualified. vLLM and custom OCI images run on RunPod and Kubernetes;
SGLang runs on Kubernetes, as on NVIDIA. A custom OCI image must itself be a ROCm build.

## Deploy on RunPod

```yaml title="examples/runpod-mi300x.yaml" theme={"theme":"css-variables"}
apiVersion: infercrane.dev/v1
kind: Deployment
name: qwen3-8b-mi300x
model:
  id: Qwen/Qwen3-8B
  revision: b968826d9c46dd6066d109eabc6255188de91218
runtime:
  engine: vllm
  version: 0.22.1
compute: {mode: elastic}
provider: {cloud: runpod, adapter: runpod-pods}
resources: {gpu: MI300X}
scaling: {min_replicas: 1, max_replicas: 1}
routing: {strategy: round-robin}
```

InferCrane asks RunPod for GPU type `AMD Instinct MI300X OAM` and starts the vLLM ROCm image. Prices
come from RunPod's API under that exact GPU type, the same way as NVIDIA prices. When RunPod has no
current secure-cloud offer, the price stays unknown; it is never filled in from a list price. On
2026-11-17 RunPod listed the MI300X but returned no current offer.

## Deploy on Kubernetes

Install AMD's [GPU device plugin and node labeller](https://github.com/ROCm/k8s-device-plugin)
(Apache-2.0), or the AMD GPU Operator that ships both. Then point InferCrane at them:

```bash theme={"theme":"css-variables"}
export INFERCRANE_KUBERNETES_GPU_RESOURCE='amd.com/gpu'
export INFERCRANE_KUBERNETES_GPU_PRODUCT_LABEL='amd.com/gpu.product-name'
```

Use the node's `amd.com/gpu.product-name` label value as the GPU in the spec, for example
`resources: {gpu: AMD_Instinct_MI300X_OAM}`. The labeller writes the product name with spaces
replaced by underscores; check yours with `kubectl get nodes -L amd.com/gpu.product-name`.

One control plane schedules one device resource. A GPU whose device plugin does not match the
configured resource, such as an MI300X on a cluster configured for `nvidia.com/gpu`, is refused
before anything is applied. Other resource names, such as NVIDIA MIG slices, are not checked.

## How the optimizer plans on AMD

The optimizer's rules read the endpoint's diagnosis, not the vendor, so the same bottleneck chooses
the same changes on an MI300X as on an H100. Three things depend on the GPU:

* **Engine defaults.** The ROCm images run the same engine versions as the CUDA ones, so the
  recorded defaults hold. vLLM 0.22.1 sizes its batch defaults from device memory on every platform
  ([`EngineArgs.get_batch_defaults`](https://github.com/vllm-project/vllm/blob/v0.22.1/vllm/engine/arg_utils.py));
  every reviewed AMD GPU has at least 70 GB, so it starts with 1,024 sequences and 8,192 batched
  tokens and a setup that restates those values is dropped as a no-op. vLLM's ROCm platform does not
  change scheduler or cache defaults, and SGLang 0.5.12's recorded defaults (KV cache dtype,
  quantization, schedule policy, speculation) are the same on ROCm.
* **FP8.** CDNA 3 and CDNA 4 run FP8 matrix math in hardware (AMD's product pages list FP8
  throughput for all four). vLLM 0.22.1 lists FP8 W8A8 as supported on AMD GPUs and an FP8 (e4m3)
  KV cache on ROCm
  ([quantization](https://github.com/vllm-project/vllm/blob/v0.22.1/docs/features/quantization/README.md#supported-hardware),
  [cache config](https://github.com/vllm-project/vllm/blob/v0.22.1/vllm/config/cache.py)), and
  SGLang 0.5.12 runs FP8 on AMD through AITER or Triton
  ([AMD GPU guide](https://github.com/sgl-project/sglang/blob/v0.5.12/docs/platforms/amd_gpu.md#quantization-on-amd-gpus)).
  So FP8 weights and an FP8 KV cache can be chosen on all four GPUs.
* **AITER kernels.** SGLang's ROCm images turn AMD's AITER kernels on by default
  (`SGLANG_USE_AITER=1` in the image), so there is nothing to switch. vLLM 0.22.1 leaves them off
  unless `VLLM_ROCM_USE_AITER=1` is set, and the pinned image does not set it. When the diagnosis
  says the GPU's work is the bottleneck, the optimizer adds a setup that is the starting setup with
  only that switch on. See [Turn on AITER kernels for vLLM](#turn-on-aiter-kernels-for-vllm).

The custom kernel lab does not build HIP kernels yet; it still targets NVIDIA only.

### Publisher checkpoints for AMD

AMD publishes quantized checkpoints on Hugging Face, and the
[reviewed checkpoint catalog](https://docs.infercrane.com/features/optimization.md#seed-candidates-from-publisher-checkpoints)
tracks them. The catalog records each build against the compiler targets it runs on: a format the
policy reviews for CDNA 3 is `gfx942` (MI300X, MI325X), and one for CDNA 4 is `gfx950` (MI350X,
MI355X). A build is offered only on a GPU with one of its targets, so an MXFP4 build for CDNA 4 is
never proposed on an MI300X.

A build can seed a setup when every catalog gate passes and the pinned ROCm image of one of its
reviewed runtimes is at least the engine version its model card states. A build whose card needs a
newer engine stays in the catalog with that reason, for example `vllm needs 0.28.0 or newer,
pinned 0.22.1`.

The catalog reviewed on 2026-11-17 lists 46 AMD builds, each recorded as runnable: 27 on `gfx942`
and `gfx950`, and 19 on `gfx950` only. None seeds a setup yet:

* LLM Compressor builds (W4A16, W8A8) run on vLLM and can seed a setup on a matching GPU. All 18
  AMD LLM Compressor builds state vLLM 0.26.0 to 0.31.0, newer than the pinned 0.22.1.
* Quark builds (MXFP4, PTPC-FP8, INT4) stay catalog-only: Quark is not yet a tool whose
  checkpoints can seed a setup. There are 27.
* DFlash speculators stay catalog-only, as on NVIDIA. There is one.

A seeded build is unmeasured like every other candidate. It is compared with the unmodified model
on the same GPU before it can win.

## Turn on AITER kernels for vLLM

[AITER](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)
is AMD's library of fused attention, matrix-multiply, MoE, and RMSNorm kernels for MI300X, MI325X,
and the MI350 Series. vLLM uses it only when `VLLM_ROCM_USE_AITER` is `1` or `true`; the default is
off ([`vllm/envs.py` at v0.22.1](https://github.com/vllm-project/vllm/blob/v0.22.1/vllm/envs.py)),
and the pinned image sets `AITER_ROCM_ARCH=gfx942;gfx950` but not the switch. Once it is on, vLLM's
AITER sub-switches for linear layers, MoE, RMSNorm, MLA, and MHA default to on. AMD's vLLM
optimization guide and the
[vLLM recipes' AMD preset](https://github.com/vllm-project/recipes/blob/fa02ae800b4998a7985e3654116b602fb6c139df/SCHEMA.md)
both set it.

Set it in the deployment spec's `runtime.env`:

```yaml theme={"theme":"css-variables"}
runtime:
  engine: vllm
  version: 0.22.1
  env:
    VLLM_ROCM_USE_AITER: "1"
resources: {gpu: MI300X}
```

`runtime.env` accepts only reviewed engine switches with reviewed values; see
[Engine environment switches](https://docs.infercrane.com/deployment-spec.md#engine-environment-switches). This one is reviewed
for vLLM 0.22.1's pinned ROCm image on `rocm-gfx942` and `rocm-gfx950`, with the values `0` and `1`.
On an NVIDIA GPU, on SGLang, or on another vLLM version or image it is refused. RunPod Pods,
Kubernetes, AWS EC2, and GCP Compute pass it into the runtime container; SkyPilot, RunPod Serverless,
and Kubernetes Dynamo cannot, so a deployment that sets it there is refused before anything is
provisioned.

### When the optimizer chooses it

The rule `accelerator.kernels_off_by_default` fires only on a reviewed AMD Instinct GPU whose
starting setup runs vLLM 0.22.1 with the switch off, and only when the diagnosis points at kernel
time:

| Diagnosis | Priority |
| - | - |
| Writing answers is the bottleneck | 75 |
| First tokens are the bottleneck on prompts of 1,024 tokens or more | 75 |
| Attention kernels ≥ 40% or matrix-multiply kernels ≥ 50% of profiled GPU time | 70 |
| Trace: attention ≥ 35% of kernel time (`gpu.attention_heavy`) | 70 |

When the profile capture has a trace analysis, its attention finding replaces the attention share,
as it does for the FP8 KV cache rule.

A full KV cache, a queue, or short prompts are not kernel time, so the rule does not fire for
them. The setup changes only the switch: same flags, model, image, and GPU. Because it swaps the
kernels that compute each layer, it needs the answer-quality check against the starting setup, like
a precision change. A starting setup that already sets the switch is not changed again.

AMD's guide reports speedups for AITER on its own hardware and models. InferCrane has not measured
AITER on any GPU; the campaign decides whether it wins on your traffic.

## Measure on AMD

* **AIPerf** load and answer-quality checks call the OpenAI-compatible endpoint, so they run
  unchanged.
* **Engine metrics** are read from the same `/metrics` path with the same names on vLLM and SGLang.
* **GPU telemetry** comes from the
  [AMD Device Metrics Exporter](https://github.com/ROCm/device-metrics-exporter) (Apache-2.0):

  ```bash theme={"theme":"css-variables"}
  infercrane telemetry collect amd qwen3-8b-mi300x \
    --url http://127.0.0.1:5000/metrics \
    --selector pod=qwen3-8b-mi300x-0
  ```

  | Exporter field (v1.5.3) | InferCrane measurement |
  | - | - |
  | `gpu_gfx_activity` (percent) | `gpu_utilization`, maximum across GPUs |
  | `gpu_package_power`, or `gpu_power_usage` (watts) | `gpu_power`, sum |
  | `gpu_junction_temperature` (Celsius, hotspot sensor) | `gpu_temperature`, maximum |
  | distinct `serial_number` labels | `gpu_devices` |

  Use `--metric-prefix` if the exporter sets a `MetricsFieldPrefix`. Memory used, memory-controller
  activity, and the `GPU_PROF_*` profiler counters are not read: their units or their equivalence to
  NVIDIA's SM, tensor, and DRAM activity are not confirmed. So the GPU bottleneck rules (memory
  bandwidth, compute, host, interconnect) do not fire on AMD, and GPU utilization is shown but never
  diagnosed from, as on NVIDIA.
* **GPU profiles.** A recording accepts any capture tool. On AMD, capture with
  [rocprofv3](https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/latest/) and list `sass` and
  `stall` as unavailable layers; those are NVIDIA PC-sampling layers.

## Not yet measured

* No AMD run exists. The smallest qualification run (one MI300X, Qwen3-8B, about an hour) is
  planned and needs approval before any GPU is rented.
* Other AMD GPUs (MI250X, MI300A, Radeon) are not recognized.
* No publisher checkpoint has been loaded on an AMD GPU. Quark checkpoints cannot seed a setup yet
  (see [Publisher checkpoints for AMD](#publisher-checkpoints-for-amd)).
* AITER for vLLM is planned but not measured. ROCm attention backend flags
  (`--attention-backend`) and AITER sub-switches are not reviewed.
* AMD memory, memory-controller, and profiler counters are not mapped.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.