DeepSeek V4 Flash on a Single AMD MI300X

This repository contains the configuration and patches I use to run deepseek-ai/DeepSeek-V4-Flash-0731 on one AMD MI300X in production. It includes the Docker Compose stack, SHA-256-pinned file overlays, reference diffs against upstream, and tuning tables. The checkpoint runs as shipped, without additional weight quantization or offload.

Results from the pinned stack (vLLM ROCm nightly 0.26.1rc1.dev229+g124154a88.rocm723 , AITER 0.1.19 ):

The official vLLM recipe targets NVIDIA and newer AMD hardware. Running the model reliably on MI300X required fixes for its FP8 format, MoE routing at high concurrency, causal speculative verification, CPU-KV synchronization, and several untuned kernel shapes. This repository collects those fixes and pins the versions used in production.

The MI300X has 192 GB of HBM3 and 5.3 TB/s of memory bandwidth, with 2.4× the HBM capacity of an H100 SXM5 ( AMD ). Doubleword's write-up estimates that it costs roughly half as much at list price. For this 304B-parameter checkpoint, the memory capacity allows a simple single-GPU deployment:

MI300X (CDNA3) implements the AMD/Graphcore fnuz variant of E4M3, while MI325X and newer use OCP-standard FP8 ( background ). A kernel that assumes OCP semantics on MI300X can be wrong by a factor of two in the scale domain. Correctness on this FP8 implementation was the first priority; performance tuning came afterward.

Fergus Finn's MI300X worklog and the accompanying Doubleword repository identified the FP8 incompatibility, missing AITER fast paths on gfx942 , HIP-graph hazards in sparse MLA decode, and MoE routing bugs. The official vLLM recipe covers NVIDIA hardware and newer AMD GPUs (MI325X at 4K context and MI355X), but not a single-MI300X production configuration for the 0731 checkpoint.

. ├── compose.yaml # The production stack (vLLM ROCm + Caddy), digest-pinned ├── Caddyfile.example # Copy to Caddyfile; set hostname, email, and source CIDR ├── vllm-entrypoint.sh # Removes stale CPU-KV mmaps from /dev/shm before start ├── SHA256SUMS # SHA-256 pins for every runtime artifact ├── patches/ │ ├── *.py # Byte-for-byte production overlays (mounted read-only) │ ├── diffs/*.patch # Unified diffs vs. the upstream base revision │ └── README.md # Provenance and regeneration instructions └── tuning/ └── *.csv # AITER A8W8 blockscale tuning tables for gfx942 Runtime configuration The stack uses a digest-pinned official vLLM ROCm nightly with:

1. Host prerequisites One MI300X ( gfx942 , 304 CUs, ~192 GiB HBM), a working AMD kernel driver, recent Docker Compose, ~235 GiB RAM for the CPU KV tier, and ~500 GB disk (the model cache alone is ~156 GB).

VLLM_IMAGE= ' vllm/vllm-openai-rocm@sha256:e68d18b2ba50298661bfc49baf01158fbf036645c2362cccf3e8a7a79fe6c69a ' MODEL= ' deepseek-ai/DeepSeek-V4-Flash-0731 ' REVISION= ' 7872f01b1d1fe23eabc4c98b48bffcef5a386062 ' docker pull " $VLLM_IMAGE " docker run --rm --entrypoint hf \ -v /root/.cache/huggingface:/root/.cache/huggingface \ " $VLLM_IMAGE " download " $MODEL " --revision " $REVISION " 3. Prepare the files cp Caddyfile.example Caddyfile # then set your hostname, email, and remote_ip CIDR mkdir -p aiter-cache crash-dumps chmod +x vllm-entrypoint.sh sha256sum -c SHA256SUMS # verify the overlays before first start 4. Start docker compose config -q docker compose up -d docker compose logs -f inference A healthy start takes ~5 minutes and must show all of:

Model loading took 156.67 GiB DSpark draft model loaded: 96 params GPU KV cache size: 1,927,444 tokens Maximum concurrency for 262,144 tokens per request: 7.35x Created mmap file /dev/shm/vllm_offload_...mmap (103.08 GB) Capturing CUDA graphs (FULL) Application startup complete After graph capture, run rocm-smi --showmeminfo vram . The warmed high-water mark is ~204.5 GB of 205.8 GB. If only a few hundred MB remain, the server may start but fail on the first request.

HOST= ' your-host.example.com ' curl -fsS " https:// $HOST /v1/models " curl -sS " https:// $HOST /v1/completions " \ -H ' Content-Type: application/json ' \ -d " { \" model \" : \" deepseek-ai/DeepSeek-V4-Flash-0731 \" , \" prompt \" : \" Calculate 17 * 23. Answer with the number only. \" , \" temperature \" : 0, \" max_tokens \" : 32} " The patches Each patches/*.py file is a full-file overlay mounted read-only over its counterpart in the container; compose.yaml contains the target paths. The corresponding diffs/*.patch records the change from its upstream base. The base image remains digest-pinned, so upgrades require changing the image reference and revalidating the stack.

MXFP4 routing. The MoE bitmatrix kernel pads its block columns to a Triton block size, but the padding lanes were masked against the global tensor bound instead of the logical block size. Under load, padded lanes corrupted the routing matrix, causing near-match tool names and forgotten schemas on long prompts. The one-line fix is mask = (offs_local < BLOCK_SIZE) & (offs_global < nonzero_indx_size) , taken from Doubleword commit c32932bb9 . The overlay also includes fused-SiLU and fast-routing changes for grouped MXFP4 experts.

FP8 format. DeepSeek V4's Lightning Indexer cache uses FP8. The stock writer emits OCP E4M3 bytes in row-major order, while AITER on MI300X consumes AMD FNUZ E4M3 bytes in a preshuffled 16×16 tile layout. In the worst case, interpreting one format as the other produces a factor-of-two scale error. The overlay selects float8e4b8 with FP8_MAX=224.0 and shuffled write offsets on ROCm, while leaving the OCP path unchanged elsewhere.

This stack uses probabilistic drafting with block rejection. The two Gumbel overlays keep draft-proposal noise independent of rejection and recovery noise.

Key optimizations in the production configuration:

Distinct ~400-word prompts, streaming, temperature=1.0, top_p=0.95 ; C1–C8 at 512 output tokens, C64 at 256:

DSpark acceptance is prompt-dependent; treat these as gates for this exact image, not universal model benchmarks.

With the tuned kernels, uncached prefill reaches 7.9–8.5K tok/s , depending on scheduler budget: 7.90–7.99K at C1 with an 8,192-token budget and 8.46–8.51K at C4. The production profile uses a 2,048-token budget for latency isolation, giving 6,988–7,019 tok/s on fresh prompts. With the 1,024-token long-prefill cap, an 8.9K-token prompt reaches 5.20–5.29K tok/s at C1. In exchange, TTFT for a short request queued behind a 52K cold prefill drops from 8.2 s to 0.5 s. Warm recall of 380K cached tokens takes 0.64–2.65 s after a 120–125 s cold prefill.

HBM headroom is limited. The warmed high-water mark is 204.5 of 205.8 GB. A 30 GB KV pool loads but fails during graph capture with HSA_STATUS_ERROR_OUT_OF_RESOURCES . Do not raise --kv-cache-memory-bytes ; monitor HBM usage for growth. The CPU KV tier stores cache entries, not weights. --kv-offloading-size 96 --kv-offloading-backend native maps ~103 GB in /dev/shm for evicted prefix-cache entries. The entrypoint removes stale mappings after crashes. The 1,664-token scheduler warning is expected. DSpark-7 reserves draft slots from the 2,048-token budget. Raising the budget reserves more in-flight sliding-window state and reduces usable KV capacity. Warm the kernels after restart. The first prefill initializes kernels and takes 5.3 s for 8.9K tokens; subsequent runs take 1.7 s. Run one uncached prefill before admitting traffic. Test correctness as well as throughput. The validation suite includes two-turn tool-calling fixtures, a BFCL subset (74–76/90 exact calls), OpenCode tool-schema checks, and 380K-token needle recall on both native and DSpark paths. Cold and cached prefills can take different floating-point paths, so test both. License and provenance The stack, documentation, and vLLM-derived overlays are Apache-2.0 (see LICENSE ); the AITER-derived overlay keeps its MIT header. Upstream base revisions for every diff are recorded in patches/README.md . The model itself is MIT-licensed .