PAN 2 FP8 for 48 GB of GPU memory (community quantization)

This is an FP8 quantization of IFM/PAN2, the 29B world model from the Institute of Foundation Models (IFM) at MBZUAI, with a recipe for serving it with vLLM-Omni on 48 GB of GPU memory: 2Γ— 24 GB GPUs (for example RTX 4090) or one 48 GB GPU.

It is an independent community conversion. It is not made or endorsed by IFM or MBZUAI. Read the original model card for the model itself, its intended uses, out-of-scope uses, limitations and safety guidance. All of it applies here.

What's in this repository

Path Contents
transformer/ The PAN 2 diffusion transformer. The 528 joint-block attention and FFN linears are FP8 E4M3 with one FP32 scale per output channel; activations are quantized per token at run time (ModelOpt FP8_PER_CHANNEL_PER_TOKEN, auto-detected by vLLM-Omni). The adaLN modulation (FP32), token refiner, embedders and output head are unchanged.
text_encoder/, vae/, tokenizer/, scheduler/, safety_checker/ Unchanged copies from IFM/PAN2: the Qwen3.5-9B text encoder and the Wan 2.2 VAE.
recipe/pan2_serve.sh Self-contained setup, serve and client script (below).
recipe/examples/ The structured T2V and I2V example prompts and the I2V first frame, from the PAN 2 repository.
tools/convert_pan2_fp8.py The CPU-only converter that produced transformer/.
results/ Conversion report and fidelity measurements.

The transformer is 43.8 GB, down from 67.4 GB. 17.7 GB of what remains is the FP32 modulation, which the recipe keeps in host RAM, so about 13 GB of transformer weights sit on each of two GPUs.

Quick start

hf download Eisenhawer/PAN2-FP8 --include 'recipe/*' --local-dir PAN2-FP8 && cd PAN2-FP8/recipe   # 2 MB
./pan2_serve.sh setup                         # new Python env: vLLM 0.33.0 + vLLM-Omni with the PAN 2 patches
./pan2_serve.sh serve                         # serves this checkpoint; auto-detects 2x24 GB / 1x48 GB / 2x48 GB
# in another shell:
./pan2_serve.sh generate --prompt-file examples/prompt.json --negative-file examples/negative_prompt.json \
    --frames 121 --size 1248x704 --out swim.mp4
./pan2_serve.sh generate --prompt-file examples/i2v_prompt.json --negative-file examples/i2v_negative_prompt.json \
    --image examples/i2v_first_frame.png --frames 121 --out house.mp4

./pan2_serve.sh print shows the selected profile and the exact vllm serve command without starting anything.

setup builds its own environment (~/.venvs/pan2-omni, override with PAN2_VENV) and leaves any existing vLLM-Omni alone. To patch your own vLLM-Omni instead, or to have setup install from your own clone (PAN2_VLLM_OMNI_SRC=/path/to/vllm-omni), see Patching your own vLLM-Omni.

Ready to run. By default the script serves this repository as it is: a 59 GB download that loads in about 85 s. There is nothing to convert.

Optional: ConvRot INT8. --format convrot instead serves IFM's original IFM/PAN2 BF16 weights and quantizes them to rotated INT8 (ConvRot) while loading, which happens on its own. It is closer to BF16 (see the table below), but it downloads the 86 GB BF16 repository and loads in about 130 s.

Why FP8 is the default, though ConvRot is more faithful. vLLM-Omni can only apply ConvRot while loading: it cannot save a ConvRot-quantized model or load one, so there is no ready-to-run ConvRot checkpoint to publish. Using ConvRot means downloading IFM's full BF16 weights and quantizing them at every server start. Both formats predict the clean video about as well as BF16 and give clean videos (see Fidelity). If the larger download and slower start do not matter to you, --format convrot is the more accurate choice.

Prompts. PAN 2 is trained on structured JSON world-state captions. Plain one-line prompts give poor video. Write captions with PAN 2's prompt rewriter (ifm-ai/PAN2, inference/pan-prompt-rewriter) and follow the format of recipe/examples/.

How it fits in 2Γ— 24 GB

The recipe uses the vLLM-Omni branch Eisenh/vllm-omni@pan2-host-modulation, which adds the following to upstream vLLM-Omni:

  1. pan2_host_modulation: PAN 2's 44 adaLN modulation linears (4.43B FP32 parameters, 17.7 GB) are replicated on every GPU, even under tensor parallelism. Their input depends only on the timestep, so the branch computes each step's modulation table (4.3 MB) on the CPU from the memory-mapped checkpoint and keeps the weights off the GPUs. Both GPUs' copies are replaced by one host copy that the two GPU workers share, and the table is computed once per denoising step (all 50 in one pass when the request starts), not streamed layer by layer. This is exact: latents are bit-identical to the original path in eager mode. Without it, the 2Γ— 24 GB configuration runs out of memory at startup. It turns on automatically on GPUs this small ("auto", the default).
  2. guardrail_device: cpu: the safety guardrails run on the CPU, because they do not fit next to the transformer on 24 GB cards.
  3. Two fixes. Pre-quantized checkpoints of any format, including this ModelOpt FP8 one, now stay on the CPU under CPU offload (before, they were loaded onto the GPU in full and ran out of memory at startup). The attention output split is made contiguous for quantized kernels at batch size above 1.

The script also enables model-level CPU offload (the text encoder and the transformer take turns on the GPUs), tiled and patch-parallel VAE decoding, and longer request timeouts. Out of the box, any clip that takes more than 10 minutes fails. The vLLM-Omni recipe page is recipes/IFM/PAN2-RTX-4090.md.

How this differs from upstream's 48 GB and larger recipes. Upstream vLLM-Omni already runs the original BF16 PAN 2 on one 48 GB GPU (PAN2-L40S), including 20 s clips, by streaming most transformer blocks from pinned host memory every step; that profile peaks at 145 GiB of system RAM. This recipe instead keeps the whole 8-bit transformer split across two 24 GB GPUs with no per-step layer streaming, and recommends 64 GB of host RAM. It is limited to 10 s clips.

Measured on 2Γ— RTX 4090 (24 GB, PCIe), TP = 2

704Γ—1248, 24 fps, 50 steps, CFG 2.5, eager mode, with GPU peer-to-peer (P2P) enabled (see below):

Clip Format Total time Denoising Peak memory per GPU
5 s (121 frames) FP8 9.5 min 8.4 min 19.4 GiB
5 s (121 frames) ConvRot INT8 9.5 min 7.9 min 19.4 GiB
10 s (241 frames) FP8 28.9 min 21.0 min 22.9 GiB
10 s (241 frames) ConvRot INT8 34.4 min 20.1 min 23.0 GiB

At 480Γ—832, 33 frames and 8 steps, ConvRot is about 30% faster than FP8 (35 s against 51 s), because INT8 tensor throughput is twice FP8's on GeForce Ada. At 720p, attention dominates and the two formats run at about the same speed. 20 s clips (481 frames) at 720p do not fit in 2Γ— 24 GB.

GPU peer-to-peer (P2P) matters. With tensor parallelism the two GPUs exchange activations twice per block, about 4,400 times per request. NVIDIA's standard driver does not enable P2P on GeForce cards such as the RTX 4090, so NCCL copies the data through host memory instead. These measurements used NVIDIA's open kernel modules 610.43.02 with the community P2P patch from aikitoria/open-gpu-kernel-modules (a community patch, not an NVIDIA release; its current branch targets driver 615.78.08). Follow that project's README: it needs Resizable BAR, Above 4G Decoding and the IOMMU in passthrough mode, which removes DMA isolation. Timing NCCL all-reduce at PAN 2's activation sizes on the same machine:

Data per all-reduce (BF16) P2P No P2P Communication per request, P2P / no P2P
480Γ—832, 41 frames: 70 MB 3.1 ms 19.3 ms 13 s / 85 s
704Γ—1248, 121 frames: 436 MB 17.9 ms 121 ms 79 s / 533 s
704Γ—1248, 241 frames: 858 MB 35.2 ms 237 ms 155 s / 1,044 s

The per-request column multiplies by 4,400 calls. It is an estimate: full clips were not timed without P2P. With the standard driver, expect a 5 s 720p clip to take roughly 8 minutes longer and a 10 s clip roughly 15 minutes longer. Check P2P with nvidia-smi topo -p2p r (OK between the GPUs); the script warns when it is off. RTX 3090s can get P2P from an NVLink bridge with the standard driver.

Other GPUs. These profiles are untested:

  • 2Γ— RTX 3090 (24 GB, Ampere): the same memory as 2Γ— 4090, so the same clips are expected to fit. Ampere has no FP8 tensor cores, so vLLM runs this checkpoint as FP8 weights with 16-bit compute (FP8 Marlin), which is expected to be about twice as slow as on 4090s. --format convrot uses Ampere's INT8 tensor cores and may be faster there.
  • 1Γ— RTX 5090 (32 GB): not supported by the script yet. About 26 GB of FP8 weights leaves too little room for 720p activations on one card, so it would need partial layer offload.
  • 1Γ— 48 GB (L40S, RTX 6000 Ada, A6000): about 26 GB of weights on one GPU. 10 s at 720p is expected to fit. For a tested route with the original BF16 weights, use upstream's PAN2-L40S recipe (above), which needs much more host RAM.
  • 2Γ— 48 GB: 20 s clips are expected to fit.
  • A single 24 GB GPU is not a target of this recipe. The script keeps an experimental --profile 1x24 fallback (layerwise offload, FP8 only); it ran at 480Γ—832 only and is very slow.

Host RAM: 64 GB or more is recommended (the text encoder, the guardrails and the modulation page cache). An unquantized BF16 reference does not fit in 125 GB with vLLM-Omni's ordinary loader.

Fidelity

Comparing pixels against a reference does not measure quantization quality for this model. The same FP8 weights run at TP = 2 and on one GPU give videos only 16–20 dB apart (PSNR), because the sampler is chaotic: tiny rounding differences lead to a different but equally valid video.

Fidelity was therefore measured one transformer step at a time, with every variant receiving identical real inputs:

  • Prompt embeddings from the real text encoder.
  • A VAE-encoded clean clip x0, noised at six points of the real 50-step schedule.
  • Conditional and unconditional branches, and the CFG-guided result.

The BF16 reference streams one block at a time. The table compares each variant with BF16:

Metric FP8 W8A8 (this repo) ConvRot INT8 (online) FP8 weights only (W8A16, reference point)
Per-block output SNR, median / worst 33.3 / 25.1 dB 46.6 / 37.6 dB 41.1 / 31.9 dB
Guided velocity SNR, by noise level Οƒ = 1.0 β†’ 0.16 28 β†’ 13 dB 35 β†’ 20 dB 33 β†’ 20 dB
x0-prediction error against the true clean latent at Οƒ = 0.16 (BF16: 0.106) 0.111 0.106 0.106
x0-prediction error, Οƒ = 0.93 / 0.69 (BF16: 0.616 / 0.314) 0.606 / 0.317 0.617 / 0.314 0.614 / 0.314

How to read this:

  • Both 8-bit formats predict the clean video about as well as BF16. The deviation from BF16 is small next to the model's own error against the true latent.
  • ConvRot is 4–9 dB closer to BF16 than FP8 at every noise level. FP8's error is dominated by activation outliers in a few blocks (0, 12, 26, 38 and 39), which ConvRot's Hadamard rotation spreads out.
  • Videos from both formats look clean.

Per-tensor weight error of this checkpoint is 2.64% (median) to 2.66% (max) relative RMS. That is what E4M3's 3 mantissa bits give: rounding error is about 2.6% RMS at any magnitude. The full data is in results/.

Reproducing the checkpoint (optional)

You do not need this to run the model; the checkpoint here is ready to serve.

hf download IFM/PAN2 --exclude "assets/*" --local-dir PAN2
python -I tools/convert_pan2_fp8.py --src PAN2 --dst PAN2-FP8    # CPU only, ~7 min, streams one shard at a time

To reproduce the fidelity table, run tools/fidelity_single_step.py on one 24 GB GPU (about 5 min) in the environment from recipe/pan2_serve.sh setup. It needs a local copy of IFM/PAN2 and a clean 480Γ—832 clip of 33 or more frames, saved as a uint8 .npy array of shape [F, H, W, 3]:

python tools/fidelity_single_step.py --bf16 PAN2 --clip clip.npy   # writes results/fidelity_single_step.local.json

License and attribution

  • License. Apache-2.0, the same as IFM/PAN2. LICENSE, NOTICE and THIRD_PARTY_LICENSES are carried over unchanged.
  • Bundled components. The text encoder (Qwen3.5-9B) and the VAE (Wan 2.2) are under their own licenses, listed in THIRD_PARTY_LICENSES.
  • Changes from the original. The transformer's joint-block linear weights were converted to FP8 with per-channel scales, transformer/config.json gained a quantization_config, and modular_model_index.json points at this repository. Everything else is byte-identical.

The original authors strongly recommend running PAN 2 behind its guardrails; the recipe turns them on by default (on the CPU). Users remain responsible for the content they generate. See the original card's Safety Guardrails and Out-of-Scope Uses.

Downloads last month
44
Safetensors
Model size
29B params
Tensor type
F32
Β·
BF16
Β·
F8_E4M3
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Eisenhawer/PAN2-FP8

Base model

IFM/PAN2
Quantized
(1)
this model

Paper for Eisenhawer/PAN2-FP8