Qwen3.5-9B-MXFP4-QAD-W4A4-LR1e-5-s3000

MXFP4 QAD (W4A4-trained) checkpoint of Qwen/Qwen3.5-9B @ c202236235762e1c871ad0ccb60c8ee5ba337b9a, optimizer step 3000 of 4000.

MXFP4 W4A4-trained arm (stage-2): trains against the numerics that are deployed at W4A4 serving. Single-knob contrast vs the W4A16-trained arm: only qad.quantize_activations: true differs. Result: under A4 serving it wins every metric (KL -18%, GSM8K +5.8 to BF16 parity 83.8, GPQA +2.0, MMLU-Pro +5.5 vs the W4A16-trained checkpoint served A4), at a modest A16 cost (-2.2 GSM8K, +0.019 KL). For MXFP4, train in the mode you serve.

How this checkpoint was produced

item	value
training repo	QATFactory branch weili/w4a4 @ a9315e4 (PR) — scripts/train_llm_qat.py
method	QAD (quantization-aware distillation): student trains with fake-quantized forward; frozen BF16 teacher = the base model itself
objective	pure KL at temperature 1.0 (distill_weight 1.0, hard_label_weight 0.0), loss on assistant tokens only
base model / teacher	Qwen/Qwen3.5-9B @ c202236235762e1c871ad0ccb60c8ee5ba337b9a
dataset	openperfectblend_100k_Qwen3.5-9B_think — ~100K ChatML conversations (OpenPerfectBlend-derived prompts with Qwen3.5-9B think-mode responses; native <think> spans in the assistant turns), prepared in the QATFactory project. Train file from togethercomputer/Qwen3.5-9B-reasonmix @ b88c109 (940,793,581 bytes, md5 0406bb3a7a482352360716a1bc5e9e04; ~84.3k train conversations by the trainer's epoch accounting). Held-out eval = a disjoint 256-conversation split (md5 af10c8c304c146a81ddc35e439d7ac4b), 818,944 scored positions — the same corpus used for the serving-KL rows below
preprocessing	model chat template (ChatML), assistant-only loss mask, max_seq_len 8192, right-truncated
this checkpoint	optimizer step 3000 of 4000 (24,000 conversations consumed ~= 0.28 epoch, no data repetition)
batch per weight update	8 sequences = 1/GPU x 8 GPUs x grad-accum 1 (<= 8 x 8192 = 65,536 tokens/update)
learning rate	peak 1.0e-5, cosine decay to 0 over 4000 steps, linear warmup 1% (40 steps) — inherited from the stage-1.5 MXFP4 sweep winner
optimizer	AdamW (adamw_torch, beta1 0.9 / beta2 0.999), weight_decay 0.0, max_grad_norm 1.0
precision / parallelism	bf16, FSDP2 full_shard on jbom 8xB200 (single node), gradient checkpointing (non-reentrant), sdpa attention
fake-quantized modules	all linear projections (q/k/v/o_proj, gate/up/down_proj, GatedDeltaNet in/out projections); embeddings, lm_head, norms and the vision tower stay BF16
quantization config	quant_format: mxfp4, fused_runtime_scales: false, quantize_activations: true
seed / bookkeeping	seed 42; held-out eval every 100 steps; checkpoint every 1000 steps; step time 3.6 s/step (1.6x the W4A16 forward; ~3h57m total)
in-loop held-out eval KL	~0.10 (W4A4 init) -> 0.06274 (s4000; converged from ~s1800)
W&B	wbi8swsf (project qatfactory-qat, public)

Training mode: W4A4 (weights + input activations fake-quantized). Input activations are fake-quantized per-token in block-32 E2M1 with CEIL-rule UE8M0 scales — bit-identical to the serving kernel (flashinfer mxfp4_quantize cute-dsl; GPU fuzz 0/7,248,896 mismatches on B200), and deliberately distinct from the EVEN weight-scale rule. Stateless (no extra tensors in the artifact).

Serving

compressed-tensors format mxfp4-pack-quantized (FP4 E2M1 weights, block-32 E8M0 scales). One artifact serves both modes in vLLM >= 0.25.1:

• W4A4 (default on SM100+/Blackwell): FlashInfer FP4 kernel, activations quantized dynamically per-token to MXFP4 at runtime. First load runs a one-time fp4_gemm autotune (~10-20 min; cached under VLLM_CACHE_ROOT).
• W4A16 (any Marlin-capable GPU, or forced on Blackwell): --linear-backend marlin (CLI) or LLM(..., linear_backend="marlin").

vllm serve weili-0234/Qwen3.5-9B-MXFP4-QAD-W4A4-LR1e-5-s3000 --max-model-len 24576

Evaluation (step 3000)

This mid-run checkpoint was uploaded for archival completeness; no serving-eval rows were measured at step 3000. Measured rows exist at s1000, s2000, s4000. In-loop held-out eval trajectory: ~0.10 (W4A4 init) -> 0.06274 (s4000; converged from ~s1800).

Related checkpoints

• Other steps of this arm: s1000, s2000, s4000
• QAD-trained BF16 master weights (pre-export): Qwen3.5-9B-MXFP4-QAD-W4A4-LR1e-5-s4000-BF16
• W4A16-trained contrast: MXFP4-QAD-LR1e-5

---

Part of a monitored QAD experiment series with full bookkeeping (pre-registered predictions, exact SHAs/configs/seeds per run). Produced with AI assistance (Claude).