Qwen3.5-9B-NVFP4-QAD-W4A4-LR1e-5-s4000
NVFP4 QAD (W4A4-trained) checkpoint of Qwen/Qwen3.5-9B @ c202236235762e1c871ad0ccb60c8ee5ba337b9a, optimizer step 4000 of 4000.
NVFP4 W4A4-trained arm (stage-2): the full static-input-scale lifecycle (bootstrap -> running amax -> per-optimizer-step commit -> nvfp4_runtime_state.pt) exercised in a real 4000-step run, stable at lr 1e-5. Scientific verdict: W4A4 training does not pay for dense NVFP4 at this scale — under A4 serving it landed at KL 0.0517 / GSM8K 84.8 vs the weight-only-trained sibling's 0.0497 / 86.6 (activation gap is only 0.024). It is harmless-to-good at A16 serving (best GSM8K 87.6 and MMLU-Pro 77.3 of the whole 2x2 matrix).
How this checkpoint was produced
| item | value |
|---|---|
| training repo | QATFactory branch weili/w4a4 @ a9315e4 (PR) — scripts/train_llm_qat.py |
| method | QAD (quantization-aware distillation): student trains with fake-quantized forward; frozen BF16 teacher = the base model itself |
| objective | pure KL at temperature 1.0 (distill_weight 1.0, hard_label_weight 0.0), loss on assistant tokens only |
| base model / teacher | Qwen/Qwen3.5-9B @ c202236235762e1c871ad0ccb60c8ee5ba337b9a |
| dataset | openperfectblend_100k_Qwen3.5-9B_think — ~100K ChatML conversations (OpenPerfectBlend-derived prompts with Qwen3.5-9B think-mode responses; native <think> spans in the assistant turns), prepared in the QATFactory project. Train file from togethercomputer/Qwen3.5-9B-reasonmix @ b88c109 (940,793,581 bytes, md5 0406bb3a7a482352360716a1bc5e9e04; ~84.3k train conversations by the trainer's epoch accounting). Held-out eval = a disjoint 256-conversation split (md5 af10c8c304c146a81ddc35e439d7ac4b), 818,944 scored positions — the same corpus used for the serving-KL rows below |
| preprocessing | model chat template (ChatML), assistant-only loss mask, max_seq_len 8192, right-truncated |
| this checkpoint | optimizer step 4000 of 4000 (32,000 conversations consumed ~= 0.38 epoch, no data repetition) |
| batch per weight update | 8 sequences = 1/GPU x 8 GPUs x grad-accum 1 (<= 8 x 8192 = 65,536 tokens/update) |
| learning rate | peak 1.0e-5, cosine decay to 0 over 4000 steps, linear warmup 1% (40 steps) — inherited from the NVFP4 sweep winner (same dose wins for both formats) |
| optimizer | AdamW (adamw_torch, beta1 0.9 / beta2 0.999), weight_decay 0.0, max_grad_norm 1.0 |
| precision / parallelism | bf16, FSDP2 full_shard on jbom 8xB200 (single node), gradient checkpointing (non-reentrant), sdpa attention |
| fake-quantized modules | all linear projections (q/k/v/o_proj, gate/up/down_proj, GatedDeltaNet in/out projections); embeddings, lm_head, norms and the vision tower stay BF16 |
| quantization config | quant_format: nvfp4, fused_runtime_scales: true, quantize_activations: true |
| seed / bookkeeping | seed 42; held-out eval every 100 steps; checkpoint every 1000 steps; step time |
| in-loop held-out eval KL | 0.03541 (s100, W4A4 init) -> 0.0299 (s4000; plateau from ~s1700) |
| W&B | pofn3djc (project qatfactory-qat, public) |
Training mode: W4A4 (weights + input activations fake-quantized). Input activations use the serving static-global-scale scheme: bootstrap on first forward -> running amax -> per-optimizer-step commit (gradient-checkpoint recompute-deterministic) -> persisted in nvfp4_runtime_state.pt and exported as input_global_scale tensors.
Serving
compressed-tensors NVFP4 artifact carrying the full W4A4 schema (FP4 E2M1 weights,
block-16 FP8-E4M3 scales + per-tensor global scales, static input-activation scales
learned during training). Default load in vLLM >= 0.25.1 on SM100+ serves W4A4
(measured 12.3-15.2k tok/s single-GPU greedy vs ~7.7-8.3k for W4A16 — ~1.6x). For clean
weight-only W4A16 serving use the sibling --weight-only export:
Qwen3.5-9B-NVFP4-QAD-W4A4-LR1e-5-s4000-weight-only.
vllm serve weili-0234/Qwen3.5-9B-NVFP4-QAD-W4A4-LR1e-5-s4000 --max-model-len 24576
Evaluation (step 4000)
| model | KL vs BF16 | GSM8K | GPQA-D | MMLU-Pro |
|---|---|---|---|---|
| BF16 teacher | — (reference) | 83.8 | 67.68 | 77.0 |
| this ckpt, A16 (W4A16 serving) | 0.0319 | 87.6 | 71.7 | 77.3 |
| this ckpt, A4 (W4A4 serving) | 0.0517 | 84.8 | 64.7 | 75.9 |
All rows come from one frozen harness: vLLM 0.25.1 on B200 (SM100), temperature 0; KL = top-20 + residual-bucket KL vs the BF16 teacher's next-token distribution over 818,944 held-out positions (nats, lower is better; pipeline validated by a BF16-self run at 0.0002); GSM8K test[:500] 0-shot CoT; GPQA-Diamond n=198 (single seed — treat deltas < ~6 points as noise); MMLU-Pro 1000-question subset.
Related checkpoints
- Other steps of this arm: s1000, s2000, s3000
- QAD-trained BF16 master weights (pre-export): Qwen3.5-9B-NVFP4-QAD-W4A4-LR1e-5-s4000-BF16
- Weight-only (clean W4A16) export of s4000: Qwen3.5-9B-NVFP4-QAD-W4A4-LR1e-5-s4000-weight-only
- W4A16-trained contrast: NVFP4-QAD-LR1e-5
Part of a monitored QAD experiment series with full bookkeeping (pre-registered predictions, exact SHAs/configs/seeds per run). Produced with AI assistance (Claude).
- Downloads last month
- 5