Huihui-Qwen3.8-27B-abliterated-NVFP4

NVFP4 (W4A4, group 16) quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated — all credit for the model goes to @huihui-ai. This repo carries only the quantized weights.

55.6 GB → 20.0 GB. Fits on two 16 GB cards with real KV headroom.

🔄 Re-quantized 2026-08-24 from the upstream Latest update 3

Upstream narrowed the ablation to layers 18–51 only (previously everything from layer 15 up), leaving the rest of the network untouched, to retain more of the original model's performance. MTP and the vision tower are unmodified.

I diffed the shards before re-baking: 6 of 18 changed, and they are exactly the shards holding layers 15–17 and 52–63. The changelog matches the bits.

This is not a cosmetic update. The reasoning_effort: xhigh thinking-runaway that the previous revision failed is gone — see the gotcha section. If you were pinning medium because of it, you no longer have to.

The previous NVFP4 revision remains available at commit history / revision= if you need it.

• MTP draft head preserved in bf16 and wired into the index — speculative decoding works out of the box.
• bf16 kept for: lm_head, the vision tower, the DeltaNet conv1d, and the MTP head. Everything else is NVFP4 W4A4.
• Built for SM120 / Blackwell with vanilla vLLM v0.22.0 — compressed-tensors is auto-detected, no --quantization flag.
• Quantized with the bf16 held in CPU RAM while llm-compressor's sequential pipeline onloaded one subgraph at a time to a single GPU: 2 min 10 s end to end, ~10.5 GB of VRAM. (A 4-GPU fully-resident attempt OOMs — 55.6 GB into 65 GB leaves no room for activations. The CPU-held path is both faster and leaves the rest of the boards free.)

Measured — this revision, 2026-08-24

Throughput (TP=4 on RTX PRO 2000 Blackwell ×4, 32k ctx, KV fp8, MTP n=3, 512 tok fixed, reasoning_effort: low):

concurrency	aggregate t/s	MTP acceptance
1	66.6	44.6 %
4	191.3	46.7 %
8	367.9	47.3 %

Single-stream prefill: 3,250 tok/s on a 14.5k prompt (prefix cache cold, mean of 3). GPU KV cache at this config: 556,063 tokens (16.97× concurrency at 32k/request).

Against the previous revision, same hardware, same day, same conditions (TP=2, 32k, MTP n=3): 41.0 t/s vs 41.1 t/s, MTP acceptance 47.1 % vs 49.8 %. The narrower ablation costs nothing in speed.

Capability: an 8-probe set (arithmetic, multi-step probability, code generation, strict-format instruction following, Japanese prose, translation, an uncensored-behaviour check, and an epistemic-humility check) — both revisions pass every mechanically-checkable face, and the prose faces read equally well. On this probe set the "retains more of the original model's performance" claim does not show up as a measurable gain. It does not show up as a loss either. Where it does show up is the xhigh gate below.

Serve

vllm serve sakamakismile/Huihui-Qwen3.8-27B-abliterated-NVFP4 \
  --trust-remote-code --tensor-parallel-size 4 \
  --max-model-len 32768 --kv-cache-dtype fp8 --reasoning-parser qwen3 \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

On boards without P2P add NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce.

⚠️ Gotchas

• The xhigh thinking-runaway is fixed in this revision. On the previous revision, reasoning_effort at xhigh would sometimes never terminate the <think> phase: it grew past 19,000 characters, degenerated into repeating a single line, and the token budget was gone before any answer was emitted — 1 failure in a 9-case gate (French/English long-form, temperature 0 and 0.7). The same gate on this revision passes 9 of 9, with thinking between 1.5k and 8k characters and every run finishing on stop. medium remains a sensible default for long-form work, but it is no longer a workaround for a defect.

  Worth knowing what these modes actually are: they are not three levels of capability, they are three system prompts. Reading the chat template — xhigh injects "think carefully through the task, validate key assumptions, consider plausible alternatives…", low injects "keep your thinking brief and focused, moving directly to the conclusion", and medium injects nothing at all — it is simply the model with no deliberation instruction.

  "chat_template_kwargs": {"reasoning_effort": "medium"}
  

• The 15 mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.

• W4A16 (NVFP4A16) does not serve on this architecture in vLLM 0.22 (gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.

• Long single-file code generation drops a closing paren roughly 1–2 times in 14 regardless of sampling temperature (measured with a JS parser on the base model). Put a syntax check in the loop rather than tuning temperature.

• Text-only serving shown above; the vision tower ships in bf16 but multimodal serving was not benchmarked here.

Recipe

llm-compressor NVFP4 (W4A4, group 16), targets: [Linear], ignore: [lm_head, re:.*visual.*, re:.*conv1d.*, re:.*mtp.*], 32 calibration samples × 8192 tok from neuralmagic/calibration. MTP tensors grafted back in bf16 after saving and appended to quantization_config.ignore.

🙏 @huihui-ai for the model, Qwen team for the base, vLLM & llm-compressor teams for the tooling.