cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF

Qwen3.8-27B-OBLITERATED — low-bit imatrix GGUFs (runs on 12 GB VRAM)

Rebuilt 2026-08-23 — re-download if you pulled these before that date

The author of the base model stated publicly on 2026-08-23 that every GGUF he had published until then "weren't being properly converted", and his repo carries the commit V3: fixed merge (abliterated weights, not stock). The files that were in this repo were quantized on 2026-08-22, before both.

Every file here has been rebuilt from the corrected Q8_0, with a fresh importance matrix computed on the corrected weights and the current upstream chat template baked in. Filenames and URLs are unchanged, so -hf and Ollama commands keep working. The previous files remain reachable in this repo's git history at revision ad034226 — their sha256 are listed in the changelog at the bottom.

IQ2/IQ3_XXS quantizations of OBLITERATUS/Qwen3.8-27B-OBLITERATED, sized so 12 GB (and even 10 GB) GPUs run this 27B fully offloaded. The smallest GGUF in the base repo is Q2_K at 10.9 GB, with no headroom for context on a 12 GB card and no IQ-class quants at all.

Why these levels need work that the base repo does not do

1. The public imatrix for this model is missing entries for several tensors, so llama-quantize refuses to produce any IQ2/IQ3-class quant from it. imatrix collection for hybrid SSM+attention models was buggy before llama.cpp #14994; we compute a fresh imatrix (shipped here as imatrix.gguf) on current master with the single-sequence batching workaround.
2. blk.64 is an MTP (multi-token-prediction) layer that never runs during normal decoding, so no imatrix data can exist for it and the IQ quantizers abort. Its 8 weight tensors are pinned to q4_K via --tensor-type. That costs roughly 120 MB of file size for weights a plain llama.cpp decode never executes; it is kept so speculative-decoding front-ends still find the head.

Benchmarks

File	Size	PPL (wikitext-2)	vs corrected Q8_0	Fits a 12 GB card
IQ3_XXS	11.43 GB	8.2767 ± 0.124	+6.6 %	headless, or ctx ≤4k with a desktop
IQ2_M	10.24 GB	8.8054 ± 0.133	+13.4 %	yes, comfortably
IQ2_S	9.60 GB	9.2450 ± 0.140	+19.1 %	yes, with context headroom
IQ2_XXS	8.67 GB	10.2938 ± 0.159	+32.6 %	8-10 GB cards, visibly degraded
Q8_0 (base repo, our source)	29.05 GB	7.7630 ± 0.116	—	no

Pick: IQ3_XXS for best quality on a 12 GB card, IQ2_M if you want context headroom or run a desktop on the same GPU. IQ2_XXS exists so 8-10 GB cards can run the model at all; expect visibly weaker long-form coherence, code and maths.

The rebuild, measured

Perplexity alone cannot tell you the old files came from a different model, so here is the direct test: how often each model picks a different top token than the corrected Q8_0, over the same text.

Comparison	Same top token	Median KLD	RMS Δp
Old Q8_0 vs corrected Q8_0	89.26 ± 0.43 %	0.0328	6.94 %
Our rebuilt IQ2_M vs its own Q8_0	81.86 ± 0.54 %	0.0911	13.28 %

Two Q8_0 quantizations of the same weights agree on the top token about 99.9 % of the time. These two disagree 10.7 % of the time, so they are not the same model. For scale, quantizing all the way from 8-bit to 2-bit costs 18.1 % disagreement: the upstream merge fix is worth 59 % of that.

One counter-intuitive detail worth stating plainly: the old Q8_0 scores a better perplexity (7.3158 ± 0.108 vs 7.7630 ± 0.116). That is consistent with the upstream commit message V3: fixed merge (abliterated weights, not stock) — the old file sat closer to the stock model, which reads better on wikitext and is worse at the one thing this model exists for. The same applies to the IQ2_M we previously shipped (8.2428 ± 0.122).

We are not claiming the old files were broken in use. We did not measure a refusal difference and we are not publishing one. The case for the rebuild is the author's own statement, his commit history, and the 10.7 % top-token divergence above.

How to run

# llama.cpp
llama-server -hf cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M \
  -ngl 999 -c 8192 -fa on -ctk q8_0 -ctv q8_0

# Ollama
ollama run hf.co/cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M

Also works in LM Studio, Jan and koboldcpp. The base model is multimodal; these files are text-only — for vision add mmproj-model-bf16.gguf from the original repo (~1 GB extra VRAM).

The author's recommended sampling for this model: temp 0, repetition_penalty 1.15, no system prompt, thinking off for coding.

Provenance & reproducibility

• Source: Qwen3.8-27B-OBLITERATED-Q8_0.gguf from the base repo, as of 2026-08-23 (the corrected merge). Requantized with --allow-requantize.
• Importance matrix: imatrix.gguf here, computed by us on the corrected weights with bartowski's calibration_datav3 (129 chunks, -c 512 -b 512 -ub 512 --parse-special). The calibration corpus is deliberately not the evaluation corpus.
• llama-quantize from llama.cpp commit c060ca9, with --imatrix and --tensor-type "blk\.64\..*=q4_K".
• Chat template: patched with gguf_new_metadata from the upstream chat_template.jinja of 2026-08-23 17:42, verified byte-identical inside every shipped file (sha256 58b76d2f9875fb3e9359a686567d2e30). eos_token_id=248046, padding_token_id=248044.
• Evaluation: wikitext-2 raw test, 120 chunks, -c 512 -fa on, same binary for every row of the table. Perplexity is reported with its standard error; differences smaller than the error bars are not meaningful.

Changelog

2026-08-23 — all four files rebuilt. Source changed from the base repo's Q8_0 as of 2026-08-22 to the corrected one of 2026-08-23. New importance matrix, upstream chat template patched in. Filenames unchanged.

The previous files are still reachable at revision ad034226 of this repo, so anything you pinned still resolves:

File	sha256 before	sha256 now
IQ3_XXS	3b7ac301cba8bc56…	83cef0f72449b8c0…
IQ2_M	e4978299d3d97900…	2b4552dbb7f6d410…
IQ2_S	b3f540d3786e666d…	9f00561bd9d9a706…
IQ2_XXS	7efc31c3b158ab2d…	3edb134e49153c27…

Credits

• OBLITERATUS — original abliterated model, and for publicly flagging the conversion bug that prompted this rebuild.
• mradermacher — first GGUF/imatrix work on this model.
• llama.cpp — and compilade's hybrid-model imatrix fix (#14994).
• bartowski — the calibration_datav3 corpus.