Patrick Hill PRO

Thank you—this is a valid concern about the historical Qwen2.5-14B and Qwen2.5-32B runs.

Since those runs were published, quant_eval’s Modal evaluation path has been substantially hardened. Current remote runs now require:

• A caller-supplied seed that is returned in runtime provenance
• Pinned llama-cpp-python
• Identical accelerator classes across paired lanes
• Matching context limits
• Identical effective decoding settings across lanes
• Recorded artifact, backend and runtime identities
• Repeat-run stability evidence for non-greedy decoding
• Holm-adjusted significance across the eight reported families

Those controls address the principal weaknesses you identified for future runs. They do not retroactively quantify the noise in the original 14B and 32B measurements, however.

I agree that an F16-versus-F16 control is the appropriate next experiment. I am extending quant_eval with an explicit same-artifact A/A control that verifies identical weight hashes, fixtures, scoring contracts and decoding conditions while measuring per-case movement, raw-output identity, discordance and family-level significance.

I intend to run repeated A/A controls rather than interpret one run as a complete noise floor. A single A/A pair is one realization of the null condition; repeated pairs are needed to distinguish ordinary stochastic movement from a stable directional backend or accelerator effect. The A/A results will not be subtracted mechanically from the quantization results.

The experiment is now underway. I will report the findings when it is complete, including what they imply for the historical Qwen2.5-14B and Qwen2.5-32B conclusions.

Dipankar — thank you again. Your criticism was substantiated. The earlier stateful_followup construction placed too much evidentiary weight on one boundary prompt and could conceal opposing operation-level effects behind a single family aggregate. I did not treat that as merely a reporting problem; I used it to redesign and revalidate the family before returning it to production.

Updated stateful evaluation design

The updated production evaluation now uses a locked, disjoint 200-case stateful split crossing:

• Four primary operations:
  • reverse_primary_tags
  • rotate_queue_left
  • expire_decoy_record
  • raise_primary_priority
• Four construction bands:
  • foundation
  • operational
  • stress
  • boundary

All 16 cells contain 12 or 13 unique paired cases. The 200 evaluation cases are semantically unique and have zero content overlap with the calibration split. The bands are explicitly described as construction strata, not as an empirically ordered difficulty scale.

Before promotion, I ran the proposed boundary ablation, decode-stability experiments, state-transition isolation experiments, and crossed-construction calibration on Mistral-Nemo and Qwen2.5-7B against Q4_K_M, Q5_K_M, and Q8_0. The revised design then received six full 200-case-per-family publication runs.

Decoding and the reported Qwen wobble

The current runs are not greedy.

For Qwen2.5-7B, both F16 and quantized lanes now use the publisher-declared decoding configuration:

• temperature=0.7
• top_p=0.8
• top_k=20
• repetition_penalty=1.05
• Fixed seed 42 for stateful follow-up

The old deposit and these revised runs should not be pooled: the evaluation contract, stateful construction, and Qwen decoding contract changed.

There is strong repeatability evidence within the new contract. Qwen’s F16 baseline was independently executed in the Q4, Q5, and Q8 publication comparisons. Across all three runs:

• Case-level pass/fail movement: 0/1,600
• Raw-output movement: 0/1,600

Mistral-Nemo’s F16 baseline also reproduced exactly across its three precision comparisons:

• Case-level pass/fail movement: 0/1,600
• Raw-output movement: 0/1,600

That directly addresses the earlier concern about interpreting a small Qwen movement from one non-greedy run.

Mistral-Nemo F16 versus Q4_K_M

The redesigned stateful result is:

• F16: 127/200
• Q4_K_M: 22/200
• F16-only passes: 105
• Q4-only passes: 0
• Difference: −0.525
• Exact two-sided McNemar: p ≈ 4.93 × 10⁻³²

Most importantly, the 105 discordant cases do not share the original boundary-only signature. They are distributed across every construction band:

• Boundary: 28
• Foundation: 21
• Operational: 30
• Stress: 26

The largest band contains only 28.6% of the discordance.

The effect also separates by operation:

• expire_decoy_record: −0.70
• raise_primary_priority: −0.86
• rotate_queue_left: −0.54
• reverse_primary_tags: 0.00

The last result is itself important: both Mistral lanes scored 0/50 on reverse_primary_tags. The revised harness therefore exposes a baseline model weakness instead of automatically attributing every failure to quantization.

The Q4 loss is genuine under this contract, but it is not the former single boundary-instruction artifact.

Mistral Q5_K_M and Q8_0 did not reproduce the Q4 cliff:

• Q5 stateful: F16 127/200, Q5 129/200, difference +0.010
• Q8 stateful: F16 127/200, Q8 126/200, difference −0.005

Both are classified as preserved within margin.

Qwen2.5-7B F16 versus Q4_K_M

The redesigned family-level result is:

• F16: 152/200
• Q4_K_M: 144/200
• F16-only passes: 27
• Q4-only passes: 19
• Difference: −0.040
• Exact two-sided McNemar: p ≈ 0.302

The appropriate conclusion is inconclusive—not a Q4 win or a demonstrated Q4 loss.

The operation-level decomposition explains why the aggregate must not be interpreted alone:

• reverse_primary_tags: −0.42
• expire_decoy_record: +0.14
• rotate_queue_left: +0.12
• raise_primary_priority: 0.00

This is precisely the pattern the earlier evaluation could hide: the same quantized model degrades materially on one transition while improving on others.

By construction band:

• Boundary: −0.021
• Foundation: −0.173
• Operational: +0.038
• Stress: 0.000

Qwen’s other precisions were stable at the family level:

• Q5_K_M: F16 152/200, Q5 160/200, difference +0.040, preserved within margin
• Q8_0: F16 152/200, Q8 155/200, difference +0.015, preserved within margin

What changed in quant_eval

Since your comment, the following has been added or corrected:

1. Publisher-declared Qwen decoding parameters are enforced for both local and Modal runners.
2. Decoding controls and their effective backend values are recorded in the manifest.
3. Seeded non-greedy repeatability is distinguished from guaranteed deterministic decoding.
4. Decode-stability experiments were added before production promotion.
5. Self-fed versus oracle-fed state-transition isolation was added.
6. Initialization correctness, conditional-transition correctness, and end-to-end correctness are reported separately.
7. The narrow stateful construction was replaced with the crossed four-operation × four-band design.
8. Calibration and locked evaluation splits are disjoint.
9. Every stateful publication row records operation_id, construction_band, and transition_load.
10. Those fields are defined in the versioned public data dictionary.
11. Reports disclose results by operation and construction band.
12. Failure signatures include failed gates and normalized JSON-difference paths.
13. Family-level effect concentration is reported so a narrow sub-capability cannot silently become a broad family claim.
14. Exact paired McNemar inference remains the headline test; individual crossed cells are descriptive strata.
15. Holm adjustment is applied across the reported family-level tests.
16. Machine-readable claim lineage maps conclusions to rollup fields and statistical rules.
17. Publication bundles are independently verified for checksums, schemas, telemetry joins, token reconciliation, statistics, and claim-lineage pointers.
18. The revised production contract was validated with six full publication runs covering both models and Q4_K_M, Q5_K_M, and Q8_0.

Your observation therefore changed both the fixture design and the evidentiary limits of the resulting claims. The current harness no longer treats the original boundary behavior as a sufficient characterization of stateful-follow-up capability.

Publication runs and supporting evidence

The six publication run IDs are:

• Mistral_Nemo_Instruct_2407_20261018_235327
• Mistral_Nemo_Instruct_2407_20261019_040329
• Mistral_Nemo_Instruct_2407_20261019_122449
• Qwen2.5_7B_Instruct_20261019_153512
• Qwen2.5_7B_Instruct_20261019_181307
• Qwen2.5_7B_Instruct_20261019_201739

If you would like the complete publication bundle for any or all of these runs—including the 3,200 per-case rows, locked fixtures, telemetry, manifests, provenance, rollups, failure-signature analysis, data dictionary, claim lineage, and checksums—please identify the run IDs in a request through pbhappliedsystems.com/contact.

I would be interested to know whether the revised design leaves any part of your original concern unresolved, or whether anything else in the approach warrants closer examination.

And thank you: this was exactly the kind of external scrutiny the project needed.

Thank you—your criticism was correct, and we tested it directly.
We ran a parallel ablation across Mistral-Nemo and Qwen2.5-7B, comparing each F16 baseline against Q4_K_M, Q5_K_M, and Q8_0. Each of 48 identical boundary scenarios was tested with the original Do not add keys wording, with that sentence removed, and with an explicit retention contract.
The original wording reproduced the asymmetry you identified. Mistral-Nemo Q4 passed 14/48 control cases while F16 passed 48/48. Qwen2.5-7B moved in the opposite direction: F16 passed 15/48 and Q4 passed 32/48. But every tested model and precision passed 48/48 when the phrase was removed, and 48/48 under explicit existing-key retention.
So the old result was not a general Q4 cliff. It was an instruction interaction that affected models and precisions differently.
We did not simply remove the constraint. We replaced that construction with a four-tier stateful family covering foundation, operational, stress, and boundary transformations. Retention and key admission are now stated separately, with an explicit allowed-key set. Calibration and locked evaluation splits are disjoint.
In the new 200-case production evaluation:

• Mistral-Nemo F16 vs Q4_K_M: 150/200 vs 80/200, with 70 F16-only passes and no Q4-only passes.
• Mistral-Nemo F16 vs Q5_K_M: 150/200 vs 150/200.
• Mistral-Nemo F16 vs Q8_0: 150/200 vs 150/200.
• Qwen2.5-7B F16 vs Q8_0: 145/200 vs 147/200, with 9 F16-only and 11 Q8-only passes; statistically inconclusive.
  One limitation remains visible: Mistral scored 0/50 in the new boundary tier in both lanes, so that tier is currently non-discriminating for this model. The Mistral Q4 degradation now comes from the other constructions rather than the original one-instruction boundary artifact.

Your comment materially improved the evaluation. Thank you for reproducing the deposited data and identifying the confound.

Your observation directly motivated this ablation and the subsequent revision of the stateful_followup evaluation. If I publish the updated methodology or results, may I acknowledge you by name and reference your Hugging Face comment? If so, please let me know your preferred name, affiliation, and profile or citation link.
If you would like to receive the results and available publication/ bundles, please contact me through pbhappliedsystems.com/contact and reference these run names:

• Mistral_Nemo_Instruct_2407_20261009_033750
• Mistral_Nemo_Instruct_2407_20261009_071015
• Mistral_Nemo_Instruct_2407_20261009_091019
• Qwen2.5_7B_Instruct_20261009_191201
• Qwen2.5_7B_Instruct_20261009_215528
• Qwen2.5_7B_Instruct_20261009_221516