Emin Temiz PRO

🚀

not what i expected 😃

getting higher in MMLU. real alignment score is finding a stable ground.

i dont do any training regarding MMLU but since evolution optimizes for MMLU, its scores are getting higher.. now i am overfitting to evolution face palm

but alignment quick test is getting better.

not what i expected 😃

getting higher in MMLU. real alignment score is finding a stable ground.

Three things from my own testing that bear on the evals you describe and on the base-3.8 lineage — offered because I think they will save you time.

On the CPU-side degeneracy detector. I would be cautious about expecting a static, inference-free check to catch this class. In my testing the same weights pass and fail depending on input size: Ostrich-27B-260721 i1 is clean at 12k and 24k characters and enters a broken-record loop at 48k, and stock Qwen/Qwen3.6-27B shows the same shape at 36k. The static quant of those identical weights fails at 12k where the imatrix build passes.

it is possible to predict some failure using cpu but it is not perfect.

If degeneracy only appears above a threshold that depends on input length and on quantization, then a math check over the tensors has no input to scale — it may be measuring something real, but not the thing that actually breaks in use. Worth validating the detector against a model you can already make fail at 48k but not at 12k, before trusting a green result.

i have short (majority) and also long context probs (a few times). degeneration can happen in short contexts too.

On </think> closing. Good to see this in stage 1. One suggestion from the same data: run it at more than one input size. Every "fixed" verdict in this bug family that I have checked — including one I sent you myself in July — was measured at ~12k characters, and the failure reappears higher up. A single-scale check will keep returning PASS on a model that still has it.

One more, and this one is about the base-3.8 lineage specifically rather than about anything you have done.

Qwen/Qwen3.8-27B's chat_template.jinja sets reasoning_effort|default('xhigh') — the most expensive of its three settings is what a caller gets by setting nothing. I have measured that default returning an empty answer with finish_reason: "stop" on 18 of 93 calls (19.4%), against 0 of 95 at low or medium, on bf16 through vLLM. The traces end mid-deliberation, and the </think> never closes — which is exactly the check you have in stage 1, so I expect your eval to start firing on 3.8-derived models for a reason that is inherited rather than introduced.

i have seen these empty responses. unsloth released new quants today. maybe they are healed there.

⚠ The part that matters most for anything you publish as GGUF: llama.cpp and LM Studio ignore reasoning_effort entirely. I verified it by watching prompt_tokens, which is identical at none/low/medium/xhigh — the parameter never reaches the template. So a GGUF user cannot turn the default off through the API. They inherit xhigh with no escape short of hand-assembling ChatML over /v1/completions.

my setup is llama-cpp-python so these work for me

llm = Llama(
model_path=model,
n_gpu_layers=-1,
n_ctx=g_context_len,
verbose=False,
chat_template_kwargs={"enable_thinking": False}
)

llm = Llama(
model_path=model,
n_gpu_layers=-1,
n_ctx=g_context_len,
verbose=False,
chat_template_kwargs={"enable_thinking": True, "reasoning_effort": "medium"}
)

If you ship a base-3.8 lineage, it is worth changing that one line in the template you publish — default('medium') — rather than passing the upstream default through. medium is also the only one of the three that injects no instruction at all (the template has branches for xhigh and low and none for medium), so it is the neutral choice rather than an opinionated one.

will think about it

I have filed this with Qwen as QwenLM/Qwen3.8#216 if you want the full measurements. Setting repetition_penalty 1.1 also removed the failure in my testing (8/48 → 0/23), though that only helps callers who set it.

Separately, I have a longer write-up of a different problem in 260815 — verbatim training-data regurgitation under distribution shift, which is a corpus question rather than a decoding one and which a degeneracy detector would not flag. I will put it in the model repo discussions rather than here, since it is about that release and not about the work you are announcing.

Good luck with the base-3.8 lineage. I would be glad to compare notes on what the 3.8 base does and does not fix relative to 3.6.

260721 may work better for you for a while if you need zero faults , good capabilities but it has lesser alignment. everything comes with a cost. when i go higher in alignment i break things. current 3.8 versions are "surgery", quick and dirty. not proper fine tuning. after proper evolutions and fine tunings i may release something more capable with less errors and higher alignment.

maybe you can train a model and i can use it as a benchmark 😄