Wisp-5M-Instruct

An instruction-tuned version of Wisp-5M, a 5.1M-parameter LLaMA-style model pretrained from scratch. It was fully fine-tuned for two epochs on smol-smoltalk with the loss computed only on assistant tokens (end-of-turn tokens weighted 5×, so the model learns to stop).

At 5M parameters this is a tiny research model. It follows the chat format and answers simple requests, but factual knowledge, reasoning and arithmetic are very weak. See the limitations below.

Quickstart

Use the recommended sampling settings (they are stored in generation_config.json). With near-greedy decoding this model often falls into repetition loops.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "DedeProGames/Wisp-5M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()

messages = [{"role": "user", "content": "What is photosynthesis?"}]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False)
out = model.generate(input_ids, max_new_tokens=256)  # temperature 0.7, top-p 0.9, repetition penalty 1.1 from generation_config.json
print(tokenizer.decode(out[0, input_ids.shape[1]:], skip_special_tokens=True))

Keep prompt + answer within the 2,048-token context.

Chat format

The Wisp tokenizer already has <|im_start|> and <|im_end|>, so the model uses ChatML. apply_chat_template renders it:

<|endoftext|><|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
{assistant reply}<|im_end|>

<|im_end|> (id 3) ends a turn, and <|endoftext|> (id 0) is the BOS. The model stops on either one.

Architecture

Same as Wisp-5M: LlamaForCausalLM, 9 layers, hidden size 192, 6 heads with 2 KV heads, SwiGLU (intermediate 512), RMSNorm, RoPE (θ = 10,000), tied embeddings, 8,192-token byte-level BPE vocabulary, 2,048-token context. Weights in BF16 (model.safetensors, 20 MB).

Training

Base	DedeProGames/Wisp-5M
Data	smol-smoltalk train split, conversations of up to 1,024 tokens (with this tokenizer)
Epochs	2 (14,502 steps, batch 32 = 16 × 2 gradient accumulation)
Optimizer	AdamW (β = 0.9, 0.95), weight decay 0.1 on 2D matrices, grad clip 1.0
Schedule	lr 2e-4, 300 warmup steps, cosine to 2e-5
Loss	assistant tokens; `<
Precision	bf16 autocast, fp32 master weights
Hardware	1× NVIDIA RTX 3060 12 GB, 47 minutes

Held-out loss on assistant tokens (smol-smoltalk test split): 1.791 at the end of training.

Evaluation

Benchmarks follow the protocol of the Kiyo-230M-Instruct card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512, and the model's full 2,048-token context. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card (GPQA is sampled at T = 0.6, as on the card). The second setting is the recommended one shipped in generation_config.json.

Benchmark	Metric	Wisp-5M-Instruct, protocol	Wisp-5M-Instruct, recommended sampling
GPQA Diamond	pass@1 (198 × 5 samples, T = 0.6)	9.80	not run
IFEval	prompt-level strict	12.38	11.65
LiveBench instruction following	strict, 200 questions	16.53	not run
CaseReportBench	token F1	not run (not on the Hub)	not run
EOS rate (IFEval)	share of answers that stop	0.82	0.98

For reference, the Kiyo card reports (not re-run here): Supra2-100M-Instruct GPQA 5.56, IFEval 9.06, LiveBench 11.28; SmolLM2-135M-Instruct GPQA 16.57, IFEval 20.89, LiveBench 24.23.

Notes:

• With the protocol setting, about 18% of IFEval answers never stop and loop on a sentence. Under the recommended sampling this drops to about 2%.
• GPQA: among answered questions, accuracy is 19.5%, which is below the 25% chance level. The model cannot solve these questions.
• The GPQA prompt was written for this run, not taken from the Kiyo card, so the number is not strictly comparable.
• Single run, single seed. Differences of a point or two are within noise.

Per-sample outputs and summaries are in eval/.

Limitations

• It is tiny (5.1M parameters). Expect factual errors, invented details, arithmetic mistakes, weak multi-step reasoning, and repetitive text in long answers.
• Under near-greedy decoding it can loop. Keep the recommended sampling settings.
• Context is 2,048 tokens, but it was fine-tuned on conversations of up to 1,024 tokens.
• English only.
• No safety tuning and no safety evaluation. Do not use it for medical, legal, financial or other high-stakes decisions.

License

Apache-2.0, as for the base model. The SFT data (smol-smoltalk) is also Apache-2.0.