Instructions to use mikecovlee/tinymixtral with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mikecovlee/tinymixtral with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mikecovlee/tinymixtral", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mikecovlee/tinymixtral with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mikecovlee/tinymixtral"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mikecovlee/tinymixtral

SGLang

How to use mikecovlee/tinymixtral with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mikecovlee/tinymixtral" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mikecovlee/tinymixtral" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mikecovlee/tinymixtral with Docker Model Runner:
```
docker model run hf.co/mikecovlee/tinymixtral
```

tinymixtral — v3.0 MoE flagship

Flagship release. A Mixtral-style sparse MoE language model: ~477.5M total / ~276.1M active parameters (top-2 of 4 routed experts), trained from scratch on 8.05B tokens on a single GPU.

HF repo: mikecovlee/tinymixtral

Model details

Parameter	Value
hidden_size	1024
num_layers	16
Attention	Grouped Query Attention (16 heads / 4 KV heads)
Head dim	64
RoPE theta	1,000,000
Norm	RMSNorm + per-head QK-Norm (pre-RoPE)
Experts	4 routed (top-2), aux loss 1e-3
Expert FFN	SwiGLU, intermediate = 2048
Vocab size	32,000 (tied embeddings)
Max position	2,048
Total params	~477.5M
Active params	~276.1M

Design decisions (validated by a 400M-token ablation matrix):

top-2 routing — top-1 was +12.2–19.5% worse in val PPL at 400M tokens; maintain top-2.
QK-Norm kept — neutral at 400M tokens, retained as the trunk default.
aux = 1e-3 — the auxiliary-loss coefficient is insensitive in the 4-expert / top-2 ablation.
LR ladder 5e-4 / 5e-4 / 4e-4 / 3e-4 across the four segments (5e-4 was the sole significant factor, −5.3% val PPL in the matrix).

Training

Four ~2B-token segments, each with its own complete WSD schedule (warmup 700 → stable → linear decay over the final 10%); the segment boundary is the anneal point.

Seg	Tokens (cum.)	Steps	Wall	Peak LR	val PPL
S1	2.00B	40,640	38.8 h	5e-4	17.16 @40k
S2	3.94B	39,421	37.6 h	5e-4	16.22 @38k
S3	6.14B	44,704	42.8 h	4e-4	15.81 @44k
S4	8.05B	38,811	37.1 h	3e-4	15.59 @38k

bf16 weights + autocast, bf16 optimizer states (--bf16-optim), chunked cross-entropy, seed 42
batch 48 × 1024 = 49,152 tokens/step, ~14.3k tok/s
gradient clipping 1.0, AdamW (β=0.9, 0.95, wd 0.1), hourly keep-last-2 checkpoints

Training data

8.05B unique tokens, zero repetition, drawn from six public corpora and split into four strictly disjoint ~2B-token pools (no token is seen twice across the four segments). Validation always uses a held-out shard set that never appears in training.

Source	HF dataset	Share
FineWeb-Edu	`HuggingFaceFW/fineweb-edu` (`sample-10BT`)	44%
DCLM web	`mlfoundations/dclm-baseline-1.0`	20%
Cosmopedia v2 (synthetic)	`HuggingFaceTB/cosmopedia-v2`	12.5%
Code	`nvidia/OpenCodeInstruct`	12.5%
Math	`open-web-math/open-web-math`	6%
Wikipedia	`wikimedia/wikipedia` (`20231101.en`)	6%

Shares are rounded and sum to ~100%.

Evaluation

Measured with lm-evaluation-harness v0.4.12, 0-shot (cuda, bf16).

v3.0 (S4, 8.05B tokens) vs the previous releases:

Task	Metric	v3.0 (477M/276M)	tinymixtral-v1.1-1b (1182M/352M)	tinymixtral-v1.1-0.5b (432M)	tinymixtral-v2.0-beta (498M)
HellaSwag	acc_norm	0.335	0.329	0.308	0.326
PIQA	acc	0.638	0.630	0.616	0.631
WinoGrande	acc	0.515	0.523	0.524	0.506
ARC-Easy	acc	0.478	0.479	0.456	0.474
ARC-Challenge	acc_norm	0.255	0.279	0.247	0.272
OpenBookQA	acc_norm	0.296	0.306	0.288	0.290
BoolQ	acc	0.615	0.620	0.606	0.455
LAMBADA	acc	0.268	0.234	0.227	0.224
Mean	—	0.4250	0.425	0.409	0.397

Harness note. The mean above is the 8-task harness (includes BoolQ); the dense and instruction-tuned cards report a 7-task mean (excludes BoolQ). The two are not directly comparable.

Comparison with similar models

Same suite and settings, measured locally (lm-evaluation-harness v0.4.12, 0-shot, cuda, bf16):

Task	Metric	v3.0 (477M)	SmolLM2-360M	Qwen3-0.6B
HellaSwag	acc_norm	0.335	0.563	0.473
PIQA	acc	0.638	0.719	0.673
WinoGrande	acc	0.515	0.587	0.563
ARC-Easy	acc	0.478	0.705	0.609
ARC-Challenge	acc_norm	0.255	0.383	0.340
OpenBookQA	acc_norm	0.296	0.372	0.316
BoolQ	acc	0.615	0.620	0.643
LAMBADA	acc	0.268	0.532	0.401

SmolLM2-360M was trained on 4T tokens and Qwen3-0.6B on 36T tokens, versus 8.05B tokens (~500× and ~4500× less) for v3.0 on a single consumer GPU — the gap is primarily a data-budget difference, and v3.0 is competitive on BoolQ (0.615 vs 0.620 / 0.643).

Data efficiency: v3.0 matches the 1B MoE trained on the same 8B tokens using 2.5× fewer total parameters and 1.28× fewer active parameters (~1.3× fewer FLOPs per token).

Notes:

Persistent weak spot: ARC-Challenge (0.255 vs 0.279 for the 1B MoE) — long-training sensitive, but it narrowed from −3.2pp at 6B to −2.4pp at 8B.
v3.0 vs 6B (S3, mean 0.4257) is within noise; S4 is best on the most tasks.
Early (10k–14k) and late (32k–34k) val-PPL bumps were transient low-entropy shard effects that self-healed (36k=16.31 → 38k=15.59); no data-pool change was needed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral")

inputs = tokenizer("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Requires transformers and trust_remote_code=True (custom tinymixtral architecture). This is a base (pretrained) model; for chat use the instruction-tuned mikecovlee/tinymixtral-it.

Limitations

Base model: no instruction tuning or safety alignment; not a chat model.
Trained on only 8.05B tokens — knowledge tasks (MMLU, ARC-Challenge) are capacity/budget-bound and far below models trained on trillions of tokens.
English-centric.

Family

mikecovlee/tinymixtral — this model (477.5M MoE base, v3.0 flagship)
mikecovlee/tinymixtral-it — MoE instruction-tuned (v3.0-it, 3M SFT)
mikecovlee/tinymistral-276m — 276M dense base (iso-active ablation)
mikecovlee/tinymistral-276m-it — 276M dense instruction-tuned (3M SFT)
mikecovlee/tinymixtral-v1.1-1b — 1B MoE (earlier flagship)
mikecovlee/tinymixtral-v1.1-0.5b — 0.5B-class MoE (data-quality ablation)
mikecovlee/tinymixtral-v2.0-beta — shared-expert experiment (beta)
mikecovlee/tinymixtral-v1.0 — legacy (C4)

Naming. The MoE family is published under tinymixtral; the dense 276M iso-active ablation companions use the tinymistral spelling. Both belong to the same project.

Citation

@misc{tinymixtral2026,
  title  = {TinyMixtral: a small Mixture-of-Experts language-model family},
  author = {Michael Lee},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/mikecovlee/tinymixtral}}
}

License

Downloads last month: 290

Safetensors

Model size

0.5B params

Tensor type

F32

Model tree for mikecovlee/tinymixtral

Finetunes

1 model

Datasets used to train mikecovlee/tinymixtral

Collection including mikecovlee/tinymixtral

Tiny Mixtral

Collection

A small Mixtral-style Mixture-of-Experts causal language model for pretraining research on a single consumer GPU. • 8 items • Updated about 17 hours ago • 1