--- license: apache-2.0 library_name: transformers tags: - causal-lm - text-generation - transformer - decoder-only - fixed-embeddings - binary-token-codes - research language: - en --- # Fixed Minimal Binary Code Model This is an anonymized research checkpoint for the paper: **Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes** ## Model variant This repository contains the **fixed minimal binary token-code model**. Instead of a trainable input embedding table, each token ID is represented by its exact minimal binary code. For vocabulary size: ```text V = 65,536 ``` the minimal injective binary code width is: ```text K = ceil(log2(V)) = 16 ``` The 16-dimensional binary code is tiled to model width 1024. The model therefore uses: ```text 0 trainable input-embedding parameters ``` The output projection remains standard and trainable. ## Architecture - decoder-only Transformer - vocabulary size: 65,536 - model width: 1024 - number of layers: 32 - number of attention heads: 32 - context length: 1024 - rotary positional embeddings - GELU activations - untied trainable output projection ## Loading example ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM repo_id = "E6E831728/fixed-minimal-binary-code" tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True) model.eval() prompt = "Question: What is the capital of France?\nAnswer:" input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long) with torch.no_grad(): output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False) print(tokenizer.decode(output_ids[0].tolist())) ``` ## Standardized base-model evaluation The checkpoint was evaluated as a base causal language model with EleutherAI LM Evaluation Harness `v0.4.10`. Evaluation protocol: - Hugging Face backend: `hf` - maximum context length: 1,024 - `add_bos_token=False` - no chat template - deterministic likelihood-based evaluation - harness seeds: `0,1234,1234,1234` - base checkpoints only; no SFT or instruction checkpoints | Metric | Learned input table | Fixed Binary-16 | Affine GF(2), table-free | SmolLM2-135M | SmolLM2-360M | |---|---:|---:|---:|---:|---:| | HellaSwag acc | 28.49 ± 0.45 | 29.04 ± 0.45 | 29.04 ± 0.45 | 35.36 ± 0.48 | 43.05 ± 0.49 | | HellaSwag acc_norm | 31.32 ± 0.46 | 32.32 ± 0.47 | 31.80 ± 0.46 | 43.02 ± 0.49 | 56.28 ± 0.50 | | ARC-Easy acc | 46.38 ± 1.02 | 47.90 ± 1.03 | 47.64 ± 1.02 | 64.44 ± 0.98 | 70.24 ± 0.94 | | ARC-Easy acc_norm | 40.70 ± 1.01 | 40.87 ± 1.01 | 41.20 ± 1.01 | 58.75 ± 1.01 | 68.18 ± 0.96 | | ARC-Challenge acc | 20.39 ± 1.18 | 19.62 ± 1.16 | 21.33 ± 1.20 | 28.07 ± 1.31 | 36.26 ± 1.40 | | ARC-Challenge acc_norm | 25.85 ± 1.28 | 26.19 ± 1.28 | 24.83 ± 1.26 | 29.61 ± 1.33 | 38.05 ± 1.42 | | PIQA acc | 62.35 ± 1.13 | 62.57 ± 1.13 | 62.68 ± 1.13 | 68.44 ± 1.08 | 71.38 ± 1.05 | | PIQA acc_norm | 60.61 ± 1.14 | 62.08 ± 1.13 | 60.94 ± 1.14 | 68.39 ± 1.08 | 71.82 ± 1.05 | | WinoGrande acc | 50.20 ± 1.41 | 50.12 ± 1.41 | 50.43 ± 1.41 | 52.57 ± 1.40 | 59.35 ± 1.38 | | OpenBookQA acc | 18.40 ± 1.73 | 17.20 ± 1.69 | 17.60 ± 1.70 | 22.00 ± 1.85 | 24.80 ± 1.93 | | OpenBookQA acc_norm | 29.20 ± 2.04 | 31.00 ± 2.07 | 29.40 ± 2.04 | 32.60 ± 2.10 | 37.80 ± 2.17 | | CommonsenseQA acc | 20.31 ± 1.15 | 19.90 ± 1.14 | 20.23 ± 1.15 | 19.90 ± 1.14 | 21.05 ± 1.17 | | MMLU 0-shot | 24.13 ± 0.36 | 23.86 ± 0.36 | 24.11 ± 0.36 | 24.24 ± 0.36 | 25.47 ± 0.37 | | MMLU 5-shot | 25.68 ± 0.37 | 25.60 ± 0.37 | 25.66 ± 0.37 | 25.39 ± 0.37 | 25.05 ± 0.37 | | LAMBADA accuracy | 22.38 ± 0.58 | 21.23 ± 0.57 | 21.99 ± 0.58 | 42.97 ± 0.69 | 53.31 ± 0.70 | | LAMBADA perplexity | 95.14 ± 4.01 | 101.74 ± 4.27 | 100.61 ± 4.17 | 19.06 ± 0.63 | 9.38 ± 0.27 | | WikiText word perplexity | 81.04 | 74.87 | 76.17 | 25.53 | 18.84 | | WikiText byte perplexity | 2.27 | 2.24 | 2.25 | 1.83 | 1.73 | | WikiText bits/byte | 1.19 | 1.16 | 1.17 | 0.87 | 0.79 | The three paper checkpoints form the controlled architectural comparison. SmolLM2-135M and SmolLM2-360M are external reference models, not matched baselines: they use different architectures, tokenizers, training mixtures, and much larger pretraining budgets. SmolLM2-135M was trained on approximately 2T tokens and SmolLM2-360M on approximately 4T tokens, whereas the paper checkpoints saw approximately 16–17B tokens. Their scores therefore provide context for absolute capability and must not be interpreted as isolating the effect of the input parameterization. Perplexity values should be interpreted especially cautiously across different tokenizers. The primary controlled comparison is among the three paper models, which share the same tokenizer, data pipeline, and architecture. ## Input-interface audit This checkpoint stores a deterministic `65,536 × 16` binary codebook as a frozen `nn.Embedding` for computational convenience. During training, the table was initialized from the fixed token codes, marked with `requires_grad=False`, and excluded from the optimizer. It therefore contained 1,048,576 stored but non-trainable values and contributed zero trainable input parameters. The released checkpoint can be audited directly: ```python import torch from transformers import AutoModelForCausalLM repo_id = "E6E831728/fixed-minimal-binary-code" model = AutoModelForCausalLM.from_pretrained( repo_id, trust_remote_code=True, torch_dtype=torch.float32, ).cpu().eval() embedding = model.get_input_embeddings() weight = embedding.weight.detach() vocab_size, code_bits = weight.shape ids = torch.arange(vocab_size, dtype=torch.long) positions = torch.arange(code_bits, dtype=torch.long) expected = ((ids[:, None] >> positions[None, :]) & 1).float() expected[model.config.pad_token_id].zero_() print("shape:", tuple(weight.shape)) print("unique values:", torch.unique(weight).tolist()) print("all entries binary:", bool(torch.all((weight == 0) | (weight == 1)))) print("exact canonical-code match:", bool(torch.equal(weight, expected))) print("mismatching entries:", int((weight != expected).sum().item())) assert tuple(weight.shape) == (65536, 16) assert torch.all((weight == 0) | (weight == 1)) assert torch.equal(weight, expected) ``` Expected audit properties: ```text shape: (65536, 16) unique values: [0.0, 1.0] all entries binary: True exact canonical-code match: True mismatching entries: 0 ``` ## Intended use This checkpoint is provided for anonymous review and reproducibility of the paper's main claim: a trainable input embedding table is not necessary for useful language modeling in the studied regime. ## Limitations This model is a research checkpoint. It is not intended for deployment. It may produce incorrect, biased, unsafe, or nonsensical outputs. ## Training data The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.