prism-ml/Ternary-Bonsai-27B-gguf
Text Generation • 27B • Updated • 613k • • 1.27k
Stephen Genusa PRO
StephenGenusa
AI & ML interests
NLP, RAG/Hybrid/Graph, LLM, SLM, Optimized Quantization, Vision, Multimodality
Recent Activity
liked a model about 7 hours ago
prism-ml/Ternary-Bonsai-27B-gguf liked a model 11 days ago
ornith-ai/Ornith-1.5-35B-A3B liked a model 14 days ago
empero-ai/Qwen3.8-27B-Ridge-GGUFOrganizations
liked a
model about 7 hours ago
liked a
model 11 days ago
liked a
model 14 days ago
liked 2
models 17 days ago
liked 2
models 22 days ago
reacted to Banaxi-Tech's post with 🔥 about 2 months ago
Post
4250
Today we are releasing BananaMind-KV1-8M-2Bit-Experimental, a KV-cache-aware trained model that stores its generation KV cache in 2-bit precision instead of the usual 16-bit precision.
Result: 5.33x smaller KV cache vs FP16, with 0.0916 mean KLD against a 16-bit KV cache reference on WikiText-2.
Model: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
The important part: this is not just post-training KV cache quantization.
Instead we take the BitNet approach.
KV1 is trained with a 2-bit-aware K/V path. Instead of training a normal model and quantizing the cache afterwards, the model learns during training to operate under the low-bit KV constraint, closer in spirit to the BitNet idea of training for the low-bit regime.
During generation, each K/V vector is quantized into 4 affine levels and packed into uint8 tensors, with four 2-bit values stored per byte.
WikiText-2 eval vs 16-bit KV cache reference:
Mean KLD: 0.0916 nats/token
Mean KLD: 0.1322 bits/token
Average KV cache shrink vs FP16: 5.33x
Evaluated positions: 372,675
If this actually gets used in models like Qwen or Gemma, then it may be possible to run 128K or even 256K Context on a Normal Machine!
Try it here: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
Code: https://github.com/Banaxi-Tech/kv1
Result: 5.33x smaller KV cache vs FP16, with 0.0916 mean KLD against a 16-bit KV cache reference on WikiText-2.
Model: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
The important part: this is not just post-training KV cache quantization.
Instead we take the BitNet approach.
KV1 is trained with a 2-bit-aware K/V path. Instead of training a normal model and quantizing the cache afterwards, the model learns during training to operate under the low-bit KV constraint, closer in spirit to the BitNet idea of training for the low-bit regime.
During generation, each K/V vector is quantized into 4 affine levels and packed into uint8 tensors, with four 2-bit values stored per byte.
WikiText-2 eval vs 16-bit KV cache reference:
Mean KLD: 0.0916 nats/token
Mean KLD: 0.1322 bits/token
Average KV cache shrink vs FP16: 5.33x
Evaluated positions: 372,675
If this actually gets used in models like Qwen or Gemma, then it may be possible to run 128K or even 256K Context on a Normal Machine!
Try it here: BananaMind/BananaMind-KV1-8M-2Bit-Experimental
Code: https://github.com/Banaxi-Tech/kv1
- 4 replies
liked 7
models 3 months ago
yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF
Text Generation • 12B • Updated • 497k • 2.87k
ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF
20B • Updated • 5.07k • 10
ManniX-ITA/gemma-4-A4B-98e-v7-coderx-it
20B • Updated • 10 • 2
ManniX-ITA/gemma-4-A4B-98e-v7-coderx-it-GGUF
20B • Updated • 1.59k • 6
ManniX-ITA/gemma-4-A4B-98e-v5-coder-it
20B • Updated • 15 • 3
ManniX-ITA/gemma-4-A4B-98e-v6-coder-it
20B • Updated • 19 • 1
google/diffusiongemma-26B-A4B-it
Image-Text-to-Text • 26B • Updated • 1.17M • 1.2k
liked 3
models 5 months ago
upvoted an article 6 months ago
Article
GGML and llama.cpp join HF to ensure the long-term progress of Local AI
- +4
ggerganov, ngxson, allozaur, lysandre, victor, julien-c
• • 510 liked a
model 6 months ago