Post
80
I’ve published an article about Horizon RunMap, an MoE inference architecture I’m developing independently to explore more memory-efficient execution.
In my RTX 5070 benchmark, OLMoE-1B-7B-0924-Instruct reached 61.26 tokens/s during decode after the first token, with 5.99 GiB peak VRAM usage, keeping all compact experts on the GPU.
The architecture combines compact expert representations for both storage and execution, custom GPU kernels, and full or partial GPU residency.
The article explains how these pieces work together, with results across three models and an instruction-following evaluation using IFEval.
If you’re working on MoE inference or GPU memory efficiency, I’d welcome your thoughts.
Read the article: https://huggingface.co/blog/Zaptrapper/horizon-runmap-moe-inference
In my RTX 5070 benchmark, OLMoE-1B-7B-0924-Instruct reached 61.26 tokens/s during decode after the first token, with 5.99 GiB peak VRAM usage, keeping all compact experts on the GPU.
The architecture combines compact expert representations for both storage and execution, custom GPU kernels, and full or partial GPU residency.
The article explains how these pieces work together, with results across three models and an instruction-following evaluation using IFEval.
If you’re working on MoE inference or GPU memory efficiency, I’d welcome your thoughts.
Read the article: https://huggingface.co/blog/Zaptrapper/horizon-runmap-moe-inference