tugot17/mango-9

LFM2.5-VL-3B DSpark draft (4 layers)

DSpark speculative-decoding draft model for LiquidAI/LFM2.5-VL-3B, for use with SGLang. 279.5M parameters, 4 layers, block size 9, Markov head rank 256, confidence head enabled.

This is a drafter, not a standalone model: it only runs alongside the target above.

Usage

python -m sglang.launch_server \
  --model-path LiquidAI/LFM2.5-VL-3B \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path tugot17/mango-9 \
  --speculative-draft-attention-backend flashinfer \
  --speculative-dspark-block-size 9 \
  --disable-radix-cache

Serving LFM2-VL with DSpark needs two delegating members on Lfm2VlForConditionalGeneration (lm_head and set_dflash_layers_to_capture), since the speculative stack introspects the top-level model while LFM2-VL keeps the decoder under language_model. Until that lands upstream, use the branch in the PR linked below.

Measured performance

1xH100, bs=1, greedy, --disable-radix-cache, full MMSpec test split (600 samples / 723 turns / 100 per subset), base/spec/base bracket:

subset	speedup	accept length
MMMU-Pro	2.56x	4.20
COCO captions	2.46x	4.59
CharXiv	1.96x	4.71
multi-turn	1.88x	3.78
TextVQA	1.73x	4.55
GQA	1.66x	4.04
overall	2.08x	4.22

537 vs 258 tok/s overall. Accept length is completion_tokens / spec_verify_ct, i.e. tokens emitted per target forward pass, bonus token included; MMSpec's MAT convention is this minus one. The number above is a request-mean; the token-weighted cycle tau is 3.95.

At concurrency on the same hardware the speedup decays as the target verify becomes compute-bound, while acceptance stays flat within 0.4%: 1.55x at batch 8, 1.32x at batch 32, 1.18x at batch 64.

Notes

• Block size 9 is the trained width and the best setting on GPU; block 8 is 4.7% slower end-to-end. On Metal the tradeoff reverses, because 8 rows exactly fill the verifier threadgroup.
• Speculative decoding preserves the target's output distribution, not a token-for-token match against a non-speculative run. Under verify the target processes a block of candidates in one pass, so reduction orders differ and near-ties can flip. Measured exact-text agreement against the non-speculative arm is 388/723 turns on this split; draft-vs-draft determinism is the correctness gate and it holds.

Related

• Target model: LiquidAI/LFM2.5-VL-3B
• Text DSpark drafts: 1.2B, 8B-A1B