pflash-request-probe

The request probe of PFlash: it finds which tokens of a long prompt carry the user's request, so the relevance head ranks the prompt against the right question.

PFlash is Lucebox's prefill compression for long prompts. A small drafter, Qwen3.5-0.8B, reads the whole prompt once, and three small components on its activations decide what the large reader model prefills:

1. Lucebox/pflash-segment-probe cuts the prompt into whole units (statements and definitions, sentences, records, chat turns, tool calls);
2. Lucebox/pflash-request-probe finds the request, wherever it sits in the prompt;
3. Lucebox/pflash-scorer-head ranks the units against that request.

The highest-ranked units that fit the budget go to the reader in their original order, and the reader prefills only those. All three read the same tap, the drafter's block-14 residual (post_block14_residual_before_block15), from the same single pass.

How it works

The request is not always at the end of the prompt. In a RAG prompt it can sit above the retrieved passages; in a chat it can be a short follow-up after a long paste, or a question inside the pasted message itself.

• Architecture: LayerNorm(1024) → Linear(1024, 256) → GELU (erf) → Linear(256, 1), one logit per token for "this token is part of the current request"; 264,705 parameters. GGUF architecture requestprobe, schema qwen3_5_0_8b_request_probe_v1.
• Query: the segment probe's units are intersected with each message of the conversation. A unit's score is the highest request logit inside it, and units are ranked against each other, not against an absolute threshold. The head's query is the latest user message's top-ranked whole units, so a question inside a long paste is found wherever it sits. The query is never empty, and there is no fixed window.
• Pin: the latest user message's top-ranked whole units, up to 512 tokens (PFLASH_QUERY_PIN), are always kept verbatim in the served prompt and charged to the budget.
• Fail closed: if the probe, the segment probe or the message mapping is missing, the server refuses the configuration instead of silently serving without it.

Files

file	sha256	size
request-probe.gguf	5915557d4b60f5a08c8ef18fdd7a30cba56c3a24fda1c2b988da4a41313ebf49	1,060,192 bytes

Using it

PFlash runs inside the Lucebox server. Download the three components and the drafter, then set:

export PFLASH_SCORING_HEAD_GGUF=/path/to/seg-scoring-head.gguf
export PFLASH_SEGMENT_PROBE_GGUF=/path/to/segment-probe-s3-line512.gguf
export PFLASH_REQUEST_PROBE_GGUF=/path/to/request-probe.gguf
export PFLASH_SELECT_SEGMENTS=probe PFLASH_SELECT_SCORE=density PFLASH_SELECT_MODE=budget_only
export PFLASH_SELECT_QUERY_PROBE=1 PFLASH_QUERY_CANDIDATES=1 PFLASH_QUERY_PIN=512
export PFLASH_SELECT_QUERY_TOKENS=128 PFLASH_SELECT_QUERY_HEAD=1
export PFLASH_SELECT_PARAGRAPH_JOIN=0 PFLASH_SELECT_CHUNK_SIZE=256
export PFLASH_ADMISSION=cache PFLASH_ARRIVE=1 PFLASH_CHAT_RECALL=1
export PFLASH_CHAT_REBUILD_RATIO=2 PFLASH_CHAT_REBUILD_MIN_TOKENS=65536
export PFLASH_FALLBACK_SPREAD=0.11 PFLASH_COLD_HISTORY=raw
# server: --prefill-compression auto --prefill-threshold 16384
#         --prefill-curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225
#         --prefill-skip-park --cache-type-k q4_0 --cache-type-v q4_0 --max-ctx 139264

• Drafter (required, checked by sha256 at load): unsloth/Qwen3.5-0.8B-GGUF Qwen3.5-0.8B-BF16.gguf, sha256 cedf89af31c9041b601fa58303285bc46d99c51baee1b13f5e919626ca526ee5. Each component refuses to load against any other drafter file.
• PFlash acts when at least 16,384 prompt tokens are new to the server and less than that is already cached. The reader then prefills 31.5 % of the prompt at 16K tokens, 27 % at 32K–64K and 22.5 % at 120K.
• When the relevance scores are too flat for the budget, the request is served with full prefill (PFLASH_FALLBACK_SPREAD). An agent's uncached earlier conversation is never compressed (PFLASH_COLD_HISTORY=raw).
• Runtime support: Luce-Org/lucebox PR #746.

Results

Measured with all three components together, reader Qwen3.8-27B IQ4_XS, the configuration above, NVIDIA L40S, greedy, each benchmark's official metric:

set	n	full prefill	PFlash
Document QA	30	27	27
LongBench v2	12	6	7
NoLiMa	24	14	14
InfiniteBench CodeDebug	30	24	24
BRIGHT	28	23	23
needle in a haystack, 16K–120K	100	100	100
public sets, unique questions (RULER QA's 15 are inside Document QA)	224	194	195
LongCodeQA, questions that need the code	114	57	62

None of these differences is outside its 95 % interval. The reader prefills a median 29 % of the prompt, and the first token arrives 2.7× sooner on average (mean 30.5 s → 11.2 s over the public requests; 60.4 s → 14.8 s median on 82K–98K-token CodeDebug prompts).

Limits

• Not lossless: a request can still lose something the answer needed.
• Cold agent histories are served with full prefill, so a resumed agent session gets no speed-up.
• "List everything matching X" over many similar units (emails, wiki pages) is the hardest case; the flat-score fallback catches most of it.
• Measured serving one request at a time.