Post
143
๐ฅฌ LettuceDetect v2 โ span-level hallucination detection for code, tool output, and structured documents.
Hallucination detectors are trained on document QA, but agents ground their answers in source code, tool output and markdown. On code-agent answers, existing detectors reach 0.17 span-F1 and even 550B zero-shot judges at most 0.22.
We built a unified span-level benchmark โ 74,285 newly constructed examples (145K+ with RAGTruth and 14-language PsiloQA folded in), every span typed and character-labeled โ and trained two detectors on it:
๐ค KRLabsOrg/lettucedect-v2-qwen-2b โ generative, typed spans + explanations in one pass, 32K context, **0.689 span-F1** (0.60 on code-agent)
โก KRLabsOrg/lettucedect-v2-mmbert-base โ 307M multilingual encoder for high-throughput setups
๐ท๏ธ KRLabsOrg/lettucedect-v2-taxonomy-head โ types the spans of any binary detector
It also reaches the best reported English PsiloQA IoU (0.724) and 81.8 RAGTruth example-F1, so specializing on code didn't cost general RAG performance.
๐ Dataset: KRLabsOrg/lettucedetect-code-hallucination
๐ Paper: https://arxiv.org/abs/2607.00895
The models are now integrated natively into vLLM Semantic Router โ joint blog post on how it works: https://vllm-sr.ai/blog/lettucedetect-v2-generative-hallucination-detection
Hallucination detectors are trained on document QA, but agents ground their answers in source code, tool output and markdown. On code-agent answers, existing detectors reach 0.17 span-F1 and even 550B zero-shot judges at most 0.22.
We built a unified span-level benchmark โ 74,285 newly constructed examples (145K+ with RAGTruth and 14-language PsiloQA folded in), every span typed and character-labeled โ and trained two detectors on it:
๐ค KRLabsOrg/lettucedect-v2-qwen-2b โ generative, typed spans + explanations in one pass, 32K context, **0.689 span-F1** (0.60 on code-agent)
โก KRLabsOrg/lettucedect-v2-mmbert-base โ 307M multilingual encoder for high-throughput setups
๐ท๏ธ KRLabsOrg/lettucedect-v2-taxonomy-head โ types the spans of any binary detector
It also reaches the best reported English PsiloQA IoU (0.724) and 81.8 RAGTruth example-F1, so specializing on code didn't cost general RAG performance.
๐ Dataset: KRLabsOrg/lettucedetect-code-hallucination
๐ Paper: https://arxiv.org/abs/2607.00895
The models are now integrated natively into vLLM Semantic Router โ joint blog post on how it works: https://vllm-sr.ai/blog/lettucedetect-v2-generative-hallucination-detection
