zeroentropy

/

zerank-2-reranker

Like 142

Follow

ZeroEntropy 219

Hi,

I can get it to work but the scores make no sense, so probably broken in a way. You can run it by adding command line args --hf-overrides '{"architectures": ["Qwen3ForSequenceClassification"], "classifier_from_token": ["no", "yes"], "is_original_qwen3_reranker": true}' . The reranking scores seem random though. Here's my test case:

curl -s -X POST "http://rtx5070:8000/v1/score" -H "Content-Type: application/json" \
 -d '
   {
      "model": "zeroentropy/zerank-2",
      "text_1": "What are the symptoms of diabetes?",
      "text_2": [
        "Common diabetes symptoms include increased thirst, frequent urination, fatigue, and blurred vision.",
        "The Great Wall of China is over 13000 miles long.",
        "Basketball was invented by James Naismith in 1891.",
        "Type 2 diabetes can often be managed with diet and exercise.",
        "Symptoms may also include slow-healing wounds and tingling in hands or feet."],
      "use_activation": false
  }'

This gives these scores:

Doc Raw Logit Softmax Score Content
0 4.25 0.986 Diabetes symptoms (thirst, urination...)
4 6.51 0.998 More symptoms (wounds, tingling)
3 2.87 0.947 Diabetes management
1 2.74 0.940 Great Wall (irrelevant)
2 2.67 0.936 Basketball (irrelevant)
The raw logits show better differentiation (4.25 vs 2.74 for relevant vs irrelevant), but after softmax they compress to 0.986 vs 0.940. In any case the scores are not great, with irrelevant documents getting half the score that a fully relevant document is getting. I assume this is caused by the way of invocation.

Hi! Hope we're not too late to the party 😅

We needed zerank-2 on vLLM too, so we made a converted checkpoint that works out of the box:
polaria-tech/zerank-2-reranker-vllm

@kread your scores looked random for two reasons:

• the ["no", "yes"] overrides are the Qwen3-Reranker recipe, but zerank-2 scores a single Yes token, with no "no" token;
• vLLM doesn't apply the model's prompt template on /score unless you pass --chat-template, so it was scoring raw query + document.

With the converted model:

TEMPLATE=$(hf download polaria-tech/zerank-2-reranker-vllm score_template.jinja --quiet)

vllm serve polaria-tech/zerank-2-reranker-vllm --runner pooling --max-model-len 32768 --chat-template "$TEMPLATE"

/score and /rerank then return the same 0-1 score as the official predict() → sigmoid(logit / 5) (we checked it against the original, within bf16 noise). It's also about 2x faster than sentence-transformers on an H100. Details and a few gotchas (truncation, template) are on the model card.

Hope it helps!