Jev-Vision (V9, 8B): an open-weight vision decision model

A LoRA adapter (r=64 on the language model's attention and MLP projections, 175M trainable) plus typed decision heads (heads.pt) on Qwen3-VL-8B-Instruct. It is non-generative: one forward pass answers every typed question asked about an image (or two images), returning one probability per question. Request and response follow TypeSafe's Jev /v1/systemone shape, so code written for Jev can point at this server. Trained on screens with labels derived from the environment, and on typed questions built from public vision datasets. Code, benchmark, evals and the closed-loop harness: https://github.com/sseanliu/Jev-Vision

What it answers

noul (yes/no probability), choice (a distribution over the supplied options, pointer readout over the option tokens) and score (ordered levels). Questions it was trained on: ground (which marked element to act on), skip (is this element already in the state the task needs), effect (did the last action change the page as intended; two images), done (is the goal achieved), the jev-ultrafast operation / click_target questions, and general-image questions (object presence, VQA multiple choice, dish classification, two-image statement truth).

Measured (held out)

Screens, 30 web sites never seen in training (labels from URL / DOM / field values):

track	skip	effect	done	ground
v0 candidate (marked screenshot + element table)	0.955	0.946	0.944	0.939
v1 candidate, fresh episodes	0.922	0.977	0.887	0.949
v1 pixel (raw screenshot + one marker, no table)	0.931	0.931	0.891	-

Jev 1.13 on the same v1 items (text only): 0.870 / 0.733 / 0.887. jev-ultrafast operation question, 615 held-out rows: 0.911; click_target 0.969. As the decider inside jev-ultrafast it completes a Google Flights one-way search (10 actions, 33 s, all 7 independent page checks).

General images, 1,500 items from held-out splits: POPE 0.907, MME 0.927, NLVR2 0.930, A-OKVQA 0.893, Food-101 0.930 (mean 0.917; the frozen backbone read from next-token logits: 0.911); yes/no ECE 0.034 to 0.047. Latency 74 ms per general item, 150 to 190 ms per screenshot with all of a step's questions, on one H100 at batch 1.

Known limits

• On text-only long documents (JevBench's public hard tier) it is 52% accurate at 91% mean confidence: badly over-confident out of domain. The frozen backbone read from next-token logits is 59% at ECE 0.35 on the same items, so most of this is the backbone's own behaviour on long text; a temperature fitted on out-of-domain images (T = 1.3 on GQA val) moves the hard-tier ECE from 0.28 to 0.24 only. Do not use the stated probabilities on long text documents as calibrated confidences. On general images the yes/no class AUROC is 0.97 to 0.98.
• Web grounding on Mind2Web-style pages 0.823 and macOS desktop grounding 0.822 are below the earlier grounding-only checkpoint (0.850 / 0.914).
• Trained and evaluated on 1280x1000 desktop web screenshots; not on mobile, logged-in or form-heavy flows.

Serving

git clone https://github.com/sseanliu/Jev-Vision && cd Jev-Vision/model
pip install transformers==5.17.* peft accelerate pillow
hf download SeanLiu/Jev-Vision --local-dir runs/v9-8b-ops/final
python serve.py runs/v9-8b-ops/final --port 8811 --temp 0.5

POST /v1/systemone with {"state": ..., "image": <base64 PNG> | "images": [...], "questions": {id: {"type": "noul" | "choice" | "score", "instructions": ..., "criteria": ...}}}.

Files: backbone/ (PEFT adapter), heads.pt, s1_config.json, eval_*.json (the numbers above).