ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
Paper ā¢ 1705.02315 ā¢ Published ā¢ 1 ā¢
1
Burton Lancaster PRO
RiverRider
AI & ML interests
Explainable AI
Recent Activity
reacted to ginigen-ai's post with š about 1 hour ago
OpenRouter Leaderboard ā every model, every provider, one comparable table. Price, precision, uptime, measured latency and language quality on the same axes.
Building it turned up three things.
We graded 330 models on Korean and two axes collapsed.
Honorifics ā only 8.5% earn an A
Knowledge of Korean institutions ā 9.4%
Every other axis sits above 31%
Fluency hides it. A model can write clean, natural Korean and still attach an honorific to a coffee cup. Fluent and wrong at the same time is worse than obviously broken, because nobody catches it in review.
A 2023 model beats the 2026 flagships. gpt-3.5-turbo-16k scores a perfect 3.00. Korean cannot be inferred from release date, parameter count or English benchmarks ā it has to be measured, per model.
Quality, value and speed are three different models. Across five axes, the same model almost never takes two columns.
425 models, latency measured on 329 on a paid API, Korean graded on 330. Three languages, three currencies, daily refresh, open API, no key.
š https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard š https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard liked a dataset 1 day ago
detection-datasets/coco posted an update 1 day ago
SWE-bench Verified scores whether an agent's patch passes the tests. It does not score whether the agent found the right file first, which is the step before it.
We measured that step on all 500 instances. A 33M-parameter encoder, BAAI/bge-small-en-v1.5 at 384 dimensions, names the correct file first for 229 of 500 (0.458). Plain text search over the same checkouts gets 35 (0.070).
The number that makes those readable is the floor. Hand the same index a bug report from an unrelated project and it still lands the gold file at rank 1 for 5 of 500 (0.010). So 0.458 is 45.8x chance, not 45.8x nothing.
That ratio is where the argument is. recall@50 reads 0.954 and sounds like a solved problem. An unrelated report reaches the same top 50 for 0.244 of instances, so the margin over chance falls from 45.8x at k=1 to 13.6x at k=10 and 3.9x at k=50. The headline that looks best is the one carrying the least.
All 500 ranked lists are published under CC BY 4.0, so the floor can be recomputed rather than believed. The article also ends with eleven corrections to claims we made earlier and got wrong, including one where the lever we proposed turned out to cost accuracy rather than buy it.
We have not measured the patch step. This is the one before it.
For anyone who doesn't live in SWE-bench: it gives model a real GitHub issue from a real project and scores whether the code it writes makes that project's tests pass. That single score covers two jobs, finding the file that needs changing and then changing it correctly, and only the pair is ever scored. The first step is what we measured. Nothing here writes code or runs a test, so 0.458 is a hit rate for naming the right file, not a SWE-bench resolve rate.
Article: https://huggingface.co/blog/RiverRider/finding-the-file-localisation-on-swe-bench-verifie
Data: https://huggingface.co/datasets/RiverRider/swebench-localisationOrganizations
commented a
paper 18 days ago
max(0, Ī) rewards noisy measurement, and the free baseline for adapter gain
#3 opened 19 days ago
by RiverRider
WTF I'm blown away!
š¤Æ 1
10
#1 opened 5 months ago
by 
apehex
Future of Agentic Models
š„ 2
12
#18 opened 5 months ago
by 
MohamedRashad
RiverRider/srt-adapter-v8a
#9516 opened 5 months ago
by RiverRider