MRareBench

A Multimodal Rare-Disease Benchmark for Evidence–Diagnosis Correspondence

Can multimodal models connect patient-specific visual evidence to a rare-disease diagnosis?

News

• Dataset release: the curated benchmark is available on Hugging Face.

Overview

MRareBench evaluates multimodal large language models in a weak-prior clinical regime. Rare diseases are individually uncommon, their decisive findings are often visual, and the evidence may be distributed across radiology, pathology, dermatology, ophthalmology, and other modalities.

The benchmark connects two complementary questions:

Can the model name the disease?	Can the model report the evidence?
T1 asks for a ranked differential diagnosis from the clinical context and images.	T2 supplies the diagnosis and asks for the supporting findings visible in the images.

Together, the two tracks measure whether diagnostic predictions and patient-specific visual evidence remain aligned on the same source cases.

Benchmark at a glance

Track	Released scale	Controlled evaluation instances
T1 · Forward Diagnosis	300 cases	900 inputs across FD / LC / TO
T2 · Evidence Verification	608 items	1,824 inputs across Grd / Rec / NoDx
Total	908 released items	2,724 item–condition inputs

Why MRareBench?

Rare diseases create a stringent multimodal setting

For common diseases, prevalence and familiar symptom patterns may narrow the answer before the image is fully examined. MRareBench shifts evaluation toward long-tail diseases for which the model must make greater use of the case-specific visual evidence.

Multi-image evidence is part of the task

Each released item contains multiple medical images. A diagnosis may depend on findings spread across different views, time points, or modalities rather than on a single canonical image.

Diagnosis and evidence are evaluated together

Forward accuracy alone does not show whether a prediction is supported by the images. MRareBench therefore pairs ranked diagnosis with evidence verification and links the two outcomes on shared PMC cases.

Dataset Coverage

T1 covers a wide variety of imaging and examination modalities, such as pathology microscopy, CT, MRI, ultrasound, radiography, dermatology, and ophthalmology. Every released item contains at least two medical images, allowing the benchmark to represent evidence distributed across views, time points, and modalities.

Benchmark Design

Two complementary tracks

Track	Input	Output	Main metric
T1 · Forward Diagnosis	Clinical context + multiple images	Ranked top-10 rare-disease differential	Recall@1/3/5/10, MRR
T2 · Evidence Verification	Specified diagnosis + context + multiple images	Rubric-aligned visible evidence	Hierarchical required-evidence recall

T2 is evidence attribution rather than re-diagnosis: the target disease is fixed, and the model must identify the visual findings that support it.

Construction Pipeline

MRareBench is built from real PubMed Central case reports through a traceable nine-stage pipeline.

From source PMC reports to the released 300 T1 and 608 T2 items.

The pipeline combines three layers:

1. Source grounding: structured extraction, disease-role tagging, task construction, and ORPHA normalization.
2. Multimodal integrity: image-evidence value filtering, diagnosis masking, text–image leakage control, and figure-link repair.
3. Quality control: source-based judging, human verification, deterministic filtering, and frozen release manifests.

Evaluation

We evaluate 37 MLLMs in three cohorts: 14 closed-source models, 13 open-source general models, and 10 medical-specialized models. Every model is evaluated under the same controlled evidence views.

Per-model performance profiles across the three access cohorts.

Metrics

• T1: Recall@1/3/5/10, mean reciprocal rank, and mean first-hit rank over the ranked differential.
• T2: hierarchical required-evidence recall, which scores visible findings before cross-image relations and diagnostic attribution.
• Cross-track correspondence: the rate of correct T1 Top-1 diagnoses among source-linked model–case pairs that meet the T2 evidence threshold.

Main Results

Result	Observation
Best image-grounded diagnosis	54.7% T1 LC Recall@1
Best supplied-diagnosis evidence score	80.5% T2 full-tier recall
Image contribution to T1	LC exceeds TO by 11.5 / 5.8 / 5.0 points for closed/open/medical cohorts
Diagnosis conditioning in T2	Mean Level-1 gain of 24.7 points; positive for all 37 models
Evidence–diagnosis correspondence	26.3% of T2-positive model–case pairs also have a correct T1 Top-1 diagnosis

Images make a consistent difference

When the diagnosis remains supplied but images are removed from T2, evidence reporting falls for all 37 models. The cohort-mean drops are 45.5 points for closed-source models, 26.6 for open-source models, and 15.1 for medical models.

Evidence reporting with the diagnosis and images versus the diagnosis alone.

The diagnosis name also changes evidence reporting

With the images, question, response instruction, and scoring rubric held fixed, providing the diagnosis improves matched Level-1 visible-finding recall across every evaluated model.

Removing the image channel and withholding the diagnosis name produce distinct performance changes.