Multimodal Document Reasoning with an Agentic REPL

DocVQA 2026 @ ICDAR — Multimodal Reasoning over Documents

Solution code for the ICDAR 2026 Competition on Multimodal Reasoning over Documents, hosted on the Robust Reading Competition (RRC).

The task is visual question answering over multi-page documents from eight very different domains. Systems must locate evidence, reason across text and visuals, and return an answer that follows strict formatting rules.

Result: two zero-shot submissions — 9.38% in the ≤8B track with Qwen3-VL-8B + an RLM harness, and 27.50% in the >35B track with Gemini 3 Flash + an agentic REPL. The final REPL configuration reached 46.25% on the public validation set but was not submitted to the test leaderboard.

The Task

Each sample contains a document with one or more page images and several questions. The documents span eight domains:

Domain	What makes it difficult
Business Reports	Long documents, dense tables, exact financial values
Comics	Dialogue transcription and temporal reasoning across panels
Engineering Drawings	Tiny labels, dimensions, and spatial annotations
Infographics	Mixed charts, icons, labels, and visual layouts
Maps	Route following, geographic labels, and spatial reasoning
Science Papers	Evidence distributed across text, figures, and tables
Science Posters	Non-linear layouts with small, scattered content
Slides	Searching across decks and interpreting visual elements

Answers are evaluated per domain and overall using the official competition evaluation. Formatting is part of the task: dates must use YYYY-MM-DD, thousands separators are forbidden, units must be abbreviated, multiple answers must be comma-separated, and unanswerable questions must return exactly Unknown.

The dataset is available on Hugging Face: VLR-CVC/DocVQA-2026.

Approach

The core insight is that model capability matters more than increasingly complex scaffolding around a weak model, but a capable model still benefits substantially from well-designed document exploration tools.

The project developed two main systems:

1. Qwen3-VL-8B RLM harness

• Scans document pages in parallel with question- and domain-specific prompts
• Extracts and scores candidate evidence
• Selects the most relevant pages under a context budget
• Aggregates OCR text and rendered page images
• Repairs common formatting errors before submission

This pipeline improved Qwen3-VL-8B from 3.75% to 17.50% validation accuracy and matched the official 8B-thinking baseline at 9.38% on the test set.

2. Gemini 3 Flash agentic REPL

The best system gives the VLM a Python tool environment and lets it decide how to inspect each document. At every step, the model writes code, the harness executes it, and the resulting text or images are returned to the model.

Available tools include:

• scan_pages() and view_pages() for document-level navigation
• view_page(), zoom(), and pan() for visual inspection
• ocr_page() and ocr_region() for precise text extraction
• search_text() for locating evidence in long documents
• SUBMIT() for terminating with a formatted answer

The agent follows a skim → spot → drill → submit workflow rather than a fixed page-processing pipeline.

Best validation configuration — modal_rlm_agent.py, 46.25%:

• Backbone: Gemini 3 Flash via Replicate
• Reasoning: thinking_budget=4096
• Loop: agentic code-execution REPL, max_steps=10
• OCR: GLM-OCR on Modal
• Maps: overlapping quadrant OCR to retain spatial context
• Long documents: cached full-document keyword search
• Short documents: page image and OCR preloaded at step 0
• Inference: six parallel workers

See experiment_plan.md for the complete experiment log and paper/competition_report.tex for the competition report.

Key Findings

• A stronger model was the step change. The same REPL scored 8.75% with Qwen3-VL-8B and 43.75% with Gemini 3 Flash.
• Tool design beat prompt iteration. Quadrant OCR raised maps from 0% to 20%, while text search raised slides from 30% to 50% on validation.
• Question-directed scanning beat scan-once summaries. Generic page summaries lost the targeting needed for specific financial and scientific questions.
• Visual grounding mattered. Returning relevant page images alongside OCR evidence improved exact values and visually encoded answers.
• Small validation sets overfit quickly. Business reports reached 50% on validation but only 5% on test, so validation gains must be interpreted cautiously.
• Formatting repairs recover real accuracy. Unit normalization, number cleanup, list deduplication, and strict Unknown handling prevented avoidable evaluation failures.
• Maps remain the hardest domain. OCR can recover labels, but route and geometric reasoning remain major capability gaps.

Repository Layout

icdar-docvqa2026/
├── AGENTS.md                  # Competition and workflow guidelines
├── experiment_plan.md         # Full experiment log and findings
├── modal_infer.py             # Direct, RLM, and OCR-first inference pipelines
├── modal_rlm_agent.py         # Agentic REPL + Gemini/Qwen backends
├── modal_ocr.py               # GLM-OCR deployment on Modal
├── modal_serve.py             # Qwen3-VL-8B vLLM deployment
├── pyproject.toml             # uv project configuration
├── src/
│   ├── dataset.py             # Hugging Face dataset loading
│   ├── evaluate.py            # Local exact-match evaluation helper
│   ├── format_answer.py       # Answer extraction and normalization
│   ├── inference.py           # Gemini/OpenAI inference helpers
│   └── prompts.py             # Baseline and domain-specific prompts
├── paper/                     # ICDAR competition report and compiled PDF
├── slides/                    # Competition overview decks
├── data/                      # Gitignored local data
└── submissions/               # Gitignored generated RRC submissions

Setup

Dependencies are managed with uv, and GPU workloads run on Modal.

uv sync

Configure the following Modal secrets:

• replicate-secret — REPLICATE_API_TOKEN for Gemini 3 Flash
• hf-secret — HF_TOKEN for Hugging Face models and dataset access

Deploy the self-hosted model and OCR services when using those backends:

uv run modal deploy modal_serve.py
uv run modal deploy modal_ocr.py

Usage

Run the agentic Gemini pipeline on a small validation sample:

uv run modal run modal_rlm_agent.py \
  --action quickval \
  --provider replicate \
  --thinking-budget 4096 \
  --workers 6

Run full validation or test inference. Use --detach for long jobs so they survive laptop sleep or disconnects:

# Full validation
uv run modal run --detach modal_rlm_agent.py \
  --action val \
  --provider replicate \
  --thinking-budget 4096 \
  --workers 6

# Full test submission
uv run modal run --detach modal_rlm_agent.py \
  --action test \
  --provider replicate \
  --thinking-budget 4096 \
  --workers 6

# Download the latest result from the Modal volume
uv run modal run modal_rlm_agent.py --action download

Run the Qwen3-VL-8B pipelines:

# Direct inference through the deployed vLLM endpoint
uv run modal run --detach modal_infer.py --action val --mode direct

# Question-directed page scanning and evidence aggregation
uv run modal run --detach modal_infer.py --action val --mode rlm

# OCR-first document processing
uv run modal run --detach modal_infer.py --action val --mode ocr

Generated submissions follow the RRC JSON format:

[
  {
    "question_id": "maps_2_q1",
    "answer": "Macadam & Gravel",
    "full_answer": ""
  }
]

Results

Approach	Model / pipeline	Validation	Test
Qwen zero-shot	Qwen3-VL-8B, direct prompting	3.75%	5.00%
Qwen RLM harness	Page scan → evidence selection → visual aggregation	17.50%	9.38%
Gemini REPL, submitted	Agentic exploration, five steps	43.75%	27.50%
Gemini REPL + domain guides	Ten steps, search, quadrant OCR, auto-context	46.25%	DNS
Official Gemini baseline	Gemini 3 Flash, zero-shot	—	35.63%
Winning system	Uni-Parser + model ensemble	—	66.25%

DNS means the final validation configuration was not submitted to the test leaderboard. Validation used the public 80-question split; test scores are from the RRC leaderboard.

Further Reading

• Competition report (PDF)
• Full experiment plan and results
• Official evaluation repository
• Competition dataset