Traffic Accident Detection with an Agentic Multimodal Harness

ACCIDENT @ CVPR — Zero-Shot Traffic Accident Detection Competition

Solution code for the ACCIDENT @ CVPR Kaggle competition, part of the AUTOPILOT @ CVPR 2026 workshop.

The task is zero-shot accident detection from real-world CCTV / dashcam traffic footage. Models are trained (or prompted) on synthetic CARLA video but must generalize to real videos with no real-world labels — a sim-to-real transfer problem.

Result: 🏅 10th place / 106 teams (top ~9%) — private LB 0.44718 (team binga).

The Task

For each test video, predict three things:

Sub-task	Symbol	What	Scoring
When	T	Timestamp of the accident	Gaussian similarity on time error
Where	S	Impact location (center_x, center_y)	Gaussian similarity on (x, y) distance
What	C	Collision type (1 of 5)	Top-1 accuracy

Final metric: ACCIDENT = 3 / (1/T + 1/S + 1/C) — the harmonic mean, so the weakest sub-metric dominates. Improving the worst component has the highest ROI.

Collision types: head-on, rear-end, sideswipe, single, t-bone

Data

Split	Videos	Source	Labels
Train (sim)	2,211	CARLA simulator	Type, timestamp, per-frame bounding boxes, labels.csv
Test (real)	2,027	Real CCTV / dashcam (YouTube)	None (zero-shot); test_metadata.csv priors only

Approach

The core insight is that VLMs are able to weakly generalize despite the sim-to-real gap — a strong zero-shot video-language model is capable of reasoning about real footage without needing real training labels. The winning pipeline is an agentic REPL loop (a harness with vision-based tools) where the VLM writes Python code to progressively explore each video (scan → zoom → compare → classify) before submitting an answer.

Best pipeline — RLM v4 (exp/rlm_v4/), public LB 0.42147:

• Backbone: Gemini 3 Flash (via Replicate), thinking_budget=4096
• Loop: agentic code-execution REPL, max_steps=8, temp=0.4
• Temporal: frame-difference motion detection (avg_spike_max) + VLM refinement
• Type: raw 5-way VLM classification (no override)
• Spatial: fixed center (0.5, 0.5) — model spatial predictions consistently hurt on real data

See [experiment_plan.md](experiment_plan.md) for the full experiment log (60+ experiments).

Key Findings

(Documented in detail in experiment_plan.md)

• Type accuracy is the bottleneck. Harmonic mean punishes the weakest component; sensitivity analysis shows type gains dominate temporal/spatial gains.
• Sim validation does not predict LB ranking. Better sim scores frequently transfer to worse LB scores because of the sim-to-real domain gap.
• Peak-based temporal > centroid-based on real data. Frame-diff spikes are robust to camera shake; motion centroids are pulled by background motion.
• Model spatial predictions hurt. Real dashcam collisions cluster near frame center — the naive (0.5, 0.5) prior beats learned spatial.
• Agentic reasoning is the step change. RLM v4 (0.421) beats single-shot Gemini (0.317) and the Qwen motion+override baseline (0.313).
• Bigger models don't automatically win. Gemini 3 Flash beats Qwen 72B and Gemma 4 31B in the agentic setting; instruction-following and code generation matter more than parameter count.

Repository Layout

accident/
├── AGENTS.md              # Agent/workflow guidelines
├── experiment_plan.md     # Full experiment log + results tables
├── modal_train.py         # Modal entrypoint (data download, EDA, inference)
├── pyproject.toml         # uv project config
├── exp/
│   ├── rlm_v4/            # Best pipeline: agentic REPL + Gemini 3 Flash (LB 0.42147)
│   ├── gemini_flash/      # Gemini 3 Flash single-shot + motion grids
│   ├── rlm_v1/            # RLM / Qwen motion + type pipeline (see its README)
│   ├── rlm_v2..v10/       # Pipeline iterations and ablations
│   └── dota_eval/         # Real-world (Nexar) temporal validation harness
├── scripts/
│   └── merge_submissions.py  # Blend two submission CSVs
├── src/                   # Reusable dataset / model / train helpers
├── data/                  # Gitignored (Modal volume only)
└── submissions/           # Gitignored generated submissions

Setup

Dependencies are managed with [uv](https://github.com/astral-sh/uv). Compute runs on Modal.

uv sync

Required Modal secrets:

• replicate-secret — for Gemini 3 Flash inference
• hf-secret — for HuggingFace-hosted models (Qwen, etc.)

Usage

All heavy jobs run on Modal. Common commands:

# Download competition data to the Modal volume (uses a Kaggle signed URL)
uv run modal run modal_train.py --action download

# EDA: inspect data structure, labels, and submission format
uv run modal run modal_train.py --action eda

# Validate the best pipeline on stratified sim holdout (60 videos)
uv run modal run exp/rlm_v4/run.py --action validate --n-videos 60 \
  --vlm gemini --thinking-budget 4096 --max-workers 4

# Full test inference (detached — survives laptop sleep / disconnect)
uv run modal run --detach exp/rlm_v4/run.py --action infer

# Download the generated submission
uv run modal run exp/rlm_v4/run.py --action download --experiment-name rlm_v4

Validation tiers: MicroCV (quick sanity, ~$0.12) → sim holdout (60 stratified videos) → LB probe. Submissions to Kaggle are limited to 5/day.

Results

Approach	Pipeline	Public LB	Private LB
RLM v6 (best private)	RLM v4 + 3-pass majority voting, thinking=4096	0.42027	0.44718
RLM v4 (best public)	Agentic REPL + Gemini 3 Flash, thinking=4096	0.42147	0.43235
Gemini Flash raw	Single-shot 5-way	0.31627	0.31221
RLM v1 (Qwen)	Motion temporal + type override (~75% single)	0.31328	0.30956
Motion + VLM	Frame-diff temporal + VLM type + 70% single	0.30223	0.29229
Constant baseline	All rear-end, center, midpoint	0.18573	0.19684

Private LB reranked the top: RLM v6 (3-pass self-consistency) generalized best at 0.44718 private despite RLM v4 leading the public board — extra reasoning passes paid off on the hidden split.

License

Apache 2.0.