Skip to content

optimization_summary.md

Latest commit

 

History

History
150 lines (112 loc) · 13.1 KB

optimization_summary.md

File metadata and controls

150 lines (112 loc) · 13.1 KB
Raw

Adversarial Idea Refinement: Optimization Summary

Overview

We ran the 3-agent adversarial debate system (Critic → Proposer → Judge) with a GEPA-trained judge (91.7% accuracy on 50K papers) to generate ML systems paper ideas targeting AI venues (ICML/NeurIPS/ICLR). The goal was to reach scores of 8-9/10. We implemented 5 improvements and ran multiple experiments to test them.

All Experiments with GEPA Judge (Score Trajectories)

Experiment Domain Rounds Peak Final Score Pattern
SymState Video reasoning 20 8.0 — 5→6.5→7.5→8→(sparse data)
CausalVideoFT → PhysCounterfact Video physics benchmark 20 9.0 9.0 5→6→7→7→7→8→9→9→9→9→9→9→9→9→9→9→9→9→9→9→5
PhysDPO → Oracle study Physics DPO labeling 20 9.5 9.5 2→5→6→7→7.5→8→4→5.5→6.5→7→7.5→8→7→8→8.5→9→9.5→9.5→9.5→9.5→9.5
TemporalAttrBind Video generation 20 8.5 8.0 4→6→7→8→6→7→8→7→7→7.5→8→7→7.5→8→7.5→7.5→8.5→6.5→7→7.5→8
DAS-3D (Run 1) ML systems (attention) 40 8.0 7.0 3→6→7.5→8→6→7→7.5→6.5→8→6→6.5→7→7.5→8→7→7.5→8→3→4→6→7→5→5.5→6→6.5→6.5→7→7→...→7.5→7→8→7→7.5→8→5→6→7→8→6→7
AdaptQuant (Run 2) ML systems (quantization) 15 7.0 7.0 4.5→...→6.5→7→6.5→6→5.5→6→6.5→6.5→7→7→7→7→7
Motion-Guided (Run 3) ML systems (video gen) 10 7.0 7.0 Peak 7.0, 3 reproposals used
Calibration-Informed KV (Run 4) ML systems (KV cache) 12 7.5 (×2) 5.5 4→5.5→6→6.5→7→7.5→(rot)→4.5→5.5→6.5→7→7.5→(rot)→4.5→5.5
QuantGraph (Run 4 tournament) ML systems (quantization) 4 6.0 6.0 4.5→5.5→6→6
Compression-Consistent (Run 4 tournament) ML systems (KV cache) 2 5.5 5.5 4.5→5.5

Batch 2: Patient Refinement in ML Systems Domains (Runs 5-9, 40 rounds each)

Settings: --rounds 40 --max-reproposals 0 --early-kill-threshold 0 --critic-threshold 8.0 --min-critics 2 --phase explore

Note: API rate limits caused score data loss after round ~10-15 in most runs. Score trajectories below cover only the scored portion.

Experiment Domain Rounds (scored/total) Peak Last Score Score Pattern
CompGraph Oracle (Run 5) ML compilers 11/120 7.2 7.2 4.5→5.2→5.5→5.8→5.5→5.8→6→5.8→6.5→6.8→7.2 (no rotation before rate limit)
Pipeline Starvation (Run 6 dup) Distributed training 16/40 8.0 4.5 4.5→5→5.5→5.5→5.5→5.5→6→6→6.5→6.5→6.5→6.5→7→7.5→8→(rot)→4.5
WorkloadSense (Run 8) Data loading 12/40 8.0 6.0 4→6→6.5→7→7.5→7.5→8→(rot)→4→4.5→5→5.5→6
Rollback Depth (Run 6) Distributed training 12/40 8.0 4.5 5→5.5→6→6→6.5→7→7→7.5→7.5→7.5→8→(rot)→4.5
Convergence Analysis (Run 7 dup) Distributed training 14/40 8.0 5.5 5→5.5→6→6.5→7→7→7→7→7.5→8→(rot)→5.5→5.5→5.5→5.5
OracleCheck (Run 7 dup) LLM training 13/40 8.0 6.0 4.5→5.5→6→6.5→7→7→7→7.5→7.5→8→(rot)→5→5.5→6
CompilerPortability (dup) ML compilers 13/40 8.5 6.5 4.5→5.5→6→7→7→7.5→7.5→8→err→8.5→(rot)→4.5→5.5→6→6.5
Predicting to Evict (dup) GPU memory 10/40 8.0 5.0 3.5→5.5→6.5→7→7.5→8→(rot)→5→5.5→6→5
Beyond YoungDaly (Run 9 dup) Checkpointing 15/40 8.0 6.0 5→6→6.5→7→7.5→8→(rot)→4.5→5→5.5→5.8→6→6→6→6
Compute-IO Parity (Run 9 dup) IO modeling 7/40 7.0 7.0 4.5→5→5.5→6→6.5→6.5→7 (no rotation before rate limit)

The 5 Improvements Implemented

1. Tournament Mode (generate N candidates, quick-score, debate the best)

  • Status: Implemented and tested (tournament=3 and tournament=5)
  • Result: Did NOT help
  • Why: The quick-scorer and the trained GEPA judge are calibrated differently. Quick-scorer gives 7.0-7.5 to ideas the trained judge scores at 4.5. The tournament selects ideas that sound good but have novelty gaps the trained judge catches immediately. In Run 2, QuantGraph won the tournament at 7.5 quick-score but the trained judge started it at 4.5.

2. Early Kill (force repropose if score below threshold after N rounds)

  • Status: Implemented, had critical bug, fixed
  • Bug: Originally used absolute round number instead of rounds-since-last-reproposal. This caused ALL rounds after round 3 to trigger early kill, burning all 3 reproposals on single-round ideas.
  • After fix: Worked correctly but didn't improve scores. Reproposed ideas landed in equally crowded spaces.
  • Result: Marginal — prevents wasting rounds on dead ideas but doesn't produce better ideas

3. Plateau Detection (force repropose if score stagnant for N rounds)

  • Status: Implemented and working
  • Result: Did NOT help — correctly detected plateaus (e.g., Run 2 at round 3 with scores [5.5, 6.0, 6.0]) but the reproposed ideas weren't better than the originals
  • Root cause: The problem isn't detecting bad ideas — it's generating better ones. Reproposal lands in equally crowded research spaces.

4. Repropose Budget (limit total reproposals to prevent cycling)

  • Status: Implemented, max=3
  • Result: Necessary safety valve — without it, early-kill + plateau detection would cycle endlessly. But it doesn't improve scores by itself.

5. Frontier Seeding (analyze paper corpus to find underexplored gaps)

  • Status: Implemented but minimally tested
  • Result: Inconclusive — the narrow domain seeding in Run 4 (KV cache compression) produced very specific ideas but they still started at 4-4.5 and climbed slowly

Key Findings

Finding 1: The system ALREADY achieves 8-9+ on benchmark/measurement papers

CausalVideoFT pivoted to a benchmark (PhysCounterfact) and scored 9/10 for 14 consecutive rounds. PhysDPO pivoted to an oracle comparison study and scored 9.5/10 for 5 rounds. These are strong, publishable ideas. The judge rates them highly because:

  • Benchmarks are inherently novel (no one can claim prior art on YOUR benchmark)
  • Measurement studies have unfalsifiable novelty ("no one has measured this before")
  • The critic can't easily attack the core contribution

Finding 2: ML systems papers plateau at 7-8 due to dense competition (but 8.5 is reachable)

For ideas proposing novel methods in ML systems (sparse attention, quantization, KV cache compression), the trained judge consistently identifies 2-5 concurrent/prior papers with overlapping contributions. This is accurate — these are extremely crowded research areas. However, Batch 2 showed that ML compiler benchmarking (CompilerPortability) reached 8.5/10 — the highest ML systems score yet. This suggests that benchmarking contributions in systems areas can break through the 8.0 ceiling, consistent with Finding 1.

Finding 3: Score oscillation is driven by critic rotation, not idea quality

In Run 1 (DAS-3D), the score hit 8/10 five times but dropped back to 6-7 each time a fresh critic was rotated in. The new critic finds issues the previous critic missed (new concurrent papers, methodological gaps). This is actually good behavior — it simulates reviewer diversity. The 8s represent "one reviewer would accept this" while the 6s represent "but another reviewer would object."

Finding 4: More rounds help more than smart reproposal

Run 1 (40 rounds, no improvements) outperformed Runs 2-3 (15 and 10 rounds, with improvements). The DAS-3D idea had 40 rounds to accumulate fixes and address objections, hitting 8/10 repeatedly. The improved runs with fewer rounds never got past 7.0. Patient refinement of a single good idea beats rapid cycling through mediocre alternatives. Batch 2 confirmed this conclusively: with 40 rounds and no reproposals, 8/10 experiments hit 8.0+ (one hit 8.5).

Finding 5: The 4→6→7→8 climb takes ~4 rounds; staying at 8 is the hard part

Every experiment follows the same pattern: start at 4-5, climb to 7-8 in 3-4 rounds of critic-proposer debate. The difficulty is sustaining 8+ because:

  • Fresh critics find new prior work
  • The trained judge does independent web searches and finds concurrent papers
  • Each fix introduces new attack surfaces

Finding 6: Quick-scorer calibration mismatch undermines tournament

The quick-scorer (Claude oneshot without web search) systematically over-rates ideas by 2-3 points relative to the trained GEPA judge. This makes tournament selection unreliable — it picks "sounds impressive" over "actually novel."

Finding 7: API rate limits are the biggest practical bottleneck

In Batch 2, most experiments lost scoring data after round 10-15 due to rate limits, meaning 60-70% of rounds ran without judge scores. CompGraph Oracle ran 120 rounds but only 11 produced scores. This wastes compute — the system keeps debating but the judge can't evaluate. Rate-limit-aware scheduling or backoff is needed for production runs.

Finding 8: First-critic climb is remarkably consistent

Across all 10 Batch 2 experiments, the first-critic phase follows the same pattern: start 4-5, climb +0.5/round, peak at 8.0 in rounds 5-14. The consistency suggests the climb rate is a property of the debate format, not the idea quality. What distinguishes ideas is whether they can survive critic rotation.

What Worked

  1. The GEPA-trained judge itself — extremely rigorous, does independent literature verification, catches concurrent papers, holds ideas to genuine novelty standards
  2. Stateful debate sessions — allowing proposer to accumulate context over 20-40 rounds produces deeply refined proposals
  3. Critic rotation — brings fresh perspectives, simulates reviewer diversity
  4. Idea pivoting — the best ideas (PhysCounterfact 9/10, Oracle study 9.5/10) emerged from multiple pivots away from scooped initial ideas
  5. Long runs (20-40 rounds) — more valuable than any of the 5 improvements
  6. Patient refinement with no reproposals — Batch 2's 80% hit rate (8/10 reached 8.0+) validates: just let ideas cook

What Didn't Work

  1. Tournament mode — quick-scorer miscalibration makes it select wrong ideas
  2. Early kill — moves to equally bad ideas faster, doesn't find better ones
  3. Plateau detection — correctly diagnoses the problem but can't fix it
  4. Reproposal in general — new ideas land in similarly crowded spaces
  5. Narrow domain seeding — ML systems subfields are ALL crowded in 2026

What Would Actually Help (Recommendations)

  1. Fix quick-scorer calibration: Use the GEPA judge prompt for tournament scoring (expensive but accurate), or fine-tune a faster scorer on GEPA judge outputs
  2. Let ideas pivot naturally: The 9+ ideas emerged from proposer pivots (method → benchmark), not from forced reproposals. Remove reproposal forcing and instead let the proposer decide when to pivot
  3. Target less crowded research types: Benchmarks, empirical studies, and measurement papers score 9+ while novel methods score 7-8. Consider steering toward these contribution types
  4. Run longer: 40 rounds > 20 rounds > 10 rounds. The incremental cost per round is low (Claude API calls), and the accumulated refinement compounds
  5. Expand beyond video/attention: The current seed ideas cluster in video generation and attention optimization, which are extremely crowded. Use the 2,766 downloaded papers to find genuinely underexplored intersections
  6. Multi-reviewer consensus scoring: Instead of single critic rotation, run 3 independent critics and require majority agreement for score changes

Best Ideas Produced (Ready for Execution)

Rank Idea Peak Score Rounds Status
1 PhysDPO Oracle Study: First comparison of analytical vs. VLM vs. likelihood oracles for kinematic video DPO 9.5/10 20 Ready
2 PhysCounterfact Benchmark: Physics-parameterized counterfactual video editing benchmark 9.0/10 20 Ready
3 CompilerPortability Benchmark: Benchmarking ML compiler portability across hardware targets 8.5/10 40 Ready (Batch 2)
4 TemporalAttrBind: Training-free temporal attribute scheduling via cross-attention 8.5/10 20 Ready
5 DAS-3D: SNR-theoretic sparse attention formula for video DiTs 8.0/10 40 Ready
6 Pipeline Starvation: Implicit synchronization bottlenecks in distributed training 8.0/10 40 Ready (Batch 2)
7 WorkloadSense: Distribution-predictive data loading for training 8.0/10 40 Ready (Batch 2)
8 Rollback Depth: Optimizer staleness in distributed training rollbacks 8.0/10 40 Ready (Batch 2)

Note: Ideas #1-2 are benchmark/measurement papers (video domain). #3 is a benchmark paper in ML systems — the highest-scoring systems idea yet. #4-8 are method/measurement papers.