Skip to content

Commit 5df5544

Browse files
committed
Complete LAGEN Table 1 with Claude coding-agent results
1 parent 7cdbb42 commit 5df5544

3 files changed

Lines changed: 44 additions & 9 deletions

File tree

‎docs/lagen/SOURCES.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ Paths in this table are relative to `../paper/`.
2727
| Framework, Figure 3 | `figures/mll/figure3_v2.png` → unchanged `public/lagen/assets/figures/framework.png` |
2828
| Measured latency structure, Figure 5 | `figures/fig5_profiling_structure/data.csv` and `plot.py` → `src/lagen/data/profiles.json`, `public/lagen/assets/data/profiles.json`, Canvas/SVG |
2929
| Held-out calibration, Figure 4 | `figures/fig4_sim2real_calibration/heldout_30cases_20261002/case_errors.csv` and `plot.py` → `src/lagen/data/calibration.json` and inline SVG |
30-
| Core results, Table 1 | Layout and formatting: `tables/table1_benchmark_overview/table.tex`, `table_caption.tex`, and `render_table.py`. VLA values: `figures/fig1_teaser/data.csv` → `src/lagen/data/results.json`. GPT values: `tables/table1_benchmark_overview/coding_agents.csv` → `src/lagen/data/coding-agents.json`. |
30+
| Core results, Table 1 | Layout and formatting: `tables/table1_benchmark_overview/table.tex`, `table_caption.tex`, and `render_table.py`. VLA values: `figures/fig1_teaser/data.csv` → `src/lagen/data/results.json`. GPT values: `tables/table1_benchmark_overview/coding_agents.csv` → `src/lagen/data/coding-agents.json`. Claude values: Table 1 of `public/lagen/assets/paper.pdf`, updated on 2026-10-18. |
3131
| Mean-delay/profile comparison, Table 2 | `tables/mean_delay_training/data.json` and `table.tex` |
3232
| Cross-delay transfer, matched-delay view of Figure 6 | `figures/fig6_latency_robustness/matched_delay/cells.csv` and `plot_matched_delay.py` → `src/lagen/data/latency-robustness.json` and interactive heatmaps |
3333
| Cross-task transfer, Figure 7 | `figures/fig21_demon_mixed_cross_task/data.csv` and `plot.py` → `src/lagen/data/cross-task.json` and inline SVG |
@@ -390,22 +390,23 @@ wait for an external font stylesheet before initializing its interactions.
390390

391391
## Core-results table
392392

393-
The table follows Table 1 at paper revision
394-
`8f7996b91940746107ca1b9b323bd5238199c0ce`, verified on 2026-10-14.
393+
The table follows Table 1 of `public/lagen/assets/paper.pdf` (page 8),
394+
verified on 2026-10-18. The PDF is committed in website revision `7cdbb42`.
395395
Tasks occupy columns under Games and Embodied Control.
396-
Model/condition combinations occupy nine VLA rows and two GPT reference rows.
396+
Model/condition combinations occupy nine VLA rows and four coding-agent reference rows.
397397
VLA values remain unnormalized mean episode returns from `results.json`.
398398
GPT values retain source precision from `coding_agents.csv`.
399+
Claude Opus 5.5 and Claude Haiku 5.5 values use the precision displayed in the PDF.
399400
Its `Intercept Grab Fast` key uses the website's `Intercept Grab (Fast)` name.
400401

401402
Values with absolute magnitude at least 1,000 use two decimals and `k`.
402403
Other values use one decimal. Decimal ties round away from zero, matching
403404
the paper renderer's `ROUND_HALF_UP`.
404405
Target/Real cells use `#F0EEF2`, bold returns and `#B34A4A` improvement arrows.
405-
Model cells stay white. GPT model names are italic, while their conditions
406+
Model cells stay white. Coding-agent model names are italic, while their conditions
406407
and values use regular weight and `#46414F`.
407408

408-
Both GPT models use Low reasoning effort and simulated OpenVLA latency.
409+
All four coding agents use Low reasoning effort and simulated OpenVLA latency.
409410
The caption and footnote preserve these evaluation conditions and comparison scope.
410411
The table stays visible at every viewport size, with local horizontal scrolling.
411412

‎src/lagen/components/CoreResultsTable.astro‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ const formatReturn = (value: number) => Math.abs(value) >= 1000 ? thousands.form
3333
Zero/Zero is the original policy without delay. Zero/Real runs that policy during live inference.
3434
Target/Real first trains with the matched profile and then evaluates in real time.
3535
Cells report mean episode return on each task's native scale.
36-
Both GPT models use Low reasoning effort to approximate the rapid responses required in game environments.
36+
All four coding agents use Low reasoning effort.
3737
</figcaption>
3838
<div class="table-scroll" tabindex="0" role="region" aria-label="Core results by model, condition, and task">
3939
<table aria-labelledby="core-results-caption" aria-describedby="core-results-note">
@@ -71,7 +71,7 @@ const formatReturn = (value: number) => Math.abs(value) >= 1000 ? thousands.form
7171
})}
7272
</table>
7373
</div>
74-
<p id="core-results-note" class="table-note"><sup>†</sup> GPT results use simulated OpenVLA latency and are not directly comparable to the VLA results.</p>
74+
<p id="core-results-note" class="table-note"><sup>†</sup> Coding-agent results use simulated OpenVLA latency and are not directly comparable to the VLA results.</p>
7575
</figure>
7676

7777
<style>
@@ -81,7 +81,7 @@ const formatReturn = (value: number) => Math.abs(value) >= 1000 ? thousands.form
8181
figcaption strong { color: var(--ink); }
8282
.table-scroll { margin-top: 16px; }
8383
table { min-width: 1100px; table-layout: fixed; font-size: 15px; border-top: 2px solid #46414f; border-bottom: 2px solid #46414f; }
84-
.model-column { width: 116px; }
84+
.model-column { width: 138px; }
8585
.condition-column { width: 98px; }
8686
th, td { padding: 6px 3px; border: 0; text-align: center; }
8787
thead th { font-weight: 600; }

‎src/lagen/data/coding-agents.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,5 +32,39 @@
3232
"Balance": 42.448506682145236,
3333
"Intercept Grab (Fast)": 0.23094618071979767
3434
}
35+
},
36+
{
37+
"model": "Claude Opus 5.5",
38+
"returns": {
39+
"Demon Attack": 235.0,
40+
"Asterix": 560.0,
41+
"Atlantis": 1300,
42+
"AirRaid": 451.3,
43+
"Deadly Corridor": 378.7,
44+
"Flappy Bird": 5.7,
45+
"Ant": 0.3,
46+
"Hopper": 179.9,
47+
"Inverted Pendulum": 8.9,
48+
"Walker2D": 150.0,
49+
"Balance": 44.7,
50+
"Intercept Grab (Fast)": 0.6
51+
}
52+
},
53+
{
54+
"model": "Claude Haiku 5.5",
55+
"returns": {
56+
"Demon Attack": 82.3,
57+
"Asterix": 0,
58+
"Atlantis": 100.0,
59+
"AirRaid": 0,
60+
"Deadly Corridor": -44.2,
61+
"Flappy Bird": 5.1,
62+
"Ant": -203.1,
63+
"Hopper": 98.6,
64+
"Inverted Pendulum": 0,
65+
"Walker2D": 33.3,
66+
"Balance": 44.7,
67+
"Intercept Grab (Fast)": 0.3
68+
}
3569
}
3670
]

0 commit comments

Comments
 (0)