You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**EdgeBench** is a benchmark of **134 real-world tasks** for evaluating how autonomous AI agents *learn from real-world environments*. Instead of measuring one-shot performance, EdgeBench places agents in executable task environments with realistic, multi-level feedback and lets them iterate for **12+ hours** per task — tracking the full trajectory of improvement, not just the final score. We publicly release **51 tasks** along with the full evaluation framework.
28
28
29
-
Analyzing ~25,000 hours of agent interaction across 1,932 runs on all 134 tasks, we find that **performance follows a log-sigmoid scaling law as a function of interaction time** ($R^2 = 0.998$), and that agent learning speed roughly doubles every three months across frontier model generations. See the [paper](assets/paper.pdf) for details.
29
+
Analyzing ~25,000 hours of agent interaction across 1,932 runs on all 134 tasks, we find that **performance follows a log-sigmoid scaling law as a function of interaction time** ($R^2 = 0.998$), and that agent learning speed roughly doubles every three months across frontier model generations. See the [tech report](assets/paper.pdf) for details.
30
30
31
31
> To evaluate on the full 134-task suite, please contact [zhongshu@bytedance.com](mailto:zhongshu@bytedance.com).
Copy file name to clipboardExpand all lines: docs/en/guide/agents.md
+9-10Lines changed: 9 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Supported Agents
2
2
3
-
SForge manages different Code Agents through the `AgentSpec`registry. Each Agent defines installation commands, run commands, API configuration, and related metadata, and is executed by the unified `run_agent()` function.
3
+
SForge manages different Code Agents through a plugin-style agent registry. To run EdgeBench evaluations, simply pick a built-in agent (e.g., `claude-code`) and provide an API key.
4
4
5
5
## Agent Registry
6
6
@@ -57,16 +57,15 @@ sforge run --task ad_placement_optimization --agent claude-code
57
57
58
58
The complete Agent lifecycle is:
59
59
60
-
1.**Build images**: build the Base, Work, and Judge images
61
-
2.**Create container**: create a Docker container from the Work image and inject environment variables such as API key and Judge URL
62
-
3.**Install Agent runtime**: run the Agent's `install_cmds` inside the container, such as installing Node.js or npm packages
63
-
4.**Install evaluation tools**:
60
+
1.**Create container**: create a Docker container from the Work image and inject environment variables such as API key and Judge URL
61
+
2.**Install Agent runtime**: run the Agent's `install_cmds` inside the container, such as installing Node.js or npm packages
62
+
3.**Install evaluation tools**:
64
63
-`sforge-submit`: installed into `/usr/local/bin/`; the Agent calls this command to submit code
65
64
- Stop Hook (optional): prevents the Agent from exiting too early
66
65
- Auto-eval daemon (optional): periodically evaluates in the background
67
-
5.**Generate enhanced prompt**: combine the original task description with evaluation instructions and strategy suggestions
68
-
6.**Run Agent**: execute the Agent's `run_cmd`; the Agent starts working inside the container
69
-
7.**Collect results**: after the Agent times out or finishes, read the best score from the state file
66
+
4.**Generate enhanced prompt**: combine the original task description with evaluation instructions and strategy suggestions
67
+
5.**Run Agent**: execute the Agent's `run_cmd`; the Agent starts working inside the container
68
+
6.**Collect results**: after the Agent times out or finishes, read the best score from the state file
70
69
71
70
During the process, the Agent can call `sforge-submit` at any time to submit code and receive feedback. The background auto-eval daemon also periodically submits evaluations.
Copy file name to clipboardExpand all lines: docs/en/guide/introduction.md
+13-5Lines changed: 13 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,6 +6,14 @@ SForge is a Code Agent evaluation framework designed for ultra-long-horizon iter
6
6
7
7
SForge natively supports EdgeBench, which contains **130+ evaluation tasks**. Tasks cover Python, Java, Go, Rust, C/C++, and other runtime environments, and support multiple evaluation types such as test-driven tasks, score optimization, interactive games, and theorem proving.
8
8
9
+
## Key Mechanisms
10
+
11
+
SForge's design centers on three mechanisms:
12
+
13
+
1.**Two-container isolation** — work and judge environments are fully separated, preventing evaluation hacking at its root. The agent never sees the test suite.
14
+
2.**Iterative evaluation with feedback** — agents don't submit once at the end for a one-shot score; instead they submit throughout the run, receive granular feedback (pass rates, failing tests, scores), and improve in a closed loop until timeout — the best result across all submissions is the final score.
15
+
3.**Long-horizon execution** — stop hooks prevent premature agent exit, auto-resume recovers from transient failures, and the Kubernetes backend enables parallel runs at scale.
16
+
9
17
## Two-Container Architecture
10
18
11
19
SForge's core design fully isolates the agent workspace from the evaluation environment:
@@ -46,10 +54,10 @@ SForge lets agents submit code for evaluation multiple times and receive feedbac
46
54
47
55
In addition, SForge includes an **auto-eval** mechanism: a background daemon periodically submits the code for evaluation (default: every 300 seconds), ensuring that agent progress is recorded in time even if the agent forgets to submit manually.
48
56
49
-
## Key Mechanisms
57
+
## Long-Horizon Execution
50
58
51
-
SForge's design centers on three mechanisms:
59
+
Evaluation tasks typically run for tens of minutes or even hours. SForge uses several mechanisms to ensure agents fully utilize the allocated time:
52
60
53
-
1.**Two-container isolation** — work and judge environments are fully separated, preventing evaluation hacking at its root. The agent never sees the test suite.
54
-
2.**Iterative evaluation with feedback** — agents don't submit once at the end for a one-shot score; instead they submit throughout the run, receive granular feedback (pass rates, failing tests, scores), and improve in a closed loop until timeout — the best result across all submissions is the final score.
55
-
3.**Long-horizon execution** — stop hooks prevent premature agent exit, auto-resume recovers from transient failures, and the Kubernetes backend enables parallel runs at scale.
61
+
-**Stop Hook**: intercepts the agent's exit requests and prevents it from quitting before time runs out. When the agent decides the "task is done," the stop hook tells it to keep checking and improving.
62
+
-**Auto-Resume**: when the agent crashes due to API disconnects, transient errors, or other abnormal exits, the harness automatically re-launches it using the agent's native session resume mechanism (e.g., `claude --continue`) and passes the remaining timeout budget to the resumed session.
Start the experiment hub server: shared file sync API plus visualizer over one runs directory. Remote harness machines push run artifacts here for centralized viewing.
271
-
272
-
```bash
273
-
sforge hub --runs-dir logs/runs --port 8090
274
-
```
275
-
276
-
| Flag | Default | Description |
277
-
|------|---------|-------------|
278
-
|`--runs-dir`|`logs/runs`| Directory to store received run folders |
279
-
|`--tasks-dir`|`tasks/`| Directory of task JSON definitions |
280
-
|`--host`|`0.0.0.0`| Bind address |
281
-
|`--port`|`8090`| Listen port |
282
-
283
-
Configure harness machines to push results to the hub with:
284
-
285
-
```bash
286
-
export SFORGE_HUB_URL="http://<hub-host>:8090"
287
-
export SFORGE_HUB_USER="alice"# optional, namespaces runs as alice-<run_id>
0 commit comments