feat: add secure OJBench alpha adapters

benchmark/wave2-code/matrix.tsv

@@ -1,6 +1,7 @@
 candidate	case	rep	metric	resource	status	notes
 ojbench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
-ojbench	official_protocol_audit	464_prompts_232_problems	0	dmoj+g++17+pypy3+git_lfs_testdata	unsupported	Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c, but faithful judging requires DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips; no hardened OpenBench Docker image has been validated for DMOJ under cap-drop/no-network, and OpenCompass only loads prompts without scoring.
+ojbench	official_protocol_audit	464_prompts_232_problems	0	dmoj+g++17+pypy3+git_lfs_testdata	unsupported	Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c. The initial audit established that faithful judging required DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips, while OpenCompass only loaded prompts without scoring; the subsequent alpha adapter row records the completed hardened-runtime work.
+ojbench	openbench_alpha_adapter	232_python+232_cpp_x8	0	lazy_pinned_testdata+dmoj_arm64	alpha_arm64_runtime_validated	Added ojbench_python and ojbench_cpp with eight official samples per problem, pinned prompts/testdata, and lazy per-problem downloads. The arm64 DMOJ image passed CPP17/PYPY3 self-tests and judged both languages over 20 real loj-2083 cases; an attempted hidden-test file read received IR. The container is network-none/read-only/no-new-privileges with bounded memory/PIDs, cap-drop ALL, and only DMOJ-required SYS_PTRACE restored. Full 7.85GB corpus validation and real model scoring remain future gates, so both IDs are alpha.
 tir_bench	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867
 tir_bench	official_protocol_audit	1215_examples_13_tasks	0	agentic_image_harness+gpt4.1_extractor	unsupported	Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF dataset revision ae9976e81e86c4797fd0d28f9119393b37d9d006 publish data and post-generation scoring, but the released scripts require a GPT-4.1 answer extractor and do not publish the agentic image/tool generation harness underlying with-CI/without-CI runs.
 codeforces_elo	registry_baseline	1	0	0	unsupported	No implementation or registry ID at baseline 7f11867

benchmark/wave2-code/ojbench.md

@@ -2,10 +2,12 @@
 
 ## Decision
 
-OJBench is recorded as unsupported for this OpenBench wave. Unlike Spider and
-MEGA, OJBench does publish a full prompt file for LLM generation, but faithful
-scoring depends on a DMOJ-based online-judge runtime that has not been validated
-inside OpenBench's hardened Docker policy.
+OJBench now has alpha `ojbench_python` and `ojbench_cpp` registry IDs backed by
+the official DMOJ judge. Each language track contains 232 problems and uses
+eight samples per problem, matching the paper's Pass@1/Pass@8 protocol. The
+paper uses each model's recommended sampling parameters rather than one global
+temperature/top-p configuration, so OpenBench deliberately leaves those model
+settings configurable.
 
 ## Canonical sources
 
@@ -53,12 +55,35 @@ JSONL file. It does not implement DMOJ setup, code extraction, test execution,
 partial verdicts, or scoring, so it is not sufficient evidence for a faithful
 OpenBench integration.
 
-## OpenBench compatibility finding
+## OpenBench implementation
 
-OJBench should be integrated only once there is a validated Docker execution
-boundary for DMOJ that preserves OpenBench's safety policy: network disabled,
-capabilities dropped, no new privileges, bounded process/memory limits, and no
-host compiler/runtime escape. That image also needs the large LFS problem zips
-or a reproducible cache step. Until that exists, adding a registry ID would risk
-either weakening the sandbox or reporting scores from an unvalidated judge. The
-candidate is therefore blocked rather than approximated.
+- Prompts are checksum-verified and loaded from the pinned Hugging Face
+  revision.
+- Test archives and custom validators are downloaded lazily per problem into a
+  host cache mounted read-only at `/problems`. A limited run therefore does not
+  require downloading the complete 7.85 GB source repository.
+- The image installs OJBench commit
+  `5e94480b1e135b98855cf5bc81213c256aff5b17` and DMOJ commit
+  `f098cd3a49a60186d1fadde5132329ec5f4f2213`, with exact Python dependencies,
+  C++17 `g++`, and PyPy3.
+- The scorer returns only the final verdict, number of executed cases, and
+  partial pass booleans. Per-case inputs, outputs, and feedback never enter
+  Inspect logs.
+
+The container runs without networking, with a read-only root filesystem,
+`no-new-privileges`, bounded memory/PIDs, and `cap_drop: ALL`. DMOJ cptbox
+requires `SYS_PTRACE` to supervise its own child process, so that single
+capability is restored explicitly. Generated programs remain under DMOJ's
+seccomp and filesystem policies; `/problems` is not in their readable policy.
+
+## Validation and remaining gate
+
+On local arm64 Docker, both CPP17 and PYPY3 executor self-tests passed. Wrong
+Python and C++ submissions were judged against 20 real `loj-2083` cases and
+returned `WA`. A submission attempting to read the mounted hidden-test
+`init.yml` returned DMOJ `IR`, confirming the anti-oracle filesystem boundary.
+
+The complete 7.85 GB problem corpus has not yet been executed end to end, and
+no credentialed model run has been scored. The two registry IDs therefore
+remain alpha until broader custom-validator coverage and real Pass@1/Pass@8
+runs are complete.

docs/snippets/benchmarks.data.mdx

@@ -6767,6 +6767,34 @@ export const benchmarksData = [
     "function_name": "ocrbenchv2",
     "is_alpha": false
   },
+  {
+    "name": "OJBench C++",
+    "description": "Competition-level C++ generation with the official DMOJ judge",
+    "category": "core",
+    "tags": [
+      "coding",
+      "cpp",
+      "execution",
+      "docker",
+      "online-judge"
+    ],
+    "function_name": "ojbench_cpp",
+    "is_alpha": true
+  },
+  {
+    "name": "OJBench Python",
+    "description": "Competition-level Python generation with the official DMOJ judge",
+    "category": "core",
+    "tags": [
+      "coding",
+      "python",
+      "execution",
+      "docker",
+      "online-judge"
+    ],
+    "function_name": "ojbench_python",
+    "is_alpha": true
+  },
   {
     "name": "OpenAI MRCR (2 Needles)",
     "description": "Memory-Recall with Contextual Retrieval - long-context evaluation that measures recall of 2 needles across million-token contexts",

packages/openbench-core/pyproject.toml

@@ -57,6 +57,7 @@ include = ["openbench*"]
   "requirements-arm64.lock",
 ]
 "openbench.evals.evalplus" = ["Dockerfile", "compose.yaml"]
+"openbench.evals.ojbench" = ["Dockerfile", "compose.yaml"]
 "openbench.evals.bfcl" = [
   "Dockerfile",
   "Dockerfile.agentic",

pyproject.toml

@@ -62,6 +62,7 @@ include = ["openbench*"]
   "requirements-arm64.lock",
 ]
 "openbench.evals.evalplus" = ["Dockerfile", "compose.yaml"]
+"openbench.evals.ojbench" = ["Dockerfile", "compose.yaml"]
 "openbench.evals.bfcl" = [
   "Dockerfile",
   "Dockerfile.agentic",

src/openbench/config.py

@@ -457,6 +457,24 @@ class EvalGroup:
         module_path="openbench.evals.livebench",
         function_name="livebench_coding_2024_11_25",
     ),
+    "ojbench_python": BenchmarkMetadata(
+        name="OJBench Python",
+        description="Competition-level Python generation with the official DMOJ judge",
+        category="core",
+        tags=["coding", "python", "execution", "docker", "online-judge"],
+        module_path="openbench.evals.ojbench",
+        function_name="ojbench_python",
+        is_alpha=True,
+    ),
+    "ojbench_cpp": BenchmarkMetadata(
+        name="OJBench C++",
+        description="Competition-level C++ generation with the official DMOJ judge",
+        category="core",
+        tags=["coding", "cpp", "execution", "docker", "online-judge"],
+        module_path="openbench.evals.ojbench",
+        function_name="ojbench_cpp",
+        is_alpha=True,
+    ),
     # Exercism benchmarks
     "exercism": BenchmarkMetadata(
         name="Exercism",

src/openbench/datasets/ojbench.py

@@ -0,0 +1,134 @@
+"""Pinned loader for the official OJBench prompt release."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+from pathlib import Path
+from typing import Literal
+
+from huggingface_hub import hf_hub_download, snapshot_download
+from inspect_ai.dataset import MemoryDataset, Sample
+from platformdirs import user_cache_dir
+
+DATASET_REPOSITORY = "He-Ren/OJBench_testdata"
+DATASET_REVISION = "61cf9986f22c25d08e1657b03742124099c74353"
+PROMPT_FILE = "prompts/full.jsonl"
+PROMPT_SHA256 = "bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c"
+PROMPT_COUNT = 464
+
+
+def problem_cache_dir() -> Path:
+    """Return the host directory mounted read-only into the judge."""
+
+    configured = os.environ.get("OPENBENCH_OJBENCH_DATA_DIR")
+    path = (
+        Path(configured).expanduser()
+        if configured
+        else Path(user_cache_dir("openbench")) / "ojbench" / "problems"
+    )
+    path.mkdir(parents=True, exist_ok=True)
+    (path / "NOI").mkdir(exist_ok=True)
+    (path / "ICPC").mkdir(exist_ok=True)
+    os.environ["OPENBENCH_OJBENCH_DATA_DIR"] = str(path.resolve())
+    return path.resolve()
+
+
+def _prompt_path() -> Path:
+    path = Path(
+        hf_hub_download(
+            repo_id=DATASET_REPOSITORY,
+            filename=PROMPT_FILE,
+            repo_type="dataset",
+            revision=DATASET_REVISION,
+        )
+    )
+    digest = hashlib.sha256(path.read_bytes()).hexdigest()
+    if digest != PROMPT_SHA256:
+        raise ValueError(
+            f"OJBench prompt checksum mismatch: expected {PROMPT_SHA256}, got {digest}"
+        )
+    return path
+
+
+def canonical_problem_id(dataset: str, record_id: int | str) -> tuple[str, str]:
+    """Map prompt identifiers to the canonical problem directory."""
+
+    if dataset == "NOI" and isinstance(record_id, int):
+        return "NOI", f"loj-{record_id}"
+    if dataset == "icpc" and isinstance(record_id, str):
+        if re.fullmatch(r"[A-Za-z0-9_-]+", record_id):
+            return "ICPC", record_id
+    raise ValueError(f"Invalid OJBench problem identity: {dataset}/{record_id!r}")
+
+
+def ensure_problem_data(dataset: str, problem_id: str) -> Path:
+    """Download one immutable problem package into the mounted host cache."""
+
+    if dataset not in {"NOI", "ICPC"}:
+        raise ValueError(f"Invalid OJBench dataset: {dataset}")
+    pattern = r"loj-[0-9]+" if dataset == "NOI" else r"[A-Za-z0-9_-]+"
+    if re.fullmatch(pattern, problem_id) is None:
+        raise ValueError(f"Invalid OJBench problem id: {problem_id}")
+
+    root = problem_cache_dir()
+    snapshot_download(
+        repo_id=DATASET_REPOSITORY,
+        repo_type="dataset",
+        revision=DATASET_REVISION,
+        allow_patterns=[f"{dataset}/{problem_id}/**"],
+        local_dir=root,
+    )
+    problem = root / dataset / problem_id
+    if not (problem / "init.yml").is_file():
+        raise FileNotFoundError(f"Missing OJBench init.yml for {dataset}/{problem_id}")
+    if not any(problem.glob("*.zip")):
+        raise FileNotFoundError(
+            f"Missing OJBench test archive for {dataset}/{problem_id}"
+        )
+    return problem
+
+
+def record_to_sample(record: dict[str, object]) -> Sample:
+    dataset = str(record["dataset"])
+    language = str(record["language"])
+    if language not in {"python", "cpp"}:
+        raise ValueError(f"Unsupported OJBench language: {language}")
+    raw_id = record["id"]
+    if not isinstance(raw_id, (int, str)):
+        raise TypeError("OJBench id must be an integer or string")
+    canonical_dataset, problem_id = canonical_problem_id(dataset, raw_id)
+    return Sample(
+        id=f"{problem_id}:{language}",
+        input=str(record["prompt"]),
+        target="",
+        metadata={
+            "record_id": raw_id,
+            "problem_id": problem_id,
+            "dataset": canonical_dataset,
+            "language": language,
+            "difficulty": str(record["difficulty"]),
+            "dataset_revision": DATASET_REVISION,
+        },
+    )
+
+
+def get_ojbench_dataset(language: Literal["python", "cpp"]) -> MemoryDataset:
+    """Load one official 232-problem language track."""
+
+    if language not in {"python", "cpp"}:
+        raise ValueError("OJBench language must be 'python' or 'cpp'")
+    problem_cache_dir()
+    rows = [json.loads(line) for line in _prompt_path().read_text().splitlines()]
+    if len(rows) != PROMPT_COUNT:
+        raise ValueError(f"OJBench expected {PROMPT_COUNT} prompts, got {len(rows)}")
+    samples = [record_to_sample(row) for row in rows if row["language"] == language]
+    if len(samples) != PROMPT_COUNT // 2:
+        raise ValueError(f"OJBench {language} expected 232 prompts, got {len(samples)}")
+    return MemoryDataset(
+        samples=samples,
+        name=f"ojbench_{language}",
+        location=DATASET_REPOSITORY,
+    )

src/openbench/evals/ojbench/Dockerfile

@@ -0,0 +1,51 @@
+FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 AS builder
+
+ARG DMOJ_REVISION=f098cd3a49a60186d1fadde5132329ec5f4f2213
+ARG OJBENCH_REVISION=5e94480b1e135b98855cf5bc81213c256aff5b17
+
+RUN apt-get update \
+    && apt-get install --yes --no-install-recommends build-essential git libseccomp-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN python -m pip install --no-cache-dir \
+        Cython==3.0.10 setuptools==69.5.1 wheel==0.43.0 \
+    && git clone https://github.com/DMOJ/judge-server.git /src/dmoj \
+    && git -C /src/dmoj checkout "${DMOJ_REVISION}" \
+    && test "$(git -C /src/dmoj rev-parse HEAD)" = "${DMOJ_REVISION}" \
+    && git clone https://github.com/He-Ren/OJBench.git /src/ojbench \
+    && git -C /src/ojbench checkout "${OJBENCH_REVISION}" \
+    && test "$(git -C /src/ojbench rev-parse HEAD)" = "${OJBENCH_REVISION}" \
+    && python -m pip wheel --no-build-isolation --no-cache-dir --wheel-dir /wheels \
+        filelock==3.18.0 loguru==0.7.2 PyYAML==6.0.2 setuptools==69.5.1 \
+        tqdm==4.66.4 watchdog==4.0.1 termcolor==2.4.0 Pygments==2.18.0 \
+        setproctitle==1.3.3 pylru==1.2.1 /src/dmoj \
+    && python -m pip wheel --no-build-isolation --no-cache-dir --no-deps \
+        --wheel-dir /wheels /src/ojbench
+
+FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7
+
+RUN apt-get update \
+    && apt-get install --yes --no-install-recommends g++ libseccomp2 pypy3 \
+    && rm -rf /var/lib/apt/lists/* \
+    && useradd --create-home --uid 1000 judge \
+    && mkdir --parents /workspace /problems /opt/ojbench \
+    && chown judge:judge /workspace
+
+COPY --from=builder /wheels /wheels
+COPY --from=builder /src/ojbench/ojbench /opt/ojbench/package/ojbench
+COPY --from=builder /src/ojbench/ojbench/config.yaml /opt/ojbench/config.yaml
+COPY --from=builder /src/ojbench/ojbench/runtime.yaml /opt/ojbench/runtime.yaml
+COPY --from=builder /src/ojbench/LICENSE /usr/share/doc/ojbench/LICENSE
+
+RUN python -m pip install --no-cache-dir /wheels/*.whl \
+    && rm -rf /wheels
+
+ENV OJBENCH_CONFIG_PATH=/opt/ojbench/config.yaml \
+    OJBENCH_RUNTIME_PATH=/opt/ojbench/runtime.yaml \
+    HOME=/home/judge \
+    PYTHONPATH=/opt/ojbench/package \
+    PYTHONUNBUFFERED=1
+
+USER judge
+WORKDIR /workspace
+CMD ["tail", "-f", "/dev/null"]

src/openbench/evals/ojbench/__init__.py

@@ -0,0 +1,5 @@
+"""OJBench tasks."""
+
+from openbench.evals.ojbench.ojbench import ojbench_cpp, ojbench_python
+
+__all__ = ["ojbench_cpp", "ojbench_python"]

src/openbench/evals/ojbench/compose.yaml

@@ -0,0 +1,22 @@
+services:
+  default:
+    build:
+      context: .
+    init: true
+    command: tail -f /dev/null
+    user: "1000:1000"
+    network_mode: none
+    read_only: true
+    volumes:
+      - ${OPENBENCH_OJBENCH_DATA_DIR:?OJBench data cache is not configured}:/problems:ro
+    tmpfs:
+      - /workspace:rw,exec,nosuid,nodev,uid=1000,gid=1000,mode=0700,size=1073741824
+      - /tmp:rw,exec,nosuid,nodev,uid=1000,gid=1000,mode=0700,size=4294967296
+    cap_drop:
+      - ALL
+    cap_add:
+      - SYS_PTRACE
+    security_opt:
+      - no-new-privileges:true
+    pids_limit: 128
+    mem_limit: 3g

src/openbench/evals/ojbench/ojbench.py

@@ -0,0 +1,37 @@
+"""Official OJBench language tracks with DMOJ scoring."""
+
+from pathlib import Path
+from typing import Literal
+
+from inspect_ai import Epochs, Task, task
+from inspect_ai.solver import generate
+
+from openbench.datasets.ojbench import get_ojbench_dataset
+from openbench.scorers.ojbench import ojbench_scorer
+
+COMPOSE_PATH = (Path(__file__).parent / "compose.yaml").resolve()
+
+
+def _ojbench_task(language: Literal["python", "cpp"], total_timeout: int) -> Task:
+    return Task(
+        name=f"ojbench_{language}",
+        dataset=get_ojbench_dataset(language),
+        solver=generate(),
+        scorer=ojbench_scorer(total_timeout=total_timeout),
+        sandbox=("docker", str(COMPOSE_PATH)),
+        epochs=Epochs(8, reducer=["mean", "pass_at_1", "pass_at_8"]),
+    )
+
+
+@task
+def ojbench_python(total_timeout: int = 3600) -> Task:
+    """Evaluate the 232 official Python prompts with eight samples each."""
+
+    return _ojbench_task("python", total_timeout)
+
+
+@task
+def ojbench_cpp(total_timeout: int = 3600) -> Task:
+    """Evaluate the 232 official C++ prompts with eight samples each."""
+
+    return _ojbench_task("cpp", total_timeout)