feat(jail): net-jail — deny-by-default worker egress - #97
Merged
Conversation
…llowlist (WORKER_NET_JAIL) Closes #94. fs-jail confines what a worker can READ. Nothing confined where it could CONNECT, so denying the harness's `webfetch` tool removed a tool while `bash` kept the whole internet: a benchmark rig was one run from a large fake result because the worker could clone the public upstream of the repo it was graded on. Enforcement is the network, not a request to the agent. Each jailed backend gets an `--internal` Docker network (no gateway, no default route, no external DNS) plus a relay container on both that network and a routable one. The relay is registered on the internal network under a network-scoped alias for every allowlisted hostname and verifies the name each connection claims — TLS SNI, or the Host header for cleartext — before forwarding a byte. TLS is spliced, never terminated, so certificate validation stays end-to-end against the real origin. There is no proxy variable, so there is nothing for the worker to unset. The model endpoint is derived from the configured base URL (ANTHROPIC_BASE_URL / OPENAI_BASE_URL / TANGLE_ROUTER_URL, else the CLI's own default), so a caller naming no hosts still gets a working agent. Unenforceable configurations fail loud rather than silently: - a host-executed backend refuses to load, naming each backend and the <NAME>_EXECUTOR=docker that fixes it. No weaker env-proxy mode is offered. - <NAME>_DOCKER_NETWORK alongside a net-jail refuses to load. - a per-request net-jail against a backend with none provisioned, or against execution.kind=sandbox, fails 501 naming the mode. - execution.netJail.allow ASSERTS the enforced allowlist; a mismatch fails rather than quietly widening, since a pooled worker cannot be re-jailed. Provisioning ends by proving the jail on a throwaway container from the same image and network a worker gets: a name that must not resolve, a name that must, and a real TLS handshake through the relay. Also fixes a pre-existing typecheck failure in tests/opencode-worker-timeout. Verified on real Docker: `git clone` exits 128, `curl https://github.com` exits 6, raw-IP egress and a worker-pinned --resolve are both refused, and a real opencode agent turn completes through the allowed endpoint (29,465 prompt tokens). Regression tests execute those same commands rather than asserting on config, and one test removes `--internal` to prove the verification step fails.
…scape `--internal` denies routing OFF the bridge and leaves the bridge itself reachable: Docker configures the host's gateway address on the jail's subnet, on-link in the worker's own network, along with every peer container. Neither is on the allowlist, neither passes the relay, and `verifyNetJail` never looked. Measured on the shipped jail: 339,594 bytes of github.com with ssl_verify_result=0, pulled from a jailed worker through a service listening on the Docker host. Each jailed container now also gets a packet filter in its OWN network namespace — default policy DROP both directions, exceptions for loopback (Docker's embedded resolver) and the relay's newly pinned address. That covers the host gateway, peer containers, the host's other interfaces, link-local metadata and all IPv6 with one rule set instead of an enumeration of things to block. The capability lives in a throwaway sidecar that joins the worker's namespace with CAP_NET_ADMIN and exits; the worker keeps Docker's default capability set, so its own iptables fails with Permission denied as uid 0 and the rules are state it cannot reach. Host DOCKER-USER rules were rejected because that chain hangs off FORWARD and never sees traffic delivered locally to the host; no Docker network option removes the bridge address from a member's reachable set. verifyNetJail now runs its probe under the same filter a worker gets, beside a listening peer container, and FAILS provisioning unless the gateway, the peer and link-local are all unreachable, the worker cannot run iptables, and the relay IS reachable — the control that keeps the four denials from being a dead network stack. Reachability is judged by errno: ECONNREFUSED means a packet came back and counts as a leak. The pool destroys any slot whose filter could not be installed. Requires iptables in the runtime image; the Dockerfile installs it and a backend whose image lacks it refuses to start rather than run a jail that does not deny.
… un-jail a worker
The deny lives inside the container's network namespace, and Docker destroys
and recreates that namespace on every restart — same container id, same
mounts, same filesystem, empty rules. A worker reaches that: when its command
exits non-cleanly the executor runs `docker restart --time 0` on the slot,
because killing the local `docker exec` client does not stop the process tree
inside the container. Measured on the real request path before this commit —
real pool, real spawner — a worker exited 3, and the next acquire of the same
slot dialled the Docker host (`OPEN`) and pulled 339,598 bytes of github.com
through a service listening there.
The fix is not an enumeration of restart sources. `afterCreate` is now
declared to be scoped to a container START rather than to a container, and the
pool enforces that in the two places a start can change:
- No Docker restart policy on a slot that uses the hook. A policy revives the
container in place, with fresh empty namespaces, and nothing in this
process observes it happening. The pool already models a dead slot
correctly (`rm -f` then `provisionSlot`, which re-runs the hook), so a dead
jailed container is REPLACED rather than revived. Nothing depended on the
old policy: the entrypoint is `tail -f /dev/null`, `on-failure` does not
fire on daemon restart, and a stopped container is rebuilt by the pool's
existing liveness path.
- Every handout compares `.State.StartedAt` against the start the hook was
run for, and re-runs the hook before the slot is usable. This is what
covers the restart the bridge performs ITSELF, which is the larger door and
the one the escape came through. `livenessTtlMs` is forced to 0 for such a
pool, because the cached answer is exactly the fact that changes inside the
cached window.
A Docker event watcher was rejected: it races the next `docker exec`, and the
loser of that race is a request running unconfined. The gate has no such
window — the restart is over before it runs, the only thing alive in the
container is the idle entrypoint, and the slot stays reserved across the whole
check. Moving the deny outside the namespace was rejected too: the only place
outside it that is on the path is the host firewall, which is the DOCKER-USER
option already rejected for hanging off FORWARD, and it would have to name the
container by an address Docker may reassign on the very restart it must
survive.
The start is read before AND after the hook and the two must match, so a
container that restarts DURING arming is never stamped with a start whose
namespace the hook never wrote to.
verifyNetJail now proves the property directly instead of proving something
adjacent to it, which is how it missed the previous two escapes. It restarts
the probe with the same `docker restart --time 0` the executor uses, re-arms it
through the same function the pool calls, and re-runs every assertion — host
gateway, peer, link-local, iptables, DNS, TLS. Provisioning fails if the
probe's start time does not move, because a restart that did not happen makes
the second round a re-run of the first. `skipRestartRearmForVerification` is
the calibration seam: with it the guard fires, naming the reachable host.
Owner
Author
Adversarial verification — measured byte egress, real DockerNot a code read. Provisioned a real jail with the production Provenance. image Escape-attempt table
N8 autopsy (why it is 0, not a leak)First pass flagged N8 at ~9,000 B. That was a measurement artifact, run down to ground truth: I timestamped "filter applied" at the call, but
The bytes I first attributed to "post-filter" flowed during the ~2 s sidecar-install window. On the real worker path that window is before slot handout, with only the idle Gates
Verdict: every worker-reachable vector measures 0 bytes off-allowlist. All three prior escapes stay closed. Merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #94. Deny-by-default network egress for jailed workers, sibling to fs-jail (#79). Three adversarial rounds; two real escapes found and closed.
Why
Denying
webfetchremoves a tool. It does not stopbashfrom runningcurlorgit clone. A worker with bash and no network policy has the whole internet regardless of its permission map — and cli-bridge is the local, subscription-backed path, so it is the one we actually use for evals and bulk work.This is not hypothetical: a DeepSWE rig was one run from a large fake result because the worker could
git clonethe repo it was graded against. Every task declaresallow_internet = false; nothing enforced it.Mechanism
An
--internalDocker network plus a relay aliased for each allowlisted hostname, verifying the name each connection claims (TLS SNI, orHostfor cleartext) before forwarding a byte. TLS is spliced, never terminated — certificate validation stays end-to-end and the relay holds no key material. On top, a per-container packet filter installed by a throwawayCAP_NET_ADMINsidecar: default DROP both directions, exceptions only for loopback and the relay.No proxy environment variables, deliberately.
HTTPS_PROXYis a request an agent can decline withunset; here the worker never learns a relay exists (env | grep -ci proxy→ 0), and clients that ignore proxy env — undici/fetch, which these CLIs use — are covered too.Model host is derived from configured base URLs, not hand-typed, with each entry's provenance printed at startup.
The two escapes, and why they matter more than the feature
Round 1 — the Docker host gateway.
--internaldenies routing off the bridge; it does not deny the bridge. The host's gateway address sits on-link in the worker's own subnet, never touches the relay, andverifyNetJailnever looked at it. Measured: 591,738 bytes of github.com withssl_verify_result=0from inside the jail.Round 2 — a restart wipes the namespace. The filter lives in the container's network namespace, and Docker recreates that namespace on restart.
Round 3 found that the round-2 diagnosis named the wrong door.
--restart on-failureis nearly impossible for a worker to trigger —kill -9 1from inside a PID namespace is ignored by the kernel (measured: RestartCount stayed 0, filter intact). The door that actually opened was the bridge's own:terminateDockerExecutionrunsdocker restart --time 0whenever a command exits non-cleanly, because killing the localdocker execclient does not stop the process tree. A worker triggers it by exiting 3. Removing the restart policy alone would have fixed nothing — and that was the proposed fix.Closed by scoping the hook to a container START, not a container: no Docker restart policy on hooked slots, and every handout compares
.State.StartedAtagainst the start the hook ran for, re-arming before the slot is usable.livenessTtlMsis forced to 0, because the cached fact is exactly the fact that changes inside the cached window. A Docker event watcher was rejected: it races the nextdocker exec, and the loser of that race is a request running unconfined.The verifier is now the thing that was wrong twice
verifyNetJailhad missed both escapes by checking something adjacent. It now restarts the probe with the samedocker restart --time 0the executor uses, re-arms through the same code path the pool calls, and re-runs all eight assertions — failing provisioning if the probe's start time does not move, since a restart that did not happen makes the second round a re-run of the first. Reachability is judged by errno:ECONNREFUSEDmeans a packet came back and counts as a leak.A calibration test provisions with the filter suppressed and asserts provisioning rejects. The guard has been watched to fail.
Proof (executed, real Docker)
EHOSTUNREACH169.254.169.254ENETUNREACHEHOSTUNREACHiptablesas uid 0Permission deniedSelf-caught during review
reprovisionSlotdid not copy the new container'sarmedStart, so after any replacement the stale stamp would never match and every request would run an enforcer sidecar forever. Fail-safe rather than unsafe, but real. Fixed and pinned.Gates
tsc 0 · 533 passed / 6 skipped ·
net-jail.test.ts50/50 on real Docker ·docker-executor.test.ts72/72.