Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,27 @@ BRIDGE_BACKENDS=claude,kimi,gemini,sandbox,passthrough
# CLAUDE_DOCKER_OAUTH_MOUNT=share
# CLAUDE_DOCKER_WORKSPACE_ROOT=/tmp/cli-bridge-workspaces
# CLAUDE_DOCKER_NETWORK=research-services
#
# net-jail — deny-by-default EGRESS for worker containers, the network sibling
# of WORKER_FS_JAIL. Workers run on an internal Docker network with no route off
# it and reach only an allowlist that always contains the backend's own model
# endpoint (derived from the configured base URL — you do not name it here).
#
# Docker only. Set alongside a host-executed backend and the bridge REFUSES to
# start: a host-spawned CLI shares this machine's network namespace and any
# proxy variable it is given can be unset, so accepting the setting would report
# a policy that is not applied. Mutually exclusive with <NAME>_DOCKER_NETWORK.
#
# Requires iptables in the runtime image: each jailed container gets a
# deny-by-default packet filter in its own network namespace, without which the
# Docker host stays reachable on-link from inside the jail. Rebuild the image
# (pnpm docker:build:runtime) before turning this on for the first time; a
# backend whose image cannot enforce the filter refuses to start.
#
# WORKER_NET_JAIL=1 # shorthand for BRIDGE_NET_JAIL_MODE=net-jail
# BRIDGE_NET_JAIL_MODE=off|net-jail
# BRIDGE_NET_JAIL_ALLOW=registry.npmjs.org,internal-svc.example:8443
#
# KIMI_EXECUTOR=docker
# KIMI_DOCKER_POOL_SIZE=2
# GEMINI_EXECUTOR=host
Expand Down
141 changes: 141 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,147 @@ Pool workers can join one existing Docker network with
`docker run --network`; it does not create or remove the network. Leave this
unset to retain Docker's default networking behavior.

### net-jail — deny-by-default egress (`WORKER_NET_JAIL`)

`WORKER_FS_JAIL` confines what a worker can READ. `WORKER_NET_JAIL=1` confines
where it can CONNECT: the worker's process tree gets no egress at all except to
an allowlist that always contains the backend's own model endpoint.

```bash
OPENCODE_EXECUTOR=docker
WORKER_NET_JAIL=1
# optional extras beyond the model endpoint, which is derived automatically
BRIDGE_NET_JAIL_ALLOW=registry.npmjs.org,internal-svc.example:8443
```

The gap it closes: denying the harness's `webfetch` tool removes a *tool*. It
does not stop `bash` from running `curl` or `git clone`, so a worker with bash
and no network policy has the whole internet regardless of its permission map.
A benchmark rig was one run from a large fake result because the worker could
clone the public upstream of the repository it was being graded on.

**How it is enforced.** Three layers, and all three are load-bearing.

*The network.* For each jailed backend the bridge creates an `--internal` Docker
network — no default route off it, no external DNS — plus a relay container
attached to both that network and an ordinary one, pinned to a fixed address.
The relay is registered on the internal network under a network-scoped alias for
every allowlisted hostname, and it checks the name each connection claims (TLS
SNI, or the `Host` header for cleartext) before forwarding a byte. TLS is never
terminated: the relay splices raw bytes, so certificate validation still happens
end-to-end against the real origin.

*The filter.* `--internal` denies routing OFF the bridge; it does not deny the
bridge. Docker still configures the host's gateway address on that subnet, and
it is on-link in the worker's own network — as is every other container there.
A jailed worker used exactly that to pull 339,594 bytes of github.com with a
valid certificate chain, through a service listening on the Docker host. So each
jailed container also gets a packet filter in its OWN network namespace: default
policy DROP in both directions, and exactly two exceptions — loopback (Docker's
embedded resolver lives there) and the relay's pinned address. The host gateway,
peer containers, the host's other interfaces, link-local metadata addresses
(`169.254.0.0/16`) and all IPv6 are rejected by the kernel.

The rules are written by a throwaway sidecar that joins the worker's network
namespace with `CAP_NET_ADMIN` and exits. The worker itself never holds that
capability, so `iptables` inside it fails with `Permission denied` even as uid
0, and the filter outlives the sidecar because it is state of the namespace. A
slot whose filter cannot be installed is destroyed rather than served. This
needs `iptables` in the runtime image; `docker/Dockerfile.cli-runtime` installs
it, and a net-jailed backend refuses to start without it.

*Surviving a restart.* A filter is state of a network namespace, and Docker
destroys and recreates that namespace every time a container restarts — same
container id, same mounts, same filesystem, empty rules. A worker can cause the
restart: when a worker's command exits non-cleanly the executor runs `docker
restart` on its slot, because killing the local `docker exec` client does not
stop the process tree inside the container. Before this was closed, exiting
non-zero and then asking for one more turn pulled 339,598 bytes of github.com
out of a real pool slot through the Docker host. So the filter is treated as
belonging to a container START rather than to a container: a jailed slot carries
**no Docker restart policy** (a dead one is replaced, which reinstalls the
filter, instead of being revived in place), and every handout compares the
container's `StartedAt` against the start the filter was installed for and
reinstalls it before the slot can be used. The window in between contains
nothing but the idle `tail -f /dev/null` entrypoint, and the slot stays reserved
across it, so no request is ever dispatched into an unfiltered namespace. A
container whose filter cannot be reinstalled is replaced, not served.

There is deliberately no proxy environment variable. `HTTPS_PROXY` is a request
an agent can decline with `unset`; here the worker never learns a relay exists,
its clients dial what they believe is the origin, and clearing every variable in
its environment changes nothing about what it can reach.

**The model endpoint is derived, not typed.** The host and port come from the
configured base URL (`ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`,
`TANGLE_ROUTER_URL`, …), falling back to the CLI's own default endpoint when
none is set. A caller asking for a net-jail names no hosts and still gets a
working agent. Multi-provider CLIs (opencode, pi) have no single default; set a
base URL or name the endpoint in `BRIDGE_NET_JAIL_ALLOW`, and the bridge refuses
to start rather than hand back a jail that denies its own model.

**Docker only, and it says so.** A CLI spawned on the host shares this machine's
network namespace, and there is no unprivileged mechanism that gives it a
partial egress policy. So the bridge REFUSES to start when `WORKER_NET_JAIL` is
set alongside a host-executed backend, naming each backend and the
`<NAME>_EXECUTOR=docker` that fixes it. No weaker env-proxy fallback is offered:
a control that can be turned off by the thing it constrains is not isolation,
and shipping it under the same name is how the leak above survived.

Startup provisioning ends by PROVING the jail on a throwaway container drawn
from the same image, the same network AND the same filter a worker gets, with a
listening peer container beside it on the bridge:

| Proven at provisioning | Why it is checked |
| --- | --- |
| no default route | the network denies routing off the bridge |
| the Docker host's gateway is unreachable | an address that answers on the worker's own subnet is a next hop that never passes the relay |
| a peer container is unreachable while listening | the relay must be the only next hop, not merely the only *external* one |
| link-local is unreachable | cloud instance metadata, and the credentials it hands out, live at `169.254.169.254` |
| the worker cannot run `iptables` | a worker that can flush the rules is not confined by them |
| the relay IS reachable | the control: a container with a dead network stack denies everything and proves nothing |
| a non-allowlisted name does not resolve | DNS is scoped to the allowlist |
| an allowlisted name resolves and completes a real TLS handshake | the jail does not deny its own model |
| **every line above, again, after restarting the probe** | a restart empties the namespace, and a worker can cause one by exiting non-zero |

Failing any one of these is fatal. Reachability is judged by the ERROR, not by
whether something answered: `ECONNREFUSED` means a packet came back, which
proves the address is reachable and merely silent on that port, so it counts as
a leak.

The restart round is run with the same `docker restart --time 0` the executor
uses to kill a worker's process tree, and the filter is put back by the same
function the pool's slot provisioning calls — verifying through a second
implementation would prove a jail no worker runs in. Provisioning also fails if
the probe's start time does not move, because a restart that did not happen
makes the second round a re-run of the first.

Each of the two escapes found so far was found because the verifier checked
something adjacent to it. The first version checked only the route table, the
two DNS facts and the handshake, and passed the jail the host-gateway escape was
performed on. The second checked the filter's presence at one instant, and
passed a jail that a worker could empty by exiting non-zero. Each round's fix is
therefore added here as a direct probe of the escape itself.

Per request, `execution.netJail` mirrors `execution.jail`:

```jsonc
{
"execution": {
"kind": "host",
"netJail": { "mode": "net-jail", "allow": ["router.tangle.tools:443"] }
}
}
```

`mode` follows the same floor rule as the fs-jail: the env setting can be raised
by a request, never lowered. `allow` ASSERTS the enforced allowlist rather than
changing it — a pooled worker joined its network when the bridge started and
cannot be re-jailed per request, so a list that differs from what is in force
fails with 501 instead of silently widening. A net-jail requested of a backend
with none provisioned, or of `execution.kind=sandbox`, fails the same way and
names the mode. Nothing is ever accepted and then not applied.

### Topology guide

cli-bridge spawns pool containers by talking to the **host docker
Expand Down
8 changes: 7 additions & 1 deletion docker/Dockerfile.cli-runtime
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,15 @@ FROM node:22-slim

ARG CLI_BRIDGE_HARNESSES=all

# iptables is here for WORKER_NET_JAIL. The jail installs a deny-by-default
# egress filter into each worker's network namespace, written by a throwaway
# sidecar that runs THIS image with CAP_NET_ADMIN and exits; the workers never
# hold that capability and cannot undo the rules. Without iptables in the image
# a net-jailed backend refuses to start rather than run a jail that does not
# deny, so this package is a hard requirement of that mode, not an extra.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git ca-certificates dumb-init curl python3 \
git ca-certificates dumb-init curl python3 iptables \
&& rm -rf /var/lib/apt/lists/*

# --- CLI installs (latest at image-build time) ---
Expand Down
23 changes: 23 additions & 0 deletions src/backends/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,24 @@ export interface ProfileMaterializationReceipt {
unsupported: Array<{ dimension: string; reason: string }>
}

/**
* Per-request net-jail requirement — the network sibling of `execution.jail`.
*
* `mode: 'net-jail'` requires deny-by-default egress for the worker process
* tree, with an allowlist that always contains the backend's own model
* endpoint. `allow` ASSERTS the exact enforced `host:port` list rather than
* changing it, because a pooled worker joined its network when the bridge
* started and cannot be re-jailed per request.
*
* No resolved form travels onward to the backends the way `jailSpec` does:
* enforcement lives in the network the container was created on, so the chat
* route either proves the jail is in force or fails the request.
*/
export interface NetJailRequest {
mode?: 'off' | 'net-jail'
allow?: string[]
}

export interface ChatRequest {
model: string
messages: ChatMessage[]
Expand Down Expand Up @@ -155,13 +173,18 @@ export interface ChatRequest {
* {@link JailSpec} on `jailSpec` by the chat route.
*/
jail?: { mode?: 'off' | 'write-jail'; root?: string }
netJail?: NetJailRequest
}
| {
kind: 'sandbox'
repoUrl?: string
gitRef?: string
capability?: string
ttlSeconds?: number
// Declared on this variant too so a net-jail asked of the sandbox mode
// reaches the gate and is REFUSED by name. Dropping it from the type
// would make the request parse and the requirement disappear.
netJail?: NetJailRequest
}
/**
* Resolved write-jail spec for this turn, set by the chat route from
Expand Down
82 changes: 81 additions & 1 deletion src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import { realpathSync, statSync } from 'node:fs'
import { isAbsolute, join, parse, relative, resolve, sep } from 'node:path'
import { assertDockerNetworkName } from './executors/docker-network.js'
import { formatAllowEntry, parseAllowList } from './jail/net-allowlist.js'

export interface Config {
host: string
Expand Down Expand Up @@ -102,6 +103,27 @@ export interface Config {
* `execution.jail.root` overrides this.
*/
jailRoot: string | null
/**
* Default net-jail mode, from `BRIDGE_NET_JAIL_MODE` (off|net-jail, default
* off; `WORKER_NET_JAIL=1` is a shorthand for net-jail). Sibling of
* {@link Config.jailMode}: fs-jail confines what a worker can READ, net-jail
* confines where it can CONNECT. In `net-jail` every worker container runs on
* an internal Docker network with no route off it, and reaches only an
* allowlist that always contains its own model endpoint.
*
* Docker-only, by construction. A CLI spawned on the host shares the host's
* network namespace and there is no unprivileged way to give it a partial
* egress policy, so this setting REFUSES to load alongside a host-executed
* backend rather than reporting a policy it cannot apply.
*/
netJailMode: 'off' | 'net-jail'
/**
* Extra `host` or `host:port` tokens added to every net-jail allowlist, from
* `BRIDGE_NET_JAIL_ALLOW`. The model endpoint is derived automatically and
* does not belong here; this is for the rest (a package registry, an internal
* service) that a specific workload legitimately needs.
*/
netJailAllow: string[]
}

export interface BackendExecutorConfig {
Expand Down Expand Up @@ -204,6 +226,9 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {
}

const defaultTimeout = Number.parseInt(env.CLI_TIMEOUT_MS ?? '300000', 10)
const executors = parseAllExecutors(env, dataDir)
const netJailMode = parseNetJailMode(env)
if (netJailMode !== 'off') assertNetJailEnforceable(backends, executors)

return {
host,
Expand Down Expand Up @@ -244,9 +269,13 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {
sandboxApiKey: env.SANDBOX_API_KEY?.trim() || null,
sandboxProfilesDir: resolve(env.SANDBOX_PROFILES_DIR ?? './profiles'),
sandboxTimeoutMs: Number.parseInt(env.SANDBOX_TIMEOUT_MS ?? '300000', 10),
executors: parseAllExecutors(env, dataDir),
executors,
jailMode: parseJailMode(env.BRIDGE_JAIL_MODE),
jailRoot: env.BRIDGE_JAIL_ROOT?.trim() || null,
netJailMode,
// Parsed (not merely split) so an unusable token is rejected at startup
// rather than at provisioning time, when it would read as a Docker failure.
netJailAllow: parseAllowList(env.BRIDGE_NET_JAIL_ALLOW, 'BRIDGE_NET_JAIL_ALLOW').map(formatAllowEntry),
}
}

Expand All @@ -256,6 +285,57 @@ function parseJailMode(value: string | undefined): 'off' | 'write-jail' | 'fs-ja
throw new Error(`invalid BRIDGE_JAIL_MODE: ${value} — expected off|write-jail|fs-jail`)
}

function parseNetJailMode(env: NodeJS.ProcessEnv): 'off' | 'net-jail' {
const value = env.BRIDGE_NET_JAIL_MODE
if (value !== undefined && value !== '' && value !== 'off' && value !== 'net-jail') {
throw new Error(`invalid BRIDGE_NET_JAIL_MODE: ${value} — expected off|net-jail`)
}
const shorthand = ['1', 'true', 'yes', 'on'].includes((env.WORKER_NET_JAIL ?? '').trim().toLowerCase())
return value === 'net-jail' || shorthand ? 'net-jail' : 'off'
}

/**
* Refuse to load a net-jail the bridge cannot enforce.
*
* Two contradictions, both fatal rather than warned:
*
* - A backend that spawns its CLI on the host. That process shares the host's
* network namespace; there is no unprivileged mechanism that gives it an
* egress allowlist, and a proxy variable it can `unset` is not one. Booting
* with the setting accepted would report a network policy to every caller
* while enforcing nothing — the failure this feature exists to remove.
* - A backend pinned to an operator's own `<NAME>_DOCKER_NETWORK`. A worker
* joins exactly one network at creation, and joining a routable one is
* indistinguishable from having no jail at all.
*/
function assertNetJailEnforceable(
backends: Set<string>,
executors: Record<string, BackendExecutorConfig>,
): void {
const hostSpawned = [...backends].filter(
(name) => !NON_HOST_SPAWN_BACKENDS.has(name) && executors[name]?.kind !== 'docker',
)
if (hostSpawned.length > 0) {
throw new Error(
`net-jail is enabled (BRIDGE_NET_JAIL_MODE / WORKER_NET_JAIL) but ${hostSpawned.join(', ')} ` +
`${hostSpawned.length === 1 ? 'runs' : 'run'} on the host execution mode, which cannot enforce it: a ` +
'host-spawned CLI shares this machine\'s network namespace and any proxy variable it is given can be ' +
'unset. Set ' + hostSpawned.map((n) => `${n.toUpperCase()}_EXECUTOR=docker`).join(' / ') +
', drop those backends from BRIDGE_BACKENDS, or unset the net-jail. Refusing to start with a network ' +
'policy that would not be applied.',
)
}
for (const cfg of Object.values(executors)) {
if (cfg.kind === 'docker' && cfg.network && backends.has(cfg.name)) {
throw new Error(
`net-jail is enabled but ${cfg.name.toUpperCase()}_DOCKER_NETWORK=${cfg.network} pins its workers to a ` +
'routable network. A container joins one network at creation, so the two settings cannot both hold. ' +
`Unset ${cfg.name.toUpperCase()}_DOCKER_NETWORK, or unset the net-jail.`,
)
}
}
}

function parsePositiveInt(value: string | undefined, fallback: number): number {
if (value === undefined || value === '') return fallback
const parsed = Number.parseInt(value, 10)
Expand Down
Loading