Skip to content

Commit 5b9fe4f

Browse files
ZixiaoLtedli-cmdclauderichard-epsilla
authored
feat(harness): add goose as a new agent backend (#164)
* feat(harness): add goose as a new agent backend Wires aaif-goose/goose (Apache-2.0) in end to end as a new "goose" backend, alongside the existing claude/codex/hermes/pi/dsh/opencode/qwen/gemini/cline/omp set. goose is a single prebuilt Rust binary fetched from GitHub releases, so it installs the way opencode's does — the release asset directly, not the vendor's download_cli.sh, which installs to a path outside the data volume. Pinned to 1.50.0: upstream ships weekly (1.49.0 -> 1.50.0 in five days) and every behavior below was read out of that release's own source. Unlike omp there is no SHA256SUMS to verify against, so the pinned tag is the whole of the guarantee, and the installer says so. It reaches models as cline and qwen do — one OpenAI chat/completions client pointed at the loopback relay — so it is wired to the eight integrations that serve that shape and needs no provider of its own. One genuine adapter difference: goose's OpenAI provider takes the endpoint in TWO pieces (OPENAI_HOST + OPENAI_BASE_PATH), so the relay's single base_url is split here rather than teaching the relay a second shape; pasting it whole would send every turn to /v1/v1/chat/completions, a 404 with no body. Everything goose keeps — config, the sessions database, the harness's skills — hangs off GOOSE_PATH_ROOT pointed inside .harness/, which is what makes the conversation travel in the checkpoint and keeps goose's own files out of the produced-file list. Not a passthrough: the stream-json schema is its own, and it carries TWO serde rename rules in one stream — snake_case event tags (message/notification/error/ complete) but camelCase content-block tags (toolRequest/toolResponse). The tool call sits two levels down under toolCall.value, and the Ok/Err discriminator is Rust's Result rendered as {status, value|error} with a STRING error. These came off the v1.50.0 source, not the docs, which name the event types and publish no field-level example — the same gap that made the first gemini normalizer guess wrong and render every tool call as a content-free row. A regression test pins each one. Two findings the source reading produced before a line was written: - goose emits cache_write_input_tokens in snake_case, which _norm_token_usage's picker does not know (it carries only the camel spelling), so the count would have billed as zero. Handled in a goose-side wrapper, the way _gemini_usage handles gemini's `cached`. - disabled tools are a HARD block here, and it survives headless approval. GOOSE_MODE=auto short-circuits the permission table, so permission.yaml's NeverAllow is not the lever; available_tools is, because fetch_all_tools gates on it while BUILDING the model's tool list. UHP 4.3 requires a hard block to be reported as one wherever the runtime supports it, so the base declares "hard" and stays out of the standing-instruction set. The extensions map written to config.yaml is deliberately PARTIAL: goose seeds every platform extension it does not find there from that definition's own default_enabled, so naming only what we restrict or add leaves Skills, Todo and Analyze alone. Writing a complete map would have switched them off — and a harness whose skills silently stopped loading still passes every routing scenario. Live-verified on a self-hosted instance against a Vercel AI Gateway connection (2026-09-12): all six catalog ids completed a first turn; gpt-5.4 also completed follow-up, a switch to claude-sonnet-4.6 and back, an artifact turn whose produced file matched the rendered card byte for byte, and a turn after a forced sandbox recycle that still recalled the first message and re-read the file. The custom-harness dimension passes all six of its claims: the skill bundle reached the agent (the answer carried a token that exists only inside its script), the script actually ran (it left its file among the produced files), the disabled tool was genuinely absent, and a declared MCP server was stored and called. Half of the catalog bar is met and the missing half is named rather than papered over: these turns are not substitution-checked, because on this backend they cannot be. The served model would have to come from metadata.inference. resolvedModel, and only the databricks provider format populates it — the openai path every turn takes never sets it. served_model is therefore always empty for goose, and the catalog comment says to read these rows as "the id completed a turn", not "the id served it". docs/harness-verification.md's registration list is expanded from five places to the sixteen that actually exist, since every one of them fails silently when missed; the console's backend union type is the only one a compiler catches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(runner): the relay reports the served model for CLIs that do not; goose's list is the full chat-shaped set, to be trimmed by the column The relay sees every provider answer, so it reads the first "model" off the bytes as they pass and the run loop stamps it on a result event the CLI left unlabelled (goose never reports one; cline and qwen do not either). goose's catalog list becomes every id the wired chat-shaped providers can serve, for the goose column to measure with the served model read by the relay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk * feat(gateway): goose's list is the 35 ids the column measured; claude-opus-5 stays out Measured 2026-09-12 on TokenRouter, Vercel, OpenRouter, OpenAI, Azure, Anthropic and the hosted door, served model read by the relay, no substitution. claude-opus-5 answers with an empty response on every tool-using turn through goose's chat wire, its own vendor included. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk * docs(matrix): the goose columns on seven providers, the served model read by the relay Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk * feat(gateway): goose's candidate list is the union of the wired providers' tables (five more ids to measure) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk * docs(matrix): every provider table lists the pairs it serves that a harness did not run, with the reason The Responses-only ids and the chat-only harnesses are now gateway constants (the chat-only test reads them); the renderer derives each column's not-run pairs from the gateway's provider tables and harness catalogs, so a model a harness cannot run on a provider is stated as a limitation rather than absent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk * fix(goose): fail the turn when the provider error arrives as prose Review of #164, all three items. A provider error ends the turn as "completed" with the error as its answer. goose does not emit StreamEvent::Error when the upstream fails mid-turn: the agent loop pushes an assistant message reading "Ran into this error: <the provider's sentence>" and then ends the run normally with `complete`. The normaliser read that as the answer, so the turn recorded status: completed — measured on TokenRouter's 503 during the support-matrix column — and the next turn in the session answered from a history carrying the prose. _GOOSE_ERR_RE now catches both prefixes the pinned 1.50.0 binary carries ("Ran into this error: " and "Ran into this error trying to compact: ", confirmed with strings, the only two it holds). A matching assistant block sets _goose_error instead of `final` and is not rendered, so `complete` yields subtype error with the provider's sentence as the reason — the same treatment _CLAUDE_ERR_RE gives claude's injected "API Error: …". _goose_eof now prefers a reported error over earlier text, for the same reason on the abnormal path. Four tests pin it, including that a turn merely TALKING about an error still passes. Pin the archives, not just the tag. Upstream publishes no SHA256SUMS, so the digests are pinned here instead, computed from the v1.50.0 release assets and verified against the download (both digests match the ones given in review). A tag can be moved and an asset re-uploaded; a digest cannot. HR_GOOSE_VERSION overriding the pin takes HR_GOOSE_SHA256 for the version chosen, or logs that the archive is UNVERIFIED — never a silent skip. Exercised on the real release for all three paths: match, mismatch, override. _goose_has_session asks the database instead of searching its bytes. The old substring search was only as good as the name was distinctive, and `harness` also occurs in the workspace path (/data/workspaces/<sid>/.harness/...) that tool output and the AGENTS.md contract carry — so a database holding one message and no session could match, add -r, and fail the turn with "No session found": the exact failure the check exists to prevent. It is now SELECT 1 FROM sessions WHERE name = ? OR id = ?, read-only, which is what the CLI itself matches on. The WAL clause goes away because sqlite reads the WAL itself — pinned with an uncheckpointed WAL so it cannot pass for the wrong reason. A schema drift on a pin bump fails the test rather than a live turn. docs/harness-verification.md gains both lessons for the next harness: check how the CLI reports a provider failure rather than assuming it has an error event, and verify the install, since a pinned tag is not verification. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(goose): the catalog says which ids the column ran, and which it has not Taking the review mirror's five commits (the relay served-model reader, the measured list, the matrix columns). Three things they left inconsistent: The block comment above goose's catalog entry still said served_model is always empty for goose, that the rows are NOT substitution-checked "because on this backend they cannot be", and that the list is six ids from a first sweep. The relay change in 8f2affd is exactly what makes that false, and the column measured zero substitutions across seven providers. It now says where the served model comes from and that the rows mean "the id served it". The list ships 40 ids; the goose column ran 36 and 35 shipped. hunyuan-3, ling-3.0-flash, minimax-m3, nemotron-3-ultra and qwen3.7-flash have no goose row in support-matrix-results.json, and docs/support-matrix.md already lists them as "not run in this column" — but the comment above the list said every id in it was measured on 2026-09-12. Keeping them is the pi precedent (ids offered so the matrix can measure them), so they stay and are labelled as what they are. Whether to ship them before a column runs is yours to say; the code no longer claims they were measured either way. docs/harness-verification.md: rule 2 says how the served model is judged but not where it comes from, which just changed for three harnesses. The question to ask of a new harness is now "does it ride the relay", not "does the CLI report a served model". Two supplements to the relay reader itself: - _served_model_in reads within one 4096-byte read, so a boundary falling inside "model":"…" loses it, silently — served_model stays empty and the turn is simply not substitution-checked. The SSE loop now carries the previous chunk's tail; one test pins it with a real split. - _relay_served_model takes the FIRST hr-relay- bearer in env as the turn's route. That is right today only because the one place that registers a second route (hermes' auxiliary vision model) runs on _run_hermes_bg, which never calls this. Written down, because reading the vision route would stamp the image model as the served model — a substitution finding invented out of nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Zixiao Li <ted.li.job@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: richard-epsilla <richard@epsilla.com> Co-authored-by: richard-epsilla <131846445+richard-epsilla@users.noreply.github.com>
1 parent 8a4c010 commit 5b9fe4f

16 files changed

Lines changed: 15394 additions & 1100 deletions

docker/entrypoint.sh

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ export HOSTNAME=0.0.0.0
177177
TOOLS="$DATA_DIR/agent-tools"
178178
export PATH="$TOOLS/bin:$PATH"
179179
export NODE_PATH="$TOOLS/lib/node_modules"
180-
export HR_BACKENDS="${HR_BACKENDS:-claude,codex,hermes,pi,dsh,opencode,qwen,gemini,cline,omp}"
180+
export HR_BACKENDS="${HR_BACKENDS:-claude,codex,hermes,pi,dsh,opencode,qwen,gemini,cline,omp,goose}"
181181

182182
wanted() { [[ ",$HR_BACKENDS," == *",$1,"* ]]; }
183183
# The executable IS the definition of "installed" — an installer that exits 0 without producing
@@ -195,6 +195,7 @@ backend_bin() {
195195
gemini) echo "$TOOLS/bin/gemini" ;;
196196
cline) echo "$TOOLS/bin/cline" ;;
197197
omp) echo "$TOOLS/bin/omp" ;;
198+
goose) echo "$TOOLS/bin/goose" ;;
198199
esac
199200
}
200201

@@ -223,6 +224,57 @@ install_opencode() {
223224
rm -rf "$oc_tmp"
224225
}
225226

227+
# goose (Apache-2.0) ships one prebuilt binary per target on GitHub releases — the archive holds
228+
# exactly ./goose — so it is fetched the same way opencode's is, and NOT through the vendor's
229+
# download_cli.sh: piping a remote script into a shell inside an entrypoint is a supply-chain
230+
# surface this product does not need, and that script installs to its own path anyway.
231+
#
232+
# Pinned EXACTLY, like every other backend here. Upstream ships weekly (1.49.0 -> 1.50.0 in five
233+
# days) and 1.50.0 is the release runner/server.py's goose code was written against: the
234+
# stream-json event schema, GOOSE_PATH_ROOT, the available_tools allowlist and -n/-r resume were
235+
# all read out of THIS version's source. A silent `latest` re-gambles all of it.
236+
#
237+
# Note the repo moved from block/goose to aaif-goose/goose. Unlike omp this project publishes no
238+
# SHA256SUMS.txt, so there is nothing to fetch and compare against — but that does not mean the
239+
# download has to go unverified: the digests are pinned HERE instead, computed from the v1.50.0
240+
# release assets (each archive holds exactly ./goose). That is strictly stronger than the tag
241+
# alone, which can be moved and whose asset can be re-uploaded.
242+
install_goose() {
243+
case "$(uname -m)" in
244+
x86_64) gs_arch="x86_64"; gs_sha="6389eea4440178de006fa148d466ac411021315ff7f72b1014beae2d445851e2" ;;
245+
aarch64|arm64) gs_arch="aarch64"; gs_sha="febd71a6a25c3aff7dbcf566f78d2864e87d886c33c4ef1fee2f67fedd334063" ;;
246+
*) echo "unsupported architecture $(uname -m) for goose"; return 1 ;;
247+
esac
248+
gs_ver="${HR_GOOSE_VERSION:-1.50.0}"; gs_ver="${gs_ver#v}"
249+
if [ "$gs_ver" != "1.50.0" ]; then
250+
# The pinned digests describe 1.50.0 and nothing else. An operator overriding the version
251+
# supplies the digest for the version they chose, or is TOLD the download is unverified —
252+
# silently skipping the check while the code above advertises one is the dishonest option.
253+
if [ -n "${HR_GOOSE_SHA256:-}" ]; then
254+
gs_sha="$HR_GOOSE_SHA256"
255+
else
256+
echo "[harnessrouter] WARN: HR_GOOSE_VERSION=$gs_ver overrides the pinned 1.50.0, and no"
257+
echo "[harnessrouter] HR_GOOSE_SHA256 was given — this goose archive is UNVERIFIED."
258+
gs_sha=""
259+
fi
260+
fi
261+
gs_url="https://github.com/aaif-goose/goose/releases/download/v${gs_ver}/goose-${gs_arch}-unknown-linux-gnu.tar.gz"
262+
gs_tmp="$(mktemp -d)"
263+
curl -fsSL "$gs_url" -o "$gs_tmp/goose.tar.gz" || { rm -rf "$gs_tmp"; return 1; }
264+
if [ -n "$gs_sha" ]; then
265+
gs_have="$(sha256sum "$gs_tmp/goose.tar.gz" | awk '{print $1}')"
266+
if [ "$gs_sha" != "$gs_have" ]; then
267+
echo "goose $gs_ver: archive digest mismatch for $gs_arch (want $gs_sha, have $gs_have)"
268+
rm -rf "$gs_tmp"; return 1
269+
fi
270+
fi
271+
tar -xzf "$gs_tmp/goose.tar.gz" -C "$gs_tmp" || { rm -rf "$gs_tmp"; return 1; }
272+
[ -f "$gs_tmp/goose" ] || { echo "release archive contained no goose binary"; rm -rf "$gs_tmp"; return 1; }
273+
mkdir -p "$TOOLS/bin" && install -m 755 "$gs_tmp/goose" "$TOOLS/bin/goose" \
274+
|| { rm -rf "$gs_tmp"; return 1; }
275+
rm -rf "$gs_tmp"
276+
}
277+
226278
# omp (Oh My Pi, MIT) ships standalone prebuilt binaries on GitHub releases, with a SHA256SUMS.txt
227279
# beside them. Pinned EXACTLY, like every other backend here: upstream releases almost daily
228280
# (18.1.8 through 18.1.13 in five days), and 18.1.13 is the release the runner's omp code was
@@ -328,6 +380,11 @@ install_backends() {
328380
try_install "Oh My Pi" install_omp || true
329381
fi
330382

383+
if wanted goose && [ ! -x "$(backend_bin goose)" ]; then
384+
echo "[harnessrouter] installing goose (Apache-2.0)…"
385+
try_install "goose" install_goose || true
386+
fi
387+
331388
# The dsh venv lives on the data volume, so a pin bump in the image must reach a volume that
332389
# already has a venv: dsh-ready names the version it was built for, and a mismatch rebuilds
333390
# the venv (sessions live under ~/.dsh, not in it). The 0.1.0rc7 -> 0.1.2rc1 move is where

docs/harness-verification.md

Lines changed: 71 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,16 @@ number or tier under any prefix (`google/gemini-3-flash-preview` for `gemini-3.8
3838
`gemini-2.5-flash-lite` for `gemini-2.5-flash`) is another model, and stays a finding.
3939
Enforced in `scripts/support-matrix/samemodel.py`, pinned by `test_samemodel.py`.
4040

41+
Where the served model comes from is the CLI's choice, and some report none: goose reports one only
42+
on a provider format it never uses, and cline and qwen report none either. That used to make this
43+
rule unenforceable for those three — half the bar, on three harnesses. It no longer is. Every turn
44+
on those backends rides the loopback relay and the provider's own answer names `model`, so the
45+
relay reads it off the bytes as they pass and the runner stamps it on a result the CLI left
46+
unlabelled (`_served_model_in` and `_relay_served_model` in `runner/server.py`, pinned by
47+
`runner/tests/test_relay_served_model.py`); a CLI that reports its own keeps it. So the question to
48+
ask of a new harness is not "does this CLI report a served model" but "does it ride the relay" — if
49+
it does, rule 2 applies whether the CLI cooperates or not.
50+
4151
**3. What the reader sees is what was stored.** The artifact row requires the rendered file cards to
4252
BE the turn's stored files: same names, same count. Asking only whether some card carried the
4353
expected name let a file rendered twice pass as a produced artifact for months.
@@ -108,16 +118,67 @@ egress, and any other URL overrides the default.
108118

109119
## Adding a harness
110120

111-
A new harness is registered in five places, and is not finished until it has a measured column:
112-
113-
1. `runner/server.py`, `BACKENDS`: its providers, default model, and the normaliser that turns its
114-
output into the shape the gateway stores.
115-
2. `gateway/app.py`, `_MODEL_CATALOG`: the ids it offers, honestly (see rule 4), and
116-
`_INTEGRATION_WIRING`: which provider types can drive it.
117-
3. `ui/src/lib/harness.ts`, `OOB`: the built-in harness the console lists, with its mark and models.
118-
4. `docker/entrypoint.sh`, how the CLI is installed on first start, under its own licence, pinned.
119-
5. `docs/support-matrix.md`, a column per provider, produced by the suite, with its notes in
120-
`docs/support-matrix-notes.md`.
121+
A new harness is registered in the places below, and is not finished until it has a measured
122+
column. Every one of them fails SILENTLY when missed — the harness still builds, still answers, and
123+
loses one capability without saying so. The single exception is the console's backend union type,
124+
which fails the type-check, and that is the only one a compiler will find for you.
125+
126+
**The runner**`runner/server.py`:
127+
128+
1. `BACKENDS`: its providers, default model, and the normaliser that turns its output into the
129+
shape the gateway stores.
130+
2. The `turn()` dispatch branch that builds its argv.
131+
3. `_write_skills`: where a skill bundle has to land for THAT CLI's loader to find it. Prefer a
132+
path under `.harness/`: the workspace root is collected as produced files, so a skills folder
133+
written there is handed back to the user as a deliverable on every turn.
134+
4. `_agent_doc_path`: `AGENTS.md` or `CLAUDE.md` — whichever the CLI actually reads. Getting it
135+
wrong writes the file and the agent never sees it, so the workspace contract never arrives.
136+
5. `_resume_lost`: how a turn that could not continue the conversation says so, if this CLI can
137+
lose one. Silence here reads as a completed turn that has forgotten everything.
138+
6. **How this CLI reports a provider failure — check, do not assume it has an error event.**
139+
Several narrate it as ordinary assistant text and then end the run normally: claude injects
140+
`API Error: …`, goose `Ran into this error: …`. Read as written, the turn completes with the
141+
failure as its answer, the run counts as a pass, and the next turn in that session answers from
142+
a history carrying the error prose. The normaliser must recognise the CLI's prefix, fail the
143+
turn, and keep the sentence as the reason — `_CLAUDE_ERR_RE` and `_GOOSE_ERR_RE`. Pin the
144+
prefix with a test, and take it from the pinned binary (`strings`), not from the docs.
145+
7. `CHECKPOINT_EXCLUDE` and `_git_ensure`'s ignore list: any file the CLI writes that can hold a
146+
credential.
147+
148+
**The gateway**`gateway/app.py`:
149+
150+
8. `_MODEL_CATALOG`: the ids it offers, honestly (see rule 4).
151+
9. `_BASE_CATALOG`: label, system prompt, the tool list **in the CLI's own tool names**, and
152+
`tool_enforcement``hard` only where the runtime really can withhold a tool, per UHP §4.3,
153+
which forbids both overstating and understating it. A tool id that matches nothing is dropped
154+
on the way through and disables nothing while the console reports it as off.
155+
10. `_INTEGRATION_WIRING`: which provider types can drive it.
156+
11. `_CUSTOM_FORMAT_BACKENDS`: which custom endpoint formats it can actually speak.
157+
158+
`_HID_PREFIX_RE`, `_backend_of_builtin` and `_backend_of_harness` are DERIVED from `_BASE_CATALOG`
159+
and need no edit — they were hand-written lists once, and each silently missed a base.
160+
161+
**The console** — and note only the first of these is caught by the type-check:
162+
163+
12. `ui/src/lib/harness.ts`: the `backend` union type, and `OOB`, the built-in harness the console
164+
lists with its mark and models.
165+
13. `ui/src/components/HarnessLogo.tsx`: its brand mark, if there is one. A mapping to a file that
166+
does not exist renders a broken image — worse than the generic glyph it falls back to.
167+
14. `ui/src/components/HarnessSettings.tsx`: the instruction-file label, which must agree with
168+
`_agent_doc_path` or the settings page names a file the runner does not write.
169+
170+
**Install and measurement:**
171+
172+
15. `docker/entrypoint.sh`: `HR_BACKENDS`, `backend_bin`, an installer, and its call in
173+
`install_backends` — the CLI installed on first start, under its own licence, pinned exactly
174+
and VERIFIED. A pinned tag is not verification: a tag can be moved and a release asset
175+
re-uploaded. Where upstream publishes checksums, fetch and compare them (`install_omp`); where
176+
it does not, compute the digests yourself and pin them beside the version (`install_goose`).
177+
An override that skips the check must say so in the log rather than skip it quietly.
178+
16. `scripts/support-matrix/custom-harness.mjs`, `BASES`: otherwise the custom-harness dimension
179+
never measures this harness at all.
180+
17. `docs/support-matrix.md`, a column per provider, produced by the suite, with its notes in
181+
`docs/support-matrix-notes.md`.
121182

122183
Its models must be honest: every id the picker offers must run as itself, and a turn that ran on a
123184
different model fails rather than quietly succeeding.

docs/self-hosting-guide.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -156,9 +156,10 @@ Wait for `ready on :3000`, then open the browser:
156156
[harnessrouter] installing Codex (Apache-2.0)…
157157
[harnessrouter] installing Pi (MIT) and its MCP adapter (MIT)…
158158
[harnessrouter] installing Oh My Pi (MIT)…
159+
[harnessrouter] installing goose (Apache-2.0)…
159160
[harnessrouter] installing DeepSeek Harness (MIT, developer preview — version-pinned)…
160161
[harnessrouter] installing Hermes (check its upstream license before use)…
161-
[harnessrouter] data=/data backends available: claude codex hermes pi dsh opencode qwen gemini cline omp
162+
[harnessrouter] data=/data backends available: claude codex hermes pi dsh opencode qwen gemini cline omp goose
162163
[harnessrouter] ready on :3000
163164
```
164165

@@ -332,16 +333,16 @@ problem, because it only offers you providers that work.
332333

333334
| Connection `provider` | Backends that can use it |
334335
|---|---|
335-
| `anthropic` | Claude Code, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
336-
| `openai` | Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
337-
| `openrouter` | Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
338-
| `azure-foundry` | Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
336+
| `anthropic` | Claude Code, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
337+
| `openai` | Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
338+
| `openrouter` | Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
339+
| `azure-foundry` | Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
339340
| `google` | Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Gemini CLI, Cline, Oh My Pi |
340341
| `bedrock` | Claude Code, Hermes |
341-
| `tokenrouter` | Claude Code, Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Gemini CLI, Cline, Oh My Pi |
342-
| `vercel` | Claude Code, Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
343-
| `llmtr` | Claude Code, Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
344-
| `custom` | Claude Code, Codex (Responses format), Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi |
342+
| `tokenrouter` | Claude Code, Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Gemini CLI, Cline, Oh My Pi, goose |
343+
| `vercel` | Claude Code, Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
344+
| `llmtr` | Claude Code, Codex, Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
345+
| `custom` | Claude Code, Codex (Responses format), Hermes, Pi, DeepSeek Harness, OpenCode, Qwen Code, Cline, Oh My Pi, goose |
345346

346347
</details>
347348

@@ -649,7 +650,7 @@ Backends are installed into your data volume rather than baked into the image, s
649650
want is a run-time setting:
650651

651652
```bash
652-
docker run -e HR_BACKENDS=claude,codex,hermes,pi,dsh,opencode,qwen,gemini,cline,omp ... # the default
653+
docker run -e HR_BACKENDS=claude,codex,hermes,pi,dsh,opencode,qwen,gemini,cline,omp,goose ... # the default
653654
docker run -e HR_BACKENDS=opencode ... # lean
654655
```
655656

docs/support-matrix-notes.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,3 +229,15 @@ scripts/support-matrix/render.py docs/support-matrix-results.json`), committed b
229229
render is reproducible. The provider-wide `tokenrouter` and `vercel` columns of 2026-09-06 are not in it:
230230
their result files were lost with the scratchpad, and their sections in the table are carried from the
231231
render of that date until the columns are measured again.
232+
233+
## goose columns (2026-09-12, candidate 0.17.0-rc.1, goose 1.50.0, codex 0.154)
234+
235+
Seven columns for the goose harness (PR #164), five scenarios per model, two workers per column,
236+
the served model read by the relay (goose's CLI never reports one): TokenRouter 158/160, Vercel
237+
177/180, OpenRouter 178/180, OpenAI 35/35, Azure OpenAI e2 35/35, Anthropic 38/40, and the hosted
238+
HarnessRouter door 158/160 through the official key ($3.36 for its 160 checks). Zero substitutions.
239+
Every miss but two is claude-opus-5 answering "The model returned an empty response" on the
240+
artifact and recall turns, on every provider including Anthropic itself, while its plain turns pass:
241+
left off goose's list. gemini-3.5-flash-lite answered one recall with a tool call on Vercel only.
242+
The custom-harness dimension for goose needs `MCP_URL=https://mcp.context7.com/mcp` (deepwiki cannot
243+
handshake with goose, reproduced through goose's own extension flag); on Vercel all six claims pass.

0 commit comments

Comments
 (0)