Commit 5b9fe4f
feat(harness): add goose as a new agent backend (#164)
* feat(harness): add goose as a new agent backend
Wires aaif-goose/goose (Apache-2.0) in end to end as a new "goose" backend,
alongside the existing claude/codex/hermes/pi/dsh/opencode/qwen/gemini/cline/omp
set. goose is a single prebuilt Rust binary fetched from GitHub releases, so it
installs the way opencode's does — the release asset directly, not the vendor's
download_cli.sh, which installs to a path outside the data volume. Pinned to
1.50.0: upstream ships weekly (1.49.0 -> 1.50.0 in five days) and every behavior
below was read out of that release's own source. Unlike omp there is no
SHA256SUMS to verify against, so the pinned tag is the whole of the guarantee,
and the installer says so.
It reaches models as cline and qwen do — one OpenAI chat/completions client
pointed at the loopback relay — so it is wired to the eight integrations that
serve that shape and needs no provider of its own. One genuine adapter
difference: goose's OpenAI provider takes the endpoint in TWO pieces
(OPENAI_HOST + OPENAI_BASE_PATH), so the relay's single base_url is split here
rather than teaching the relay a second shape; pasting it whole would send every
turn to /v1/v1/chat/completions, a 404 with no body.
Everything goose keeps — config, the sessions database, the harness's skills —
hangs off GOOSE_PATH_ROOT pointed inside .harness/, which is what makes the
conversation travel in the checkpoint and keeps goose's own files out of the
produced-file list.
Not a passthrough: the stream-json schema is its own, and it carries TWO serde
rename rules in one stream — snake_case event tags (message/notification/error/
complete) but camelCase content-block tags (toolRequest/toolResponse). The tool
call sits two levels down under toolCall.value, and the Ok/Err discriminator is
Rust's Result rendered as {status, value|error} with a STRING error. These came
off the v1.50.0 source, not the docs, which name the event types and publish no
field-level example — the same gap that made the first gemini normalizer guess
wrong and render every tool call as a content-free row. A regression test pins
each one.
Two findings the source reading produced before a line was written:
- goose emits cache_write_input_tokens in snake_case, which _norm_token_usage's
picker does not know (it carries only the camel spelling), so the count would
have billed as zero. Handled in a goose-side wrapper, the way _gemini_usage
handles gemini's `cached`.
- disabled tools are a HARD block here, and it survives headless approval.
GOOSE_MODE=auto short-circuits the permission table, so permission.yaml's
NeverAllow is not the lever; available_tools is, because fetch_all_tools gates
on it while BUILDING the model's tool list. UHP 4.3 requires a hard block to be
reported as one wherever the runtime supports it, so the base declares "hard"
and stays out of the standing-instruction set.
The extensions map written to config.yaml is deliberately PARTIAL: goose seeds
every platform extension it does not find there from that definition's own
default_enabled, so naming only what we restrict or add leaves Skills, Todo and
Analyze alone. Writing a complete map would have switched them off — and a
harness whose skills silently stopped loading still passes every routing scenario.
Live-verified on a self-hosted instance against a Vercel AI Gateway connection
(2026-09-12): all six catalog ids completed a first turn; gpt-5.4 also completed
follow-up, a switch to claude-sonnet-4.6 and back, an artifact turn whose
produced file matched the rendered card byte for byte, and a turn after a forced
sandbox recycle that still recalled the first message and re-read the file. The
custom-harness dimension passes all six of its claims: the skill bundle reached
the agent (the answer carried a token that exists only inside its script), the
script actually ran (it left its file among the produced files), the disabled
tool was genuinely absent, and a declared MCP server was stored and called.
Half of the catalog bar is met and the missing half is named rather than papered
over: these turns are not substitution-checked, because on this backend they
cannot be. The served model would have to come from metadata.inference.
resolvedModel, and only the databricks provider format populates it — the openai
path every turn takes never sets it. served_model is therefore always empty for
goose, and the catalog comment says to read these rows as "the id completed a
turn", not "the id served it".
docs/harness-verification.md's registration list is expanded from five places to
the sixteen that actually exist, since every one of them fails silently when
missed; the console's backend union type is the only one a compiler catches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(runner): the relay reports the served model for CLIs that do not; goose's list is the full chat-shaped set, to be trimmed by the column
The relay sees every provider answer, so it reads the first "model" off the
bytes as they pass and the run loop stamps it on a result event the CLI left
unlabelled (goose never reports one; cline and qwen do not either). goose's
catalog list becomes every id the wired chat-shaped providers can serve, for
the goose column to measure with the served model read by the relay.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk
* feat(gateway): goose's list is the 35 ids the column measured; claude-opus-5 stays out
Measured 2026-09-12 on TokenRouter, Vercel, OpenRouter, OpenAI, Azure,
Anthropic and the hosted door, served model read by the relay, no
substitution. claude-opus-5 answers with an empty response on every
tool-using turn through goose's chat wire, its own vendor included.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk
* docs(matrix): the goose columns on seven providers, the served model read by the relay
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk
* feat(gateway): goose's candidate list is the union of the wired providers' tables (five more ids to measure)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk
* docs(matrix): every provider table lists the pairs it serves that a harness did not run, with the reason
The Responses-only ids and the chat-only harnesses are now gateway constants
(the chat-only test reads them); the renderer derives each column's not-run
pairs from the gateway's provider tables and harness catalogs, so a model a
harness cannot run on a provider is stated as a limitation rather than absent.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej7eMwxnexLFqkiFzHq9Fk
* fix(goose): fail the turn when the provider error arrives as prose
Review of #164, all three items.
A provider error ends the turn as "completed" with the error as its answer.
goose does not emit StreamEvent::Error when the upstream fails mid-turn: the
agent loop pushes an assistant message reading "Ran into this error: <the
provider's sentence>" and then ends the run normally with `complete`. The
normaliser read that as the answer, so the turn recorded status: completed —
measured on TokenRouter's 503 during the support-matrix column — and the next
turn in the session answered from a history carrying the prose.
_GOOSE_ERR_RE now catches both prefixes the pinned 1.50.0 binary carries
("Ran into this error: " and "Ran into this error trying to compact: ",
confirmed with strings, the only two it holds). A matching assistant block
sets _goose_error instead of `final` and is not rendered, so `complete` yields
subtype error with the provider's sentence as the reason — the same treatment
_CLAUDE_ERR_RE gives claude's injected "API Error: …". _goose_eof now prefers
a reported error over earlier text, for the same reason on the abnormal path.
Four tests pin it, including that a turn merely TALKING about an error still
passes.
Pin the archives, not just the tag. Upstream publishes no SHA256SUMS, so the
digests are pinned here instead, computed from the v1.50.0 release assets and
verified against the download (both digests match the ones given in review).
A tag can be moved and an asset re-uploaded; a digest cannot. HR_GOOSE_VERSION
overriding the pin takes HR_GOOSE_SHA256 for the version chosen, or logs that
the archive is UNVERIFIED — never a silent skip. Exercised on the real release
for all three paths: match, mismatch, override.
_goose_has_session asks the database instead of searching its bytes. The old
substring search was only as good as the name was distinctive, and `harness`
also occurs in the workspace path (/data/workspaces/<sid>/.harness/...) that
tool output and the AGENTS.md contract carry — so a database holding one
message and no session could match, add -r, and fail the turn with "No session
found": the exact failure the check exists to prevent. It is now
SELECT 1 FROM sessions WHERE name = ? OR id = ?, read-only, which is what the
CLI itself matches on. The WAL clause goes away because sqlite reads the WAL
itself — pinned with an uncheckpointed WAL so it cannot pass for the wrong
reason. A schema drift on a pin bump fails the test rather than a live turn.
docs/harness-verification.md gains both lessons for the next harness: check how
the CLI reports a provider failure rather than assuming it has an error event,
and verify the install, since a pinned tag is not verification.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(goose): the catalog says which ids the column ran, and which it has not
Taking the review mirror's five commits (the relay served-model reader, the
measured list, the matrix columns). Three things they left inconsistent:
The block comment above goose's catalog entry still said served_model is
always empty for goose, that the rows are NOT substitution-checked "because on
this backend they cannot be", and that the list is six ids from a first sweep.
The relay change in 8f2affd is exactly what makes that false, and the column
measured zero substitutions across seven providers. It now says where the
served model comes from and that the rows mean "the id served it".
The list ships 40 ids; the goose column ran 36 and 35 shipped. hunyuan-3,
ling-3.0-flash, minimax-m3, nemotron-3-ultra and qwen3.7-flash have no goose
row in support-matrix-results.json, and docs/support-matrix.md already lists
them as "not run in this column" — but the comment above the list said every
id in it was measured on 2026-09-12. Keeping them is the pi precedent (ids
offered so the matrix can measure them), so they stay and are labelled as what
they are. Whether to ship them before a column runs is yours to say; the code
no longer claims they were measured either way.
docs/harness-verification.md: rule 2 says how the served model is judged but
not where it comes from, which just changed for three harnesses. The question
to ask of a new harness is now "does it ride the relay", not "does the CLI
report a served model".
Two supplements to the relay reader itself:
- _served_model_in reads within one 4096-byte read, so a boundary falling
inside "model":"…" loses it, silently — served_model stays empty and the
turn is simply not substitution-checked. The SSE loop now carries the
previous chunk's tail; one test pins it with a real split.
- _relay_served_model takes the FIRST hr-relay- bearer in env as the turn's
route. That is right today only because the one place that registers a
second route (hermes' auxiliary vision model) runs on _run_hermes_bg, which
never calls this. Written down, because reading the vision route would stamp
the image model as the served model — a substitution finding invented out of
nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Zixiao Li <ted.li.job@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: richard-epsilla <richard@epsilla.com>
Co-authored-by: richard-epsilla <131846445+richard-epsilla@users.noreply.github.com>1 parent 8a4c010 commit 5b9fe4f
16 files changed
Lines changed: 15394 additions & 1100 deletions
File tree
- docker
- docs
- gateway
- tests
- runner
- tests
- scripts/support-matrix
- ui/src
- components
- lib
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
| 180 | + | |
181 | 181 | | |
182 | 182 | | |
183 | 183 | | |
| |||
195 | 195 | | |
196 | 196 | | |
197 | 197 | | |
| 198 | + | |
198 | 199 | | |
199 | 200 | | |
200 | 201 | | |
| |||
223 | 224 | | |
224 | 225 | | |
225 | 226 | | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
226 | 278 | | |
227 | 279 | | |
228 | 280 | | |
| |||
328 | 380 | | |
329 | 381 | | |
330 | 382 | | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
331 | 388 | | |
332 | 389 | | |
333 | 390 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
41 | 51 | | |
42 | 52 | | |
43 | 53 | | |
| |||
108 | 118 | | |
109 | 119 | | |
110 | 120 | | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
121 | 182 | | |
122 | 183 | | |
123 | 184 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
| 159 | + | |
159 | 160 | | |
160 | 161 | | |
161 | | - | |
| 162 | + | |
162 | 163 | | |
163 | 164 | | |
164 | 165 | | |
| |||
332 | 333 | | |
333 | 334 | | |
334 | 335 | | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
339 | 340 | | |
340 | 341 | | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
345 | 346 | | |
346 | 347 | | |
347 | 348 | | |
| |||
649 | 650 | | |
650 | 651 | | |
651 | 652 | | |
652 | | - | |
| 653 | + | |
653 | 654 | | |
654 | 655 | | |
655 | 656 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
229 | 229 | | |
230 | 230 | | |
231 | 231 | | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
0 commit comments