Long-running, local-first, multi-agent orchestrator (v1.2.0) - #14
Merged
Conversation
v1.1 governed how a single interactive session works through one task.
v1.2 adds the persistent layer above it: `specbridge orchestrate run
<spec>` drives an approved spec end to end as a durable, resumable job --
plan, schedule bounded agent executions, verify, diagnose, repair, replan,
escalate, checkpoint, survive interruption -- until the work is verified
complete or honestly blocked. SpecBridge owns state, policy, scheduling,
budgets, and completion; agents are replaceable ephemeral workers; Git and
trusted verification own implementation truth.
Jobs (`.specbridge/jobs/<id>/`, all schemas versioned 1.0.0, atomic,
bounded, append-only history):
- a 13-status fail-closed state machine in which RUNNING -> REPAIRING
deliberately does not exist: a failure must pass through DIAGNOSING,
so "no repair without a reasoned diagnosis" is structural
- runtime execution graphs independent of tasks.md -- one node per open
required leaf task, per-node ExecutionPlan revisions (the v1.1 plan
machinery reused wholesale), node supersession that carries attempt
history and replan budgets forward, and runtime ids that never touch
.kiro
- replans can never change approved intent: the replanner must declare
impactsApprovedIntent AND a deterministic keyword screen checks the
replacement against its predecessor regardless of the declaration
- a pure deterministic scheduler: (job, graph, policy, workers, clock)
-> the single next action, reproducible in tests and quoted verbatim
in the audit trail; sequential source mutation (maxConcurrentTasks
fixed at 1) matches the evidence model
Local-first multi-agent routing:
- six roles (CLASSIFIER/PLANNER/CRITIC/DIAGNOSER/REPLANNER read-only,
EXECUTOR the only writing role) across LOCAL_SMALL and LARGE_AGENT
tiers; the executor resolves only to a repository-writing large
agent -- the local worker never declares repositoryWrite, and the
routing enum admits nothing else (local execution is a future
additive opt-in, not a latent capability)
- deterministic complexity assessment with documented signal classes;
security/distributed/public-API/architecture signals force HIGH; a
local classifier may only RAISE the class
- escalate-on-evidence with 12 stable recorded reasons and sticky
per-node escalation -- a paid worker is never selected silently, and
escalation "manual" mode asks instead of spending
- structured local-agent contracts: zod schemas + strict JSON Schemas
for constrained decoding, complete-response validation (no substring
extraction, no silent repair), one bounded correction round; no
schema has a field for chain-of-thought
LocalModelManager (@specbridge/runners): managed llama.cpp lifecycle --
validated executable/model paths, loopback-only binding (a constant;
--host/--port/-m/credential flags rejected in extraArgs/executableArgs at
parse time), observed /health readiness, bounded log ring, idle shutdown,
bounded LAZY restarts, graceful reap. One server serves every role; a
local model crash is a worker failure, never a task failure.
Execution and failure handling: executor dispatches run through the
UNCHANGED evidence pipeline (runApprovedTask) -- an unverified success
claim is classified as a verification failure; the shared taxonomy drives
bounded retries, concrete clarification questions, terminal blocks, and
diagnosis; the diagnoser proposes and policy decides, degrading only
toward caution; repair dispatches receive the diagnosis as bounded
data-only prompt observations (additive extraObservations; absent, the
frozen prompt contract is byte-identical); no-progress detection reuses
the v1.1 fingerprinting.
Surfaces: CLI orchestrate run/jobs/job/node-plan/review-plan/answer/
cancel-job + local-model doctor/status; MCP job_list/job_read/job_cancel
(50 tools; jobs are driven by the standalone process, never from MCP);
plugin /specbridge:orchestrate with the nested-agent boundary documented
(the interactive session never launches the orchestrator or any nested
agent; the standalone orchestrator invoking the Claude Code runner is the
designed worker path). No daemon in this release -- the foreground
persistent process is the honest shape, and driveJob is daemon-ready by
construction.
Additive throughout: no persisted schema version moved, SBO025-SBO038
appended, config gains optional localInference and orchestration.jobs
blocks, and v1.0/v1.1 workspaces load with no migration.
Tests: 1479 -> 1648 (+169) plus 2 perf tests; the driver suite runs the
StepRelay readiness scenarios end to end against real child-process
fakes (fake llama-server over HTTP, fake Claude CLI, mock runner through
the real evidence pipeline): local plan -> executor completion,
HIGH-complexity escalation with a human plan gate, verification failure
-> diagnosis -> bounded repair -> honest budget stop, interruption ->
same-job resume, local-model crash isolation, manual escalation, and
missing-prerequisite clarification. The vitest worker ceiling drops 8 ->
6: the driver tests' subprocess load re-triggered the documented
coordinator-RPC starvation at 8 on a 24-core machine.
… test The roster test configured the local worker with a drive-letter path, which path.isAbsolute rejects on POSIX -- so validateLocalInferenceConfig refused the config and resolveWorkers never produced the local worker on the ubuntu/macos CI runners. Existence is a start-time concern, so any absolute path satisfies the coherence check; process.execPath is absolute on every OS.
…ache Three CI follow-ups from the first matrix run: - The committed plugin bundle was stale: two post-build lint fixes to package sources (an unused import, a prefer-const) never got re-bundled, so the ubuntu reproducibility step (git diff --exit-code on the plugin dist) failed against a fresh build. Rebuilt via build:plugin; checksums regenerated; verify-plugin-bundle passes. - Small-core Windows runners drop to ONE vitest worker: process spawn is several times more expensive there, and the 4-vCPU runner still starved the coordinator RPC with two workers' worth of subprocess children (all 1,648 tests passing, exit 1 -- the documented birpc onTaskUpdate timeout). One worker trades wall time for a coordinator that is never outnumbered. - The driver now probes the Claude CLI once per run instead of once per reasoning-role invocation (three short-lived processes each). The flag surface cannot change mid-run and a vanished CLI still fails safely at the real invocation, so this is a production latency win that also removes a large share of the driver suite''s subprocess load on CI.
The onTaskUpdate timeout persisted on the 4-vCPU Windows runner even with a SINGLE vitest worker -- all 1,648 tests passing and the run still exiting 1 -- which rules out coordinator CPU starvation and points at the forks pool''s process-IPC channel itself, a known failure class on Windows. Threads exchange the worker RPC over MessagePorts and sidestep that channel entirely; the full suite, the perf suite, and the subprocess-heavy driver suite all pass under threads on Windows with identical wall time. The suite is thread-safe by construction (no test worker calls process.chdir or process.exit; those appear only inside spawned fixture processes). Linux and macOS stay on forks, which is green there, and the single-worker Windows special case is dropped -- it demonstrably did not address the real channel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v1.1 governed how a single interactive session works through one task. This PR adds the persistent layer above it:
specbridge orchestrate run <spec>drives an approved spec end to end as a durable, resumable job — plan, schedule bounded agent executions, verify, diagnose, repair, replan, escalate, checkpoint, survive process interruption — until the approved work is verified complete or honestly blocked. SpecBridge owns state, policy, scheduling, budgets, and completion decisions; agents are replaceable ephemeral workers; Git and the trusted verification commands own implementation truth.Local-first, escalate-on-evidence. Cheap reasoning (classify / plan / critique / diagnose / replan) runs on a SpecBridge-managed llama.cpp server (
LocalModelManager: loopback-only binding as a constant, reserved flags rejected at config parse, observed /health readiness, bounded lazy restarts, idle shutdown); implementation and hard reasoning run on Claude Code as bounded ephemeral worker invocations. Deterministic complexity assessment routes HIGH work straight to the large agent; every escalation records one of 12 stable reasons; the EXECUTOR structurally resolves only to a repository-writing large agent (the local worker never declaresrepositoryWrite, and the routing enum admits nothing else). Local agents answer in schema-constrained structured contracts with complete-response validation — no substring extraction, no silent repair, one bounded correction round, no field anywhere for chain-of-thought.Nothing existing weakens. Executor dispatches run through the unchanged evidence pipeline (
runApprovedTask) — an unverified success claim is classified as a verification failure;RUNNING → REPAIRINGdeliberately does not exist in the job state machine, so a repair requires a diagnosis structurally; replans cannot change approved intent (the replanner must declare impact AND a deterministic keyword screen checks the replacement regardless); approval stays human-only with no agent-reachable surface;.kirostays byte-safe; no arbitrary shell/filesystem/Git tool on any surface. Additive throughout: no persisted schema version moved, SBO025–SBO038 appended, config gains optionallocalInference+orchestration.jobsblocks, v1.0/v1.1 workspaces load with no migration. No daemon in this release — the foreground persistent process is the honest shape, anddriveJobis daemon-ready by construction (documented in docs/orchestration/long-running-jobs.md).Surfaces: CLI
orchestrate run/jobs/job/node-plan/review-plan/answer/cancel-job+local-model doctor|status; MCPjob_list/job_read/job_cancel(50 tools — jobs are driven by the standalone process, never from MCP); plugin/specbridge:orchestrate(13 skills) with the nested-agent boundary documented explicitly.Tests 1479 → 1648 (+169) plus 2 perf benchmarks. The driver suite runs the StepRelay readiness scenarios end to end against real child-process fakes (fake llama-server over HTTP, fake Claude CLI, mock runner through the real evidence pipeline): local plan → executor completion, HIGH escalation with a human plan gate, verify-fail → diagnose → bounded repair → honest budget stop, interruption → same-job resume, local-model crash isolation, manual escalation, missing-prerequisite clarification.
Reviewer notes
Timeout calling "onTaskUpdate", all tests passing, exit 1) at 8 workers on a 24-core machine.TaskRunRequestgains additiveextraObservations(repair dispatches hand the executor the latest diagnosis as bounded data-only observations); absent, the frozen prompt contract is byte-identical — asserted by the existing prompt-contract tests.competingPlansis configured but the driver runs one planner per pass (critic + escalation cover the confidence role); diff fingerprints for no-progress use (path, changeType), biasing toward earlier replan, never toward looping.orchestrateskill.Checklist
pnpm lint,pnpm typecheck, andpnpm testpass locally (116 files / 1648 tests, exit 0;pnpm test:perf14/14;pnpm smoke52/52)pnpm check:public-contractspasses — contracts changed intentionally (new CLI commands, 3 MCP job tools, v1.2 orchestration vocabulary,orchestrateskill) and every snapshot undercontracts/is regenerated through the generator in this PRpnpm check:security: 411 files, no findings)Security-relevant invariants touched (SECURITY.md / docs/security/threat-model.md reviewed): the new local-model process manager binds 127.0.0.1 only with the bind address unconfigurable and reserved flags rejected at parse time; job persistence goes through
assertInsideWorkspacewith fail-safe corrupt-state handling; worker packets fence repository/spec content as data and agent output schemas physically cannot carry commands, approvals, or permissions; no new completion path exists — verified evidence via the existing pipeline remains the only one.