Adversarial hardening: poisoned-repository prompt injection test harness

[alexanderradahl]

Why

Early launch feedback raised the right question: what happens when the bridge is pointed at a deliberately poisoned repository whose README, logs, package output, or source comments try to steer the model into using unrestricted local tools?

SECURITY.md already states the current boundary plainly:

• prompt injection in repository files, web content, logs, email, package output, or other model-read material can try to steer tool use;
• credential scrubbing is hygiene, not isolation — an unrestricted shell can still attempt to read credentials available to the macOS user;
• there is no per-conversation capability lease in the current shared-stdio HTTP architecture.

A separate "protect git push" switch would not be an honest boundary while shell_exec remains arbitrary: the shell can invoke Git directly. Any restricted mode has to reduce the actual tool/command authority, not special-case one spelling of a dangerous action.

Goal

Build a repeatable adversarial fixture and document what the bridge does — and does not — protect when model-visible local content is malicious.

Proposed test matrix

Create a fixture repository containing benign-looking but adversarial instructions in several places:

• README.md / AGENTS.md
• source comments
• test output / log files
• package-manager lifecycle output
• filenames and Git metadata
• a fake credential file / fake SSH material that must never contain a real secret

Exercise representative workflows where the model is asked to inspect the repo, run tests, diagnose CI, and prepare a change. Record whether injected content attempts to trigger:

• credential reads
• network exfiltration
• destructive file operations
• force push / remote mutation
• persistence outside the repo
• relaunching the bridge or bypassing the unlock latch

Acceptance criteria

• Fixture contains no real credentials or user data.
• Reproduction instructions are safe to run in a disposable macOS account/VM.
• Expected bridge behavior is documented separately from expected model behavior.
• Existing guarantees (secret scrubbing from child env, unlock revocation, PTY/job containment claims) get deterministic regression coverage where possible.
• Anything the bridge cannot enforce is stated explicitly rather than converted into a misleading "safety" claim.
• Document whether an optional restricted mode is worth pursuing and what authority it would actually have to remove to be meaningful.

This is public hardening work, not a report of a newly discovered authentication bypass. Please use private vulnerability reporting for any concrete exploit that crosses a guarantee claimed by the project.

[alexanderradahl]

A launch reviewer sharpened one concrete case for the matrix:

While the unlock file is armed, is authority scoped to the specific action/tool, or does a prompt injection inherit unrestricted shell/filesystem access for the whole bridge session?

Current behavior is intentionally the latter: the unlock file is a process-global revocation latch, not an action-, tool-, caller-, or conversation-scoped capability. It is re-read before every tool call so removal fails closed, but while it remains armed any tool call reaching the shared bridge has the full registered authority. The latch therefore does not distinguish an operator-intended call from one steered by malicious repo/page/log content.

Add this explicitly to the adversarial fixture and expected-results documentation:

• With the latch armed, demonstrate that malicious model-visible content can attempt any exposed shell/filesystem tool; do not claim the latch mitigates prompt injection.
• With the latch removed, verify the next call is refused and the bridge exits/reclaims live sessions as documented.
• Evaluate whether a useful restricted mode requires reduced tool authority and/or scoped, expiring capabilities; command-name special cases are not a meaningful boundary while arbitrary shell remains available.

This is already consistent with SECURITY.md sections on the explicit unlock and the lack of per-conversation capability leases, but the reviewer’s formulation is a good acceptance test.

[alexanderradahl]

A Product Hunt discussion added a second useful acceptance-test seam: re-derivation itself can have reach.

The proposed expiry -> fail closed -> re-derive from current state rule is only clean if the revalidation path is genuinely narrower than the action path. In the current bridge it is not: fs_read, shell_exec, network-capable CLIs, and write tools all sit behind the same process-global unlock, and an arbitrary shell means a nominal “read” is not a separate authority class. Even a read can disclose secrets, hit an endpoint with side effects, acquire locks, or broaden what the model sees.

Add these cases to the hardening matrix:

• Do not treat “read” as inherently safe. Classify the observable/effect and its target, not the HTTP verb or tool name.
• If scoped capabilities are explored, a revalidation capability must be unable to widen actor / target / provider / fields / network destination relative to the original approved action.
• Prefer cheap precondition checks already available from the original observation where possible: Git HEAD, file hashes, row/version tokens, resource ETags, etc. A changed precondition invalidates the plan; it should not silently refresh into broader authority.
• If current state cannot be revalidated through a bounded observation capability, expiry resolves to deny/abort rather than recursively asking for another broad grant (“turtles”).
• Include an adversarial case where the revalidation step itself encounters malicious model-visible content or a state-changing “read” endpoint.

This sharpens the possible future model: the trusted primitive is not “read access”; it is an observation capability with bounded effects, plus an action capability that can only be narrower than the human-approved scope.

[lindixu6-hash]

I can take a narrow first pass at the deterministic fixture/test seam in this issue. Proposed scope:

• generate a disposable Git repository with synthetic adversarial content in README/AGENTS, source comments, test logs, package lifecycle output, filenames, Git metadata, and fake credential/SSH files;
• use only .example destinations and unmistakable synthetic identifiers; never execute the embedded instructions;
• drive the real stdio bridge to prove the current boundary: model-visible files are returned verbatim, the armed process-global latch still exposes the registered shell/filesystem authority, and removing the latch makes the next call fail closed;
• document expected bridge behavior separately from model behavior and state explicitly that this is not a model safety benchmark or a prompt-injection mitigation.

I will keep scoped-capability/restricted-mode design out of this PR; the harness should make that later work measurable without implying the current latch mitigates injection.

[alexanderradahl]

Yes — this is the right first slice. Please keep the PR exactly this narrow: deterministic fixture + bridge-boundary tests + expected-results docs, with no scoped-capability design mixed in yet.

A few constraints I’d keep explicit in the fixture:

• embedded adversarial instructions must remain inert test data; the harness should never execute them;
• use only synthetic credentials/keys and .example destinations, and make the test fail if it attempts a real network destination;
• distinguish what the bridge exposes/returns from what a model might decide to do — this should measure the current authority boundary, not claim to benchmark model prompt-injection resistance;
• cover both sides of the latch: armed = registered authority is available; removed = the very next call fails closed and live sessions/jobs are reclaimed as documented;
• make the generated repo disposable and self-cleaning so the fixture is safe to run repeatedly in CI/local development.

That gives us a solid baseline for the later bounded-observation / scoped-capability work without implying the current latch is a security boundary it isn’t. A focused PR on that scope would be very welcome.

[alexanderradahl]

Baseline update: #4 is merged and green on the repository macOS/Node 22 CI. It lands the deterministic poisoned-repository fixture, keeps bridge guarantees separate from model behavior, and explicitly does not claim the current global unlock mitigates prompt injection.

I also split the launch-feedback design work into a focused follow-up issue so this harness can remain a measurement baseline rather than grow into an authority-model rewrite: #5 tracks scoped capabilities, bounded observation/revalidation, monotonic narrowing, and the current shared-stdio caller-identity constraint.

The remaining acceptance criterion here is therefore to use the fixture as the regression target as those narrower-authority designs become concrete; the current unrestricted mode remains documented as unrestricted.