Skip to content

test(browser): add controlled Agent Task fixture - #65

Open
seonghobae wants to merge 3 commits into
mainfrom
test/agent-task-controlled-fixture
Open

test(browser): add controlled Agent Task fixture#65
seonghobae wants to merge 3 commits into
mainfrom
test/agent-task-controlled-fixture

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Partial implementation of #28.

Buyer-visible gap

The first real Chromium Agent Task slice needs a deterministic local web application whose semantic form, hostile hidden page content, and observable post-condition are stable enough to exercise observation → policy → typed action → verification without production credentials or a live third-party site.

TDD evidence

This lane is based directly on protected main 67af7c87589edc2039545af335c95064d9b8391c.

  1. Test-only exact head d2580305f05aba93d10b5342ec1886d601c6752e introduced the contract for tests/fixtures/agent_task_basic/index.html before the fixture existed.
  2. CI run 31445088008, Rust contracts job 93637443229, checked out that exact head and failed in Python repository contracts with exactly three FileNotFoundError failures for the absent fixture. Repository checkout evidence confirms the tested revision. This is the intended fail-first boundary, not a predecessor failure reused as success evidence.
  3. Head 0888fe3a6ef6da547a37fd075733cc73dc52b2ab added the controlled fixture required by that contract.
  4. CodeRabbit then found two separate follow-ups. The credential/hidden-marker parsing defect was fixed on current exact head cd4115e1b7e06717c3fdd8464da191fa3ca0cc60: fixture tests now parse input attributes independent of quote style, reject password/one-time-code autocomplete semantics, and require the actual hidden attribute in addition to aria-hidden=true for the hostile-content marker.
  5. The separate real-browser submission finding is intentionally not folded into this fixture-only root. Stacked PR test(browser): execute controlled Agent Task on pinned Chrome #70 owns pinned-Chromium clear/type/click/post-condition and URL-invariance execution evidence. The corresponding test(browser): add controlled Agent Task fixture #65 review thread is now resolved in GitHub to reflect that ownership split, but the stacked execution evidence is not transferred to this fixture-only branch and test(browser): add controlled Agent Task fixture #65 still does not claim browser execution.

Implemented boundary

The fixture now provides:

  • a labelled semantic text field (task-text) and submit control that can later be discovered by role/name rather than CSS/XPath authority;
  • deterministic same-document state transition from data-state="idle" to data-state="submitted" while exposing only the typed synthetic value as the result;
  • one explicitly marked and actually hidden/untrusted prompt-injection string that attempts to make the consumer ignore policy and request new browser capabilities;
  • synthetic local data only, with parsed credential-surface regressions covering password and one-time-code autocomplete semantics independent of HTML quote style; and
  • no network dependency or live third-party application requirement.

Exact-head verification

On unchanged exact head cd4115e1b7e06717c3fdd8464da191fa3ca0cc60 against unchanged protected main 67af7c87589edc2039545af335c95064d9b8391c:

  • CI run 31503080956: success;
  • Rust contracts job 93817759541: repository Python contracts, canonical formatting, locked workspace/all-target check, full tests, strict Clippy, and rustdoc all success;
  • Production coverage job 93817759257: exact owned production function/line/region/branch enforcement success;
  • Security Scan run 31503080894: success;
  • SAST Semgrep run 31503080841: success;
  • CodeRabbit exact-head commit status: success;
  • both currently returned inline review threads are resolved; and
  • GitHub reports the PR mergeable and Ready for review.

No predecessor-head check, review, or stacked-branch execution evidence is transferred to this exact head.

Scope boundary

This fixture is controlled test infrastructure, not a browser adapter or action-success claim. It does not implement WebDriver BiDi/CDP, semantic extraction from Chromium, deterministic policy dispatch, native browser input, trusted clock provenance, secret handling, destination authority, post-condition evidence, profile teardown, process attribution, or the complete issue #28 vertical slice. Stacked PR #70 executes this fixture through pinned stock Chromium; its evidence remains active-PR evidence and is not treated as shipped truth for this branch.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

결정적 Agent Task HTML fixture를 추가했습니다. fixture는 입력 제출 결과와 submitted 상태를 표시합니다. 계약 테스트는 DOM 구조, 상태 변경, 비신뢰 지시문, credential 필드 부재를 검증합니다.

Changes

Agent Task fixture

Layer / File(s) Summary
Fixture 동작과 계약 검증
tests/fixtures/agent_task_basic/index.html, tests/test_agent_task_fixture_contract.py
HTML fixture가 입력 폼, idle 결과 상태, 숨겨진 prompt-injection 지시문을 정의합니다. 제출 시 기본 동작을 차단하고 입력값과 상태를 갱신합니다. 테스트는 요소 식별자, label 연결, input 이름과 속성, 버튼 타입, 결과 갱신, 비신뢰 콘텐츠 수, credential 필드 부재를 검증합니다.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 제어된 Agent Task fixture 추가라는 변경의 주요 내용을 명확하고 간결하게 설명합니다.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/agent-task-controlled-fixture

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae marked this pull request as ready for review August 11, 2026 00:14

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_agent_task_fixture_contract.py`:
- Around line 37-41: Update the HTML attribute parsing used by the
hidden-injection contract to retain the complete parsed attributes for each
input element. Increment hidden_injection_markers only when an untrusted marker
also has a hidden attribute, and validate credential inputs from parsed
attribute values rather than raw quoted text: reject type=password and
autocomplete values password or one-time-code regardless of quote style.
- Around line 62-64: HTML 문자열 검사만 수행하는
tests/test_agent_task_fixture_contract.py에 실제 브라우저 제출 검증을 추가하십시오.
scripts/ci/run_mv3_compatibility.py의 기존 Chrome WebDriver 경로에서
agent_task_basic/index.html을 로드하고 task-text를 변경해 제출한 뒤, 제출 전후 URL이 동일하고
task-result의 data-state가 submitted이며 textContent가 변경한 입력값과 정확히 일치하는지 검사하십시오.
.github/workflows/mv3-compatibility.yml의 실행 경로에 해당 fixture와 테스트 변경을 포함하십시오.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 60e1549b-6ea0-49e9-b88e-5e7de0c066c0

📥 Commits

Reviewing files that changed from the base of the PR and between 67af7c8 and 0888fe3.

📒 Files selected for processing (2)
  • tests/fixtures/agent_task_basic/index.html
  • tests/test_agent_task_fixture_contract.py

Comment thread tests/test_agent_task_fixture_contract.py
Comment thread tests/test_agent_task_fixture_contract.py

Copy link
Copy Markdown
Contributor Author

Fresh review-thread revalidation: the remaining CodeRabbit finding asks this fixture-only root to add real pinned-browser submit execution. That requirement is valid for the complete #28 vertical slice, but it is no longer a defect in #65's bounded claim. Exact current #65 head cd4115e1b7e06717c3fdd8464da191fa3ca0cc60 explicitly claims only deterministic fixture semantics and does not claim browser execution. Exact dependent #70 head 33ec162e932b99f9f7bd78ea0108b5e8501e7694 owns the requested clear/type/submit path, unchanged-URL check, data-state=submitted, exact echo, profile cleanup and repeatability, with its own exact-head CI and real-browser evidence. No #70 evidence is transferred to #65. The inline finding is therefore scope-separated into the dependent implementation lane rather than duplicated into the fixture prerequisite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant