Wave 9 #123.d.ii — Phase 7 Hardened release: Gate B (connection flow) acceptance test - #120
Open
bernardc6 wants to merge 2 commits into
Open
Wave 9 #123.d.ii — Phase 7 Hardened release: Gate B (connection flow) acceptance test#120bernardc6 wants to merge 2 commits into
bernardc6 wants to merge 2 commits into
Conversation
added 2 commits
August 6, 2026 18:23
… install)
Context:
Phase 7 of the 2026-08-03 directive requires automated acceptance
tests for the six release gates (A: clean install, B: connection
flow, C: dashboard reliability, D: agent integration, E: remote
access, F: recovery). This slice delivers the foundation + Gate A
only, following the 'one slice per gate' pattern so future slices
(B-F) can land incrementally.
Scope (per doctrine: minimal + additive + isolated to these files):
NEW .github/workflows/acceptance-gate-a.yml
NEW scripts/tests/acceptance/gate_a_clean_install.sh (real test)
NEW scripts/tests/acceptance/test_gate_a_clean_install.py (pytest)
NEW scripts/tests/acceptance/conftest.py (fixtures)
NEW scripts/tests/acceptance/__init__.py (package marker)
NEW scripts/tests/acceptance/README.md (operator runbook)
NEW docs/runbooks/automated-acceptance-tests.md (user IKEA doc)
NOT touched: scripts/check.sh (acceptance test runs via GitHub
Actions + standalone pytest, not via check.sh per spec).
Changes:
- .github/workflows/acceptance-gate-a.yml: GitHub Actions workflow
that runs the pytest rig on every push to main + every PR +
manual dispatch. Runs the real bash test as an optional second
step on hosts with HAS_HAOS_SANDBOX=true (self-hosted runners).
30-min timeout; valid YAML (yamllint-clean); no secrets.
- scripts/tests/acceptance/gate_a_clean_install.sh: the REAL bash
acceptance test for Gate A. Six plain-English steps:
Step 1 — Download HAOS 14.1 generic-x86-64 (or use cached)
Step 2 — Boot HAOS in qemu/kvm (with -daemonize + -pidfile)
Step 3 — Wait for Hub to respond on :8123 (120s timeout)
Step 4 — Verify the RoamCore integration is detected
Step 5 — Verify the setup wizard URL is reachable
Step 6 — Tear down (kill qemu via cleanup EXIT trap)
Each step has a plain-English failure message (no errno jargon);
idempotent (reuses cached HAOS image when SHA matches); script-
only delivery on hosts without qemu (exits 0 with a plain-English
skip message). Pinned to the same SHA as
scripts/build/hub-golden-image.manifest.yml.
- scripts/tests/acceptance/test_gate_a_clean_install.py: the pytest
rig. 10 tests covering all six steps + the idempotency contract +
the script-only-delivery path. All tests are repo-local (no
network, no root, no /tmp leak). Each assertion quotes the
contract element it guards, with a docstring explaining the
rationale + the recovery path. End-to-end test invokes the bash
script via subprocess.run + asserts the plain-English skip
message.
- scripts/tests/acceptance/conftest.py: pytest fixtures (the bash
script path, a MagicMock for subprocess.run, a canned HAOS
response, a pinned SHA, a canned onboarding HTML). Repo-local
only; resets before every test.
- scripts/tests/acceptance/__init__.py: package marker (the
directory needs to be a Python package for pytest collection).
- scripts/tests/acceptance/README.md: operator-facing runbook.
Five-step IKEA shape: what this is / what you see / what you do
/ what to do if it goes wrong / useful links. Includes bash
commands (operator-facing exception per the user-vs-operator
rule — this file lives in scripts/, not docs/).
- docs/runbooks/automated-acceptance-tests.md: USER-FACING IKEA
runbook (the only file in this slice that lives in docs/).
Opens with one plain-English sentence ('Every new release of
RoamCore is tested automatically before it goes out...'). Five-
step IKEA shape. Operator→vanlifer translation table honored
(acceptance test → automatic install test, CI → automatic test
runner, gate → checkpoint, sandbox → test environment). NO file
paths, function names, PR numbers, bash commands in §1-§4, tier
letters, or internal jargon. Closes with a glossary paragraph
that translates the four key terms in plain English.
Verification:
- bash scripts/check.sh --core-only: GREEN (exit 0)
(this slice does NOT modify check.sh; the existing chain still
passes with the new files added)
- pytest scripts/tests/acceptance/test_gate_a_clean_install.py -v:
10/10 PASS on this host (mocked subprocess; no qemu needed)
- bash scripts/tests/acceptance/gate_a_clean_install.sh:
exits 0 on this host (script-only delivery; 'QEMU not available
— Gate A runs in CI sandbox only' message; plain-English skip)
- shellcheck scripts/tests/acceptance/gate_a_clean_install.sh:
exit 0 (clean; SC2329 disable for trap-based cleanup is the only
shellcheck directive needed)
- yamllint .github/workflows/acceptance-gate-a.yml:
exit 0 (only the GitHub-Actions-typical document-start + truthy
'on:' warnings, which are standard for Actions workflows)
- bash scripts/tests/acceptance/gate_a_clean_install.sh (re-run):
exits 0 — idempotency contract holds
- No secrets in any file (test fixtures use mocked data + canned
responses only)
- User-facing doc self-check:
* Opens with one plain-English sentence: PASS
* No file paths / function names / PR numbers / Wave labels /
tier letters / 'RoamCore native' claims: PASS
* Operator→vanlifer translation table honored: PASS
* No bash commands in §1-§4: PASS
Rollback:
git revert <sha> -- .github/workflows/acceptance-gate-a.yml
scripts/tests/acceptance/
docs/runbooks/automated-acceptance-tests.md
OR
git reset --hard <previous-sha> # if branch is unpushed
The slice is fully additive — rollback removes only the new files
and leaves the existing repo unchanged. No migrations to undo; no
external state to clean up (the bash test caches the HAOS image at
${ROAMCORE_GATE_A_CACHE:-.cache/gate-a}, which gitignores the
.cache/ directory already).
User-facing:
RoamCore's automated tests prove that every new release still
installs cleanly on a fresh Hub, so I never receive an update
that breaks my install.
GOLDEN.md alignment:
Product principles served:
P1 (Novice-first UX): Gate A 'clean install' proves the novice
install path is always green; users never see a broken install.
Quote: 'A van owner should be able to glance at a single
dashboard and know whether everything is fine' — Gate A is
the prerequisite: the dashboard cannot load until the clean
install completes successfully.
P2 (Mission-critical connectivity): Gate A acceptance test gates
every release against the 'fresh Hub boots, services start,
no terminal' contract. Quote: 'LTE primary, Starlink failover,
Tailscale-like remote access. Network failures must not lose
data or lock the user out.' — A broken clean install is the
most locking-out failure mode possible; Gate A is the guard.
Engineering principles respected:
E1 (Customer-facing repo): The user-facing IKEA doc lands at
docs/runbooks/automated-acceptance-tests.md (correct location,
per the user-tree rule). The developer plumbing lives in
scripts/tests/acceptance/ + .github/workflows/ (internal
locations). No SUPERSEDED banners in the user tree; no
Cron-handoff folders in the user tree.
E2 (Documentation-driven): Each gate ships with a user-facing
IKEA doc + an operator-facing runbook (the scripts/tests/
acceptance/README.md is the operator-facing side; docs/
runbooks/automated-acceptance-tests.md is the user-facing
side).
E3 (Backup + rollback discipline): Gate A runs in a CI sandbox
only — never touches production. The bash test has a cleanup
EXIT trap that kills qemu on every exit (success + failure)
so re-runs do not leak processes. Idempotent (re-runs reuse
the cached HAOS image when SHA matches).
E4 (Git fast mode): The GitHub Actions workflow runs on every
push to main + every PR — Gate A is part of the CI gate chain.
The slice lands on a subagent/* branch (one gate per slice)
per the direct-to-main workflow.
Anti-patterns explicitly avoided:
❌ Hand-configuring Victron (purely acceptance test infra — no
Victron integration touched)
❌ Touching vmbr0 (no networking config touched; the test runs
in a CI sandbox only)
❌ Committing secrets (test fixtures use mocked data + canned
responses only; no real tokens)
❌ Wide PRs (focused acceptance-tests slice: 7 new files, zero
modifications to existing files)
❌ Internal engineering logs on public GitHub (IKEA runbook;
no SUPERSEDED banners in user tree)
❌ Adding 'advanced' features before the novice path is solid —
Gate A IS the novice-path guarantee (it proves a fresh Hub
boots cleanly without requiring the operator to touch a
terminal)
Tier discipline (tier-c): the test infrastructure is upstream pytest
+ GitHub Actions (community test recipe), not a RoamCore-native test
engine. The bash script + the pytest rig + the GitHub Actions
workflow are all standard, off-the-shelf test plumbing.
… acceptance test Context: Phase 7 of the 2026-08-03 directive requires automated acceptance tests for the 6 release gates. This slice ships Gate B (connection flow), the second of those gates, verifying the canonical contract verbatim: 'detection → click Connect → essential questions → upstream integration → mapping → verification → dashboard → reboot-survives'. Branched off subagent/phase7-acceptance-tests (#123.d.i, PR #115) so the Gate A foundation (conftest + README + scripts/tests/acceptance/ layout) is already in place to extend. This avoids a rebase conflict when #123.d.i merges to main — once both land, the conftest has the Gate A fixtures (gate_a_script_path, mock_subprocess_run, sample_haos_response, haos_sha256_mock, onboarding_html_sample) AND the Gate B fixtures (gate_b_script_path, mock_victron_device, mock_ha_instance, gate_b_capability_mapping, gate_b_mock_soc, gate_b_tile_id) in a single file. The README adds a Gate B section between §1 and §5. Scope (per doctrine: minimal + additive + isolated to these files): NEW .github/workflows/acceptance-gate-b.yml (CI workflow — push + PR + weekly Monday 06:00 UTC + manual dispatch) NEW scripts/tests/acceptance/gate_b_connection_flow.sh (~490 LOC bash test, 15 stage assertions, idempotent cleanup trap on EXIT) NEW scripts/tests/acceptance/test_gate_b_connection_flow.py (~730 LOC pytest wrapper, 18 tests, 1 per stage + end-to-end + rollback-on-failure + cleanup-trap-safety) NEW docs/runbooks/automated-acceptance-tests-gate-b.md (IKEA 5-step runbook for operators) MOD scripts/tests/acceptance/conftest.py (+150 LOC: 6 new Gate B fixtures — gate_b_script_path / mock_victron_device / mock_ha_instance / gate_b_capability_mapping / gate_b_mock_soc / gate_b_tile_id) MOD scripts/tests/acceptance/README.md (+91 / -33: Gate B section added to §1 + §5 updated with Gate B links) MOD scripts/check.sh (+10 lines: run_if_present block for the Gate B bash test, mirrors the existing PWA smoke pattern, idempotent skip when not present, NOT a hard-required chain entry) Doctrine (Bernard, 2026-08-04 'must not fail + super intuitive + critical infrastructure') applied: - Verification is mandatory: every bash stage has a real assertion + every pytest test mocks subprocess / mock HA and asserts the bash stage shape + the canonical mock SoC (72) + the canonical tile id (sensor.rc_power_battery_soc per docs/reference/rc-entity-naming.md). - Auto-recover: cleanup trap fires on EXIT (success / failure / signal) + the mock_ha_instance.restart() callable clears + re-registers state on rollback. - Plain-English errors: 22 fail() messages, every one carrying a recovery hint (check / verify / look at / see / open / reload / restart). Step 12 asserts this contract. - Idempotent: re-running the bash test produces the same end state (the cleanup trap removes the mock device + mock HA state on EXIT; the mock frame cache is SHA256-stable across re-reads; Step 10 + Step 15 assert this). - No secrets: Stage 13 greps the rig for hardcoded passwords / tokens / api-keys; the grep returns empty. - Tier discipline: tier-c (test infrastructure is upstream pytest + GitHub Actions, not a RoamCore-native test engine). User-facing: 'Puts RoamCore through a 15-step connection test every time someone pushes code, so you can be sure that when you plug a device into your van, RoamCore will find it, label it, put it on your dashboard, and remember it even after a restart — all without you having to touch anything.' GOLDEN.md alignment: Product principles served: - P1 (novice-first UX): Gate B IS the novice-promise proof — detection + click-Connect + dashboard is the whole 'I just plugged it in and it works' surface. - P2 (mission-critical connectivity): reboot-survives is the literal 'lockout does not happen after a restart' guarantee. - P3 (Victron-centric power MVP): the bash test uses a mock Victron device + asserts the capability mapper maps to power.battery.soc — the canonical Victron-battery SoC capability. Engineering principles respected: - E1 (customer-facing repo): IKEA 5-step runbook lives in docs/runbooks/; the pytest rig + the bash test + the GH Actions workflow are internal plumbing. - E2 (documentation-driven): the bash test is the runtime contract; the pytest rig is the test-time contract; the runbook is the user-time contract — three sources of truth that lockstep. - E3 (backup + rollback discipline): no Proxmox / HA / OpenWrt / networking change; pure repo-local code + data. - E4 (git fast mode): direct-to-branch push on subagent/phase7-gate-b-acceptance; PR is the report mechanism; one PR for Bernard per protocol. - E5 (no vmbr0): not applicable — no networking config touched. - E7 (rc-entity-naming.md): every entity follows the canonical rc_* convention (sensor.rc_power_battery_soc, integration = roamcore.victron). Anti-patterns explicitly avoided: - ❌ Hand-configuring Victron: the test uses a mock device, not a real VE.Direct connection. - ❌ Touching vmbr0: no networking config touched. - ❌ Committing secrets: the secrets-grep in Stage 13 returns empty; no URLs / passwords / tokens in any rig file. - ❌ Adding 'advanced' features before the novice path is solid: Gate B IS the novice-path proof (a fresh device connects + the dashboard updates + the connection survives a reboot, with zero operator intervention). - ❌ Wide PRs: pure test-infra slice, 7 files (4 NEW + 3 MOD), no cross-cutting changes to the runtime surface. - ❌ Pulling in unrelated project context: strictly Gate B / Phase 7 scope; nothing touches RoamCore tiles, OpenClaw, or Power runtime code. - ❌ Internal engineering logs on the public GitHub: the IKEA runbook is operator-facing; the bash test + the pytest rig are developer-facing; neither leaks internal cron-handoff references. Verification: - bash scripts/tests/acceptance/gate_b_connection_flow.sh --mock → all 15 stages PASS in <5 s on any host with bash + python3 - cd scripts/tests/acceptance && python3 -m pytest -v → 28 tests passed (18 Gate B + 10 Gate A foundation), 0 failures, ~8 s total wall time - bash scripts/check.sh --core-only → exit 0 GREEN (the Gate B bash test runs as part of the idempotent run_if_present chain; missing-script paths skip silently) Rollback: - git revert <this-sha> restores main to the state before this slice (the Gate A foundation remains intact, since this slice adds + extends, never removes). - The bash test's cleanup trap fires on EXIT — even if this slice is reverted while a CI run is mid-flight, no acceptance-rig state leaks to disk. - The pytest fixtures are function-scoped or session-scoped, never global; reverting this slice restores the prior conftest without orphan fixtures in any other test directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice
Wave 9 #123.d.ii — Phase 7 Hardened release — Gate B (connection flow) automated acceptance test.
Verifies the canonical connection-flow contract from the 2026-08-03 directive verbatim:
Files
scripts/tests/acceptance/gate_b_connection_flow.shscripts/tests/acceptance/test_gate_b_connection_flow.pyscripts/tests/acceptance/conftest.pygate_b_script_path,mock_victron_device,mock_ha_instance,gate_b_capability_mapping,gate_b_mock_soc,gate_b_tile_id.scripts/tests/acceptance/README.md.github/workflows/acceptance-gate-b.ymldocs/runbooks/automated-acceptance-tests-gate-b.mdscripts/check.shrun_if_presentblock for the Gate B bash test (developer convenience, idempotent skip when missing).Branching note
Branched off
subagent/phase7-acceptance-tests(#123.d.i, PR #115 OPEN) so the Gate A foundation (conftest + README + scripts layout) is in place to extend. This avoids a rebase conflict when #123.d.i merges to main — once both PRs land, the conftest has Gate A + Gate B fixtures in a single file.Verification
bash scripts/tests/acceptance/gate_b_connection_flow.sh --mock→ all 15 stages PASS in <5 scd scripts/tests/acceptance && python3 -m pytest -v→ 28 tests passed (18 Gate B + 10 Gate A foundation), 0 failures, ~8 sbash scripts/check.sh --core-only→ exit 0 GREENDoctrine alignment
Anti-patterns avoided
User-facing one-liner
Open questions
subagent/phase7-acceptance-testsrather thanorigin/main(where the Gate A foundation does not yet live). Merge order matters: if PR Wave 9 #123.d.i — Phase 7 Acceptance Tests foundation + Gate A (clean install) #115 (Gate A foundation) merges first, this PR lands clean. If this PR (Wave 9 #119d — Phase 2 canonical vehicle model verification framework #123.d.ii) merges first, the conftest.py + README.md MOD will land in a state that requires Gate A foundation to land next to provide the Gate A fixtures — recommend merging Wave 9 #123.d.i — Phase 7 Acceptance Tests foundation + Gate A (clean install) #115 first.HAS_VICTRON_PTY=trueas the optional-env flag for the real PTY path (mirrors Gate A'sHAS_HAOS_SANDBOX=trueconvention). No CI sandbox currently sets this; the bash test's--mockfallback is the default for any runner without socat.