feat(experiments): shadow mode — observe-only candidate fan-out (Step 5.7b)

[officialCodeWork]

Summary

Step 5.7b — shadow mode: on a sampled fraction of live /v1/query retrievals, run a candidate retriever alongside the served control path observe-only (never affecting the response) and feed both variants' outcome into the 5.7a ABExperimentTracker, so GET /v1/status/experiments compares them on real traffic before anyone serves the candidate.

This is the consolidated, final 5.7b — correct to merge. It is the functional superset of the two parallel implementations and supersedes the WIP on build/phase-5/step-5.7b-shadow-mode (20c1f58, no PR). Based on main after the TRACKER restructure (#144), so it merges clean (origin/main is an ancestor — no rebase, no conflicts).

Design — hot-path safe (the Phase-4 reliability invariant, applied to A/B)

• Never delays the response — the candidate runs in a FastAPI BackgroundTask, after the response is sent.
• Degrade-open — any candidate failure is logged (experiment.shadow_failed) and dropped; a crashing candidate still returns 200.
• Deterministic sampling — sha256(request_id) < shadow_sample_rate, stable across processes; skipped on a retrieval-cache hit (no fresh retrieval to compare).
• Same read_chunk PDP — the candidate routes through HybridRetriever, so ACLs are enforced; no PolicyEngine coverage-linter entry needed. Event-only observability (no per-call span).

Key choices (best of both parallel implementations)

• ShadowRunner (rag_gateway.experiments) — isolated + unit-testable, and keeps ABExperimentTracker a pure sample holder per ADR-0032 §3 (sampling lives in the gateway, not in observability).
• Records both variants atomically (only after the candidate succeeds) so the comparison windows stay balanced even when a candidate flaps.
• Config-driven candidate — cfg.experiments.shadow_candidate RRF weights build a real second RetrievalRouter, so shadow works end-to-end from rag.yaml; production can also inject one via build_app(shadow_runner=…).
• cfg.experiments gains shadow_enabled / shadow_sample_rate / shadow_experiment / shadow_candidate; doubly opt-in (enabled AND shadow_enabled) and observe-only, so turning it on cannot change a served response.
• ragctl shadow smoke command (drives the full flow against in-process stubs, creds-free).

Verification — all gates green

ruff · mypy --strict (292 files) · RAG001 · log-schema/event-registry · policy-coverage · schema-drift · openapi-drift (unchanged — BackgroundTasks is not a body/query param) · gateway / config / observability / ragctl / contract / retrieval / core suites · 27 new tests.

Docs

ADR-0032 (5.7b landed) · architecture/ab-shadow-mode.md · reference/experiments.md · docs/README.md · TRACKER (5.7b ✅, Next action → 5.7c).

Test plan

• Sampled query records both variants; zero-rate / unwired is a no-op; crashing candidate still returns 200
• Config-driven wiring builds the runner only when enabled + shadow_enabled
• Deterministic sampling; outcome metric (mean retrieval score) units
• Merges clean against current main

🤖 Generated with Claude Code