docs(adr,architecture): finalize ADRs 0005–0009 + extend reviewer checklist (Step 1.1f)

TRACKER.md

@@ -10,7 +10,7 @@
 
 **Last updated:** 2026-05-24
 **Current phase:** Phase 1 — Ingestion + Knowledge Store
-**Next action:** Phase 1 Step 1.1f — ADRs 0005–0009 (already drafted) finalized + reviewer checklist landed in `docs/architecture/performance.md` and cross-referenced from every relevant module
+**Next action:** Phase 1 Step 1.2 — Connectors framework: `Connector` SPI implementation (now receives `RequestContext` and `ConnectorState` watermark); built-in filesystem / S3 / GCS connectors; crawler base class. With 1.1a–1.1f landed, the refactor window is closed.
 
 > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].
 
@@ -32,14 +32,14 @@
 | Phase | Title | Steps | ✅ Done | Remaining |
 |-------|-------|------:|-------:|----------:|
 | 0 | Foundation | 13 | **13** | 0 |
-| 1 | Ingestion + Knowledge Store | 16 | **5** | 11 |
+| 1 | Ingestion + Knowledge Store | 16 | **7** | 9 |
 | 2 | Retrieval Engine | 11 | 0 | 11 |
 | 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
 | 4 | Reliability | 6 | 0 | 6 |
 | 5 | Eval & Observability | 7 | 0 | 7 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **18** | **66** |
+| **Total** | | **84** | **20** | **64** |
 
 ---
 
@@ -72,8 +72,8 @@
 | 1.1b | SPI split — Retrieval/Index, bulk + streaming + ID-only | ✅ | `build/phase-1/step-1.1b-spi-split-retrieval-index` | [#45](https://github.com/officialCodeWork/AgentContextOS/pull/45) | Split `VectorStore`/`KeywordStore`/`GraphStore` into `*RetrievalBackend` (read) + `*IndexBackend` (write) composite ABCs. `retrieve_ids` returns `list[ChunkRef]`; `hydrate` lives on the retrieval side (keyword full-Chunk; vector pass-through). Bulk: `bulk_index`/`bulk_delete` (+ graph bulk node/edge variants); streaming: `stream_index` async-iterator default that batches into `bulk_index`. `Embedder` split into single `embed` + canonical `bulk_embed`. New `IndexHint` + `WriteVolume` types passed to writes (per ADR-0009). Noop impls, `PgVectorStore`, `QdrantVectorStore` migrated; conformance + integration tests updated; `spi_signature.py` extended to enforce the split. Schemas regenerated (`IndexHint.json`). |
 | 1.1c | PolicyEngine package | ✅ | `build/phase-1/step-1.1c-policy-engine-package` | [#46](https://github.com/officialCodeWork/AgentContextOS/pull/46) | New `packages/policy/` (`rag-policy` v0.1.0): `PolicyEngine` SPI + `NoopPolicyEngine` (always-ALLOW with tenant-scoped `filter_pushdown`); `PolicyDecision` enum (read_chunk / ingest_doc / egress_text / quota_check / rate_limit / execute_plan); `PolicyResult` (allow/deny/transform) with predicate helpers; `QuotaSubject` / `RateLimitSubject`; `FilterExpr` mini-language (Eq / AnyIn / And / Or / Not / TrueExpr) returned by `filter_pushdown`; `PolicyWriter` facade mirroring `AuditWriter` and emitting `policy.decision` structured logs via `rag-observability`. Coverage linter `tests/policy/coverage.py` greps for governance-relevant SPI calls without adjacent `PolicyEngine`/`PolicyWriter` consultation, with file allowlist that consumers shrink as they wire the PDP in. Workspace + pytest pythonpath updated. 20 conformance tests added. ADR-0005 finalized. |
 | 1.1d | Pipeline + Batcher primitives | ✅ | `build/phase-1/step-1.1d-pipeline-batcher-primitives` | [#50](https://github.com/officialCodeWork/AgentContextOS/pull/50) | `Pipeline` primitive in `rag-core` (`rag_core.pipeline`): async DAG with bounded queues, per-stage worker counts, backpressure, joiner-based shutdown handling per-stage worker-count differences, `on_error="fail"`/`"skip"` policies, optional `error_handler`. `Batcher[Req, Resp]` middleware (`rag_core.batcher`, DataLoader pattern) coalescing concurrent `load(req)` calls into batched `batch_fn(reqs)` invocations; size + time triggers (`max_batch_size`, `max_wait_ms`); per-request `asyncio.Future` for isolated error propagation; `flush()` for deterministic shutdown. PEP-695-typed generics. 22 unit tests (8 Batcher + 14 Pipeline). `docs/architecture/pipeline-batcher.md` + `rag-core` reference + `performance.md` backlinks. `rag-core` 0.5.0 → 0.6.0. |
-| 1.1e | Cache SPI split + perf discipline + async telemetry | 🚧 | `build/phase-1/step-1.1e-cache-split-perf-telemetry` | — | `EmbeddingCache` (`rag_core.spi.embedding_cache`, key tenant+model_id+model_version+text_hash, `invalidate_model`), `RetrievalCache` (`rag_core.spi.retrieval_cache`, key tenant+plan_hash+corpus_version, `invalidate_corpus`), `AnswerCache` (`rag_core.spi.answer_cache`, key tenant+plan_hash+corpus_version+policy_version, `invalidate_policy`). All ctx-first; signature linter extended. Noop in-memory impls + 18 conformance tests covering hit/miss/version-partition/invalidate/tenant-isolation. `AsyncTelemetrySink` in `rag-observability` — bounded buffer (default 10_000), non-blocking `submit()`, per-kind drop + exporter-error counters, idempotent start/stop, drain-timeout-cancel; 9 unit tests. Hot-path discipline doc in `performance.md` expanded with concrete hot/cold call-site table. New `docs/reference/rag-observability.md`. `caching.md` SPI shapes updated with ctx-first signatures. `rag-core` 0.6.0 → 0.7.0. |
-| 1.1f | ADRs 0005–0009 + reviewer checklist | ⏳ | 1.1a–1.1e | — | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization). Reviewer checklist in `docs/architecture/performance.md`. |
+| 1.1e | Cache SPI split + perf discipline + async telemetry | ✅ | `build/phase-1/step-1.1e-cache-split-perf-telemetry` | [#51](https://github.com/officialCodeWork/AgentContextOS/pull/51) | `EmbeddingCache` (`rag_core.spi.embedding_cache`, key tenant+model_id+model_version+text_hash, `invalidate_model`), `RetrievalCache` (`rag_core.spi.retrieval_cache`, key tenant+plan_hash+corpus_version, `invalidate_corpus`), `AnswerCache` (`rag_core.spi.answer_cache`, key tenant+plan_hash+corpus_version+policy_version, `invalidate_policy`). All ctx-first; signature linter extended. Noop in-memory impls + 18 conformance tests covering hit/miss/version-partition/invalidate/tenant-isolation. `AsyncTelemetrySink` in `rag-observability` — bounded buffer (default 10_000), non-blocking `submit()`, per-kind drop + exporter-error counters, idempotent start/stop, drain-timeout-cancel; 9 unit tests. Hot-path discipline doc in `performance.md` expanded with concrete hot/cold call-site table. New `docs/reference/rag-observability.md`. `caching.md` SPI shapes updated with ctx-first signatures. `rag-core` 0.6.0 → 0.7.0. |
+| 1.1f | ADRs 0005–0009 + reviewer checklist | 🚧 | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | — | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). |
 | 1.2 | Connectors framework | ⏳ | 1.1a–1.1f | — | `Connector` SPI implementation (now receives `RequestContext` and `ConnectorState` watermark); built-in: filesystem, S3, GCS; crawler base class |
 | 1.3 | Document parsers | ⏳ | — | — | PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection |
 | 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring |
@@ -219,6 +219,7 @@
 | [#48](https://github.com/officialCodeWork/AgentContextOS/pull/48) | chore(ci): lint scripts/ + ignore S603/S607 in scripts/** | `chore/lint-ignore-subprocess-in-scripts` | ✅ Merged | 2026-05-24 |
 | [#49](https://github.com/officialCodeWork/AgentContextOS/pull/49) | chore(tracker): sync PR links for steps 1.1a/1.1c + log #46–#48 | `chore/tracker-sync-pr44-48` | ✅ Merged | 2026-05-24 |
 | [#50](https://github.com/officialCodeWork/AgentContextOS/pull/50) | feat(core): Pipeline + Batcher primitives (Step 1.1d) | `build/phase-1/step-1.1d-pipeline-batcher-primitives` | ✅ Merged | 2026-05-24 |
+| [#51](https://github.com/officialCodeWork/AgentContextOS/pull/51) | feat(core,observability): Cache SPI split + async telemetry sink (Step 1.1e) | `build/phase-1/step-1.1e-cache-split-perf-telemetry` | ✅ Merged | 2026-05-24 |
 
 ---
 

docs/adr/ADR-0005-policy-engine.md

@@ -1,9 +1,9 @@
 # ADR-0005 — PolicyEngine as central PDP
 
-**Status:** Accepted
+**Status:** Accepted — implemented in Step 1.1c ([PR #46](https://github.com/officialCodeWork/AgentContextOS/pull/46))
 **Date:** 2026-05-24
 **Deciders:** Core team
-**Step:** Phase 1 Step 1.1c
+**Step:** Phase 1 Step 1.1c (implementation); finalized in Step 1.1f
 
 ---
 
@@ -73,8 +73,20 @@ Always-ALLOW noop for OSS / dev. Production tenants override with their own impl
 - **Embed a third-party PDP (OPA sidecar) from day 1:** Rejected for V1. Adds operational complexity (sidecar, language boundary) before the PDP-as-pattern is internalized. Future: ship an `OpaPolicyEngine` adapter.
 - **Rust-based in-process PDP:** Rejected for V1. Premature.
 
+## Implementation
+
+| Surface | Path |
+|---|---|
+| SPI | [`packages/policy/src/rag_policy/engine.py`](../../packages/policy/src/rag_policy/engine.py) |
+| Decision / result types | [`packages/policy/src/rag_policy/types.py`](../../packages/policy/src/rag_policy/types.py) |
+| Filter mini-language | [`packages/policy/src/rag_policy/filter.py`](../../packages/policy/src/rag_policy/filter.py) |
+| Noop reference impl | [`packages/policy/src/rag_policy/noop.py`](../../packages/policy/src/rag_policy/noop.py) |
+| Facade (audit + log) | [`packages/policy/src/rag_policy/writer.py`](../../packages/policy/src/rag_policy/writer.py) |
+| Coverage linter | [`tests/policy/coverage.py`](../../tests/policy/coverage.py) |
+
 ## References
 
 - [docs/architecture/policy-engine.md](../architecture/policy-engine.md)
+- [docs/architecture/performance.md](../architecture/performance.md) (reviewer checklist — PolicyEngine call-site rule)
 - TRACKER.md Step 1.1c
 - Steps 1.7, 4.5, 6.3, 6.4, 6.5 (consumers / supersedes)

docs/adr/ADR-0006-two-stage-rerank.md

@@ -1,6 +1,6 @@
 # ADR-0006 — Two-stage reranker as default SPI shape
 
-**Status:** Accepted
+**Status:** Accepted — decision finalized in Step 1.1f; implementation lands in Step 2.7
 **Date:** 2026-05-24
 **Deciders:** Core team, ML / Retrieval lead
 **Step:** Phase 1 Step 1.1f (decision); Phase 2 Step 2.7 (implementation)
@@ -57,8 +57,16 @@ Default implementations:
 - **Single-stage SPI + late refactor:** Rejected. The Risk Register already names this exact failure mode (R2). Refactoring an SPI after backends ship is the costly path.
 - **Per-tenant policy selection between single and two-stage:** Deferred to V2. The two-stage default already supports single-stage usage by trivial `fast_rerank`.
 
+## Implementation status
+
+The current `Reranker` SPI in [`packages/core/src/rag_core/spi/reranker.py`](../../packages/core/src/rag_core/spi/reranker.py)
+ships a single-method `rerank(...)` placeholder. Step 2.7 will replace it with the
+two-stage `fast_rerank` / `precise_rerank` / `should_early_exit` shape described above;
+this ADR is the forward-looking contract reviewers should hold the Step 2.7 PR to.
+
 ## References
 
 - [planning/RISK-REGISTER.md](../../planning/RISK-REGISTER.md) R2
+- [docs/architecture/performance.md](../architecture/performance.md) (per-SPI p99 budgets)
 - TRACKER.md Step 2.7
 - ADR-0009 (vector index strategy — companion scale decision)

docs/adr/ADR-0007-tiered-storage.md

@@ -1,9 +1,9 @@
 # ADR-0007 — Tiered storage with `BlobRef`
 
-**Status:** Accepted
+**Status:** Accepted — `BlobRef` shipped in Step 1.1a ([PR #44](https://github.com/officialCodeWork/AgentContextOS/pull/44)); tier-promotion job lands in Phase 6
 **Date:** 2026-05-24
 **Deciders:** Core team, SRE
-**Step:** Phase 1 Step 1.1a (`BlobRef` introduction); Phase 6 (incremental adoption)
+**Step:** Phase 1 Step 1.1a (`BlobRef` introduction); Phase 6 (incremental adoption); finalized in Step 1.1f
 
 ---
 
@@ -58,8 +58,18 @@ Round-trip test: packing a `BlobRef`-backed `Chunk` produces the same output as
 - **Separate `ChunkBody` type holding either text or ref:** Rejected. Adds indirection at every consumer for no semantic benefit.
 - **Always store chunk text in object storage, never inline:** Rejected. Pays cold-tier latency for hot chunks; bad default.
 
+## Implementation
+
+| Surface | Path |
+|---|---|
+| `BlobRef` type | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`BlobRef`) |
+| `Chunk.text: str \| BlobRef` | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`Chunk`) |
+| `Storage` SPI (resolution) | [`packages/core/src/rag_core/spi/storage.py`](../../packages/core/src/rag_core/spi/storage.py) |
+| Reviewer rule ("BlobRef-safe") | [`docs/architecture/performance.md`](../architecture/performance.md) |
+
 ## References
 
 - TRACKER.md Step 1.1a
 - [docs/architecture/storage-backends.md](../architecture/storage-backends.md)
+- [docs/architecture/performance.md](../architecture/performance.md) (reviewer checklist — "BlobRef-safe" rule)
 - ADR-0004 (storage backends)

docs/adr/ADR-0008-cost-aware-planner.md

@@ -1,9 +1,9 @@
 # ADR-0008 — Cost-aware planner replacing reactive fallback
 
-**Status:** Accepted
+**Status:** Accepted — `QueryPlan` / `PlanNode` / `Cost` shipped in Step 1.1a ([PR #44](https://github.com/officialCodeWork/AgentContextOS/pull/44)); planner uses them in Step 2.6+, Step 4.2 refactored downstream
 **Date:** 2026-05-24
 **Deciders:** Core team, ML / Retrieval lead
-**Step:** Phase 1 Step 1.1a (`QueryPlan` type); Phase 4 Step 4.2 (fallback) refactored
+**Step:** Phase 1 Step 1.1a (types); Phase 4 Step 4.2 (fallback) refactored; finalized in Step 1.1f
 
 ---
 
@@ -49,8 +49,21 @@ Initial estimates come from rolling-window observed latency / token / cost histo
 - **Static budgets per tenant tier:** Insufficient — same tenant has different budgets per query (background re-eval vs. interactive).
 - **External planner service:** Rejected for V1. The planner is in-process, called from the gateway.
 
+## Implementation
+
+| Surface | Path |
+|---|---|
+| `Cost` (ms / tokens / dollars) | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`Cost`) |
+| `PlanNode` (with `estimated_cost`) | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`PlanNode`) |
+| `QueryPlan` | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`QueryPlan`) |
+| `Budget` (per-request envelope) | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`Budget.spend()`) |
+| Online cost-estimator input (`StageEvent`) | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`StageEvent`) |
+
+Planner itself (Step 2.6) and fallback-rule refactor (Step 4.2) are tracked separately.
+
 ## References
 
 - TRACKER.md Steps 1.1a, 4.2
 - ADR-0005 (PolicyEngine — runs at planner-time too)
 - [docs/architecture/performance.md](../architecture/performance.md)
+- [docs/architecture/request-context.md](../architecture/request-context.md) (`Budget` ownership)

docs/adr/ADR-0009-vector-index-strategy.md

@@ -1,9 +1,9 @@
 # ADR-0009 — Vector index strategy by scale tier + quantization
 
-**Status:** Accepted
+**Status:** Accepted — `IndexHint`, `Embedding.dtype`, retrieval/index split shipped in Steps 1.1a/1.1b ([PR #44](https://github.com/officialCodeWork/AgentContextOS/pull/44), [PR #45](https://github.com/officialCodeWork/AgentContextOS/pull/45)); per-backend selection lands in Step 2.2
 **Date:** 2026-05-24
 **Deciders:** Core team, ML / Retrieval lead
-**Step:** Phase 1 Step 1.1b (SPI); Phase 2 Step 2.2 (backend implementation)
+**Step:** Phase 1 Step 1.1b (SPI); Phase 2 Step 2.2 (backend implementation); finalized in Step 1.1f
 
 ---
 
@@ -84,6 +84,19 @@ The "ivfflat hardcoded" gap noted in ADR-0004 is resolved by this ADR. `PgVector
 - **Per-backend explicit config (no hint abstraction):** Rejected. Forces tenants to understand backend internals.
 - **Ship int8/binary as a future ADR:** Rejected. Adding `dtype` later is a breaking change to `Embedding`.
 
+## Implementation
+
+| Surface | Path |
+|---|---|
+| `IndexHint` + `WriteVolume` | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) |
+| `Embedding.dtype` | [`packages/core/src/rag_core/types.py`](../../packages/core/src/rag_core/types.py) (`Embedding`) |
+| `VectorIndexBackend` (write surface) | [`packages/core/src/rag_core/spi/vector_store.py`](../../packages/core/src/rag_core/spi/vector_store.py) |
+| `VectorRetrievalBackend` (read surface) | [`packages/core/src/rag_core/spi/vector_store.py`](../../packages/core/src/rag_core/spi/vector_store.py) |
+| Per-backend selection (pgvector) | [`packages/backends/src/rag_backends/vector/pgvector.py`](../../packages/backends/src/rag_backends/vector/pgvector.py) |
+| Per-backend selection (Qdrant) | [`packages/backends/src/rag_backends/vector/qdrant.py`](../../packages/backends/src/rag_backends/vector/qdrant.py) |
+
+`ragctl ann tune` and the per-corpus `dtype` knob in `rag.yaml` land in Step 2.2.
+
 ## References
 
 - ADR-0004 (storage backends — superseded §2)

docs/architecture/performance.md

@@ -128,13 +128,16 @@ Apply to every PR that touches an SPI, a hot path, or an SPI consumer:
 - [ ] **Batcher for external APIs.** New Embedder / Reranker / LLM adapters sit under the `Batcher` middleware (or document why not).
 - [ ] **Budgets honored.** If the change affects a method with a published p99 budget, the conformance suite still passes.
 - [ ] **Cache key correctness.** New caching code uses the right SPI of the three (Embedding / Retrieval / Answer) for its invalidation rule.
-- [ ] **BlobRef-safe.** Code that consumes `Chunk.text` handles both `str` and `BlobRef` (or hydrates explicitly).
+- [ ] **BlobRef-safe.** Code that consumes `Chunk.text` handles both `str` and `BlobRef` (or hydrates explicitly). (ADR-0007)
+- [ ] **Plan + budget aware.** New retrieval / agent code consumes `QueryPlan` + `RequestContext.budget` rather than firing then catching timeouts; expensive nodes carry an `estimated_cost` and emit `StageEvent` for the online cost-estimator. (ADR-0008)
+- [ ] **Index-hint + dtype honored.** New `VectorIndexBackend` code accepts `IndexHint` at `initialize()` and respects `Embedding.dtype` (`float32` / `int8` / `binary`) on both write and read paths — no silent up-casting. (ADR-0009)
+- [ ] **Two-stage rerank shape.** New `Reranker` implementations (Step 2.7+) ship `fast_rerank` (ChunkRef) + `precise_rerank` (Chunk) + `should_early_exit` — not a single-stage `rerank`. (ADR-0006)
 
 ## Related
 
 - [request-context.md](request-context.md) — the per-request envelope.
 - [policy-engine.md](policy-engine.md) — governance PDP.
 - [caching.md](caching.md) — three caches.
 - [pipeline-batcher.md](pipeline-batcher.md) — `Pipeline` and `Batcher` primitives (bounded queues + DataLoader-pattern coalescing).
-- [ADR-0006](../adr/ADR-0006-two-stage-rerank.md), [ADR-0008](../adr/ADR-0008-cost-aware-planner.md), [ADR-0009](../adr/ADR-0009-vector-index-strategy.md) — performance-relevant decisions.
+- [ADR-0005](../adr/ADR-0005-policy-engine.md), [ADR-0006](../adr/ADR-0006-two-stage-rerank.md), [ADR-0007](../adr/ADR-0007-tiered-storage.md), [ADR-0008](../adr/ADR-0008-cost-aware-planner.md), [ADR-0009](../adr/ADR-0009-vector-index-strategy.md) — performance- and governance-relevant decisions.
 - TRACKER.md Steps 1.1a–1.1f (refactor window), 4.6 (latency tuning), 7.1 (load testing).