From b68c29b27adae880d51a16175970c8f632951bce Mon Sep 17 00:00:00 2001 From: Deep Kumar Singh Kushwah Date: Tue, 26 May 2026 16:56:33 +0530 Subject: [PATCH] =?UTF-8?q?chore(tracker):=20sync=20Step=202.5=20=E2=86=92?= =?UTF-8?q?=20=E2=9C=85=20and=20log=20PRs=20#85,=20#86,=20#87,=20#88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Mark Step 2.5 (Hybrid RRF fusion) as complete; PR #86 merged. - Append PRs #85 (Step 2.4 sync), #86 (Step 2.5), #87 + #88 (mypy follow-ups) to history. - Update Last updated / Next action lines. Co-Authored-By: Claude Opus 4.7 (1M context) --- TRACKER.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/TRACKER.md b/TRACKER.md index 2a6a038..1afdf85 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -8,9 +8,9 @@ > Use `Taskfile.yml` for task targets (cross-platform), `Makefile` for Unix convenience. > CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`. -**Last updated:** 2026-05-26 (Step 2.5 in progress — branch open) +**Last updated:** 2026-05-26 (Step 2.5 merged → ✅; PR #86 landed plus mypy follow-ups #87/#88) **Current phase:** Phase 2 — Retrieval Engine -**Next action:** Phase 2 Step 2.6 — Query understanding: rewriter, decomposer, HyDE, synonym / glossary expansion; `query.understand` span. Step 2.5 added `rag-retrieval` with `rrf_fuse` (pure Reciprocal Rank Fusion primitive over already-ranked `ChunkRef` lists) and `HybridRetriever` (async orchestrator fanning out to vector + keyword + graph backends in parallel via `asyncio.gather`, then fusing via RRF with configurable per-source weights and the `k = 60` smoothing constant from Cormack et al. 2009). +**Next action:** Phase 2 Step 2.6 — Query understanding: rewriter, decomposer, HyDE, synonym / glossary expansion; `query.understand` span. Step 2.5 shipped `rag-retrieval` with `rrf_fuse` (pure Reciprocal Rank Fusion primitive over already-ranked `ChunkRef` lists) and `HybridRetriever` (async orchestrator fanning out to vector + keyword + graph backends in parallel via `asyncio.gather`, then fusing via RRF with configurable per-source weights and the `k = 60` smoothing constant from Cormack et al. 2009). **Phase 1 complete — CLI demo milestone delivered.** `ragctl ingest ` runs a folder of mixed-format documents through the full pipeline (connector → parse → chunk → enrich → PII → embed → store) and `POST /v1/ingest/document` exposes the same pipeline over HTTP. Next milestone on the demo ladder is **🥈 Curl-able RAG** at Step 3.1 (gateway query surface), which requires all of Phase 2 first. @@ -96,7 +96,7 @@ | 2.2 | Vector retrieval backends | ✅ | `build/phase-2/step-2.2-vector-retrieval-backends` | [#76](https://github.com/officialCodeWork/AgentContextOS/pull/76) | Three new vector backends behind their own optional extras: `WeaviateVectorStore` (`[weaviate]`, weaviate-client v4 async; single `RagEmbeddings` collection w/ `tenant_id` filter + UUID5 `(tenant, chunk)` IDs; `text[]` `acl_labels` property pushed down via `_filter_weaviate.translate()` that lazy-imports the SDK and composes `Filter.by_property(...).equal/contains_any` with `&`/`\|`/`~`), `PineconeVectorStore` (`[pinecone]`, PineconeAsyncio v5+; namespace=tenant primary isolation primitive plus belt-and-braces `tenant_id` filter; metadata: `tenant_id`/`chunk_id`/`corpus_id`/`model`/`acl_labels`; `_filter_pinecone.translate()` emits MongoDB-style `$eq`/`$in`/`$ne`/`$nin`/`$and`/`$or` and rewrites `Not` via De Morgan since Pinecone has no `$not`), `ElasticsearchDenseVectorStore` (`[elasticsearch]`, AsyncElasticsearch 8.x w/ `dense_vector` + native kNN; single index, `tenant_id` keyword filter, composite `_id="::"`; `_filter_elasticsearch.translate()` emits query-DSL `bool.filter`/`terms`/`match_all`/`match_none`). All five backends now consume `IndexHint` via the new shared `_index_hint.select_index_variant()` helper that centralises the ADR-0009 §1 size/recall → variant table — pgvector picks ivfflat vs HNSW DDL at `initialize()` time (with a new `_index_ddl_for` helper; PQ tier falls back to HNSW w/ larger `m` since pgvector has no PQ), Qdrant configures `HnswConfigDiff` + scalar `ScalarQuantization(INT8)` for the PQ tier (new `_qdrant_index_config` helper), Weaviate picks `Configure.VectorIndex.{flat,hnsw,hnsw+pq}` via a `_vector_index_config` helper, ES picks `index=false` (flat) vs HNSW `index_options.{m,ef_construction}` per tier, Pinecone records the advisory variant in logs (managed indexes). Each SDK is imported lazily inside `__init__` so importing the module without the extra still succeeds (constructing the class raises a clear `ImportError` pointing at the extra). 53 new tests: 11 `_index_hint` parametrised tier tests, 12 `_filter_pinecone` translator units, 11 `_filter_elasticsearch` translator units, 13 `_filter_weaviate` translator units (SDK stubbed at `sys.modules` so they run without the extra) + 6 integration suites per backend gated on `pytest.importorskip` + a service-reachability fixture (Weaviate via `/v1/.well-known/ready`, ES via `_cluster/health`, Pinecone via `TEST_PINECONE_API_KEY`). Policy-engine coverage allowlist extended for the three new files (the call site, not the engine, consults `PolicyEngine`). New extras + workspace wiring in `packages/backends/pyproject.toml` (`weaviate`/`pinecone`/`elasticsearch`) and mypy overrides in root `pyproject.toml` so CI without the extras stays green. Cross-platform: all three SDKs ship pure-Python or wheel distributions for ubuntu/macos/windows. Deliberately deferred: `ragctl ann tune` (Step 2.6 — query understanding owns the planner-side tuning surface) and per-corpus `dtype: int8` knob in `rag.yaml` (Step 3.5 — corpus router lands rag.yaml → backend wiring). New docs: [reference/backends.md](docs/reference/backends.md) (Weaviate / Pinecone / ES sections + IndexHint mapping tables per backend), [architecture/vector-backends.md](docs/architecture/vector-backends.md) (IndexHint → variant table, FilterExpr translator contracts, tenant-isolation patterns, extension points). | | 2.3 | BM25 / keyword retrieval | ✅ | `build/phase-2/step-2.3-bm25-keyword-retrieval` | [#82](https://github.com/officialCodeWork/AgentContextOS/pull/82) | Two new `KeywordStore` backends behind their own optional extras, on top of the Step 2.1 read-layer contract — no new SPI surface, per the Step 2.3 constraint. `ElasticsearchKeywordStore` (`[elasticsearch]`, AsyncElasticsearch 8.x; single `rag-keyword` index for all tenants w/ `tenant_id` keyword filter + composite `_id="::"`; BM25 over operator-selected analyzer on `body`/`title`/`section_path`; `acl_labels`/`trust_level`/`document_id`/`corpus_id` keyword-filterable; `metadata` stored as `object` w/ `enabled: false` so the mapping doesn't explode; new `rag_backends.keyword._filter_es.translate()` emits ES query-DSL `bool.filter`/`terms`/`match_all`/`match_none` with the broader keyword-side field set vs. the vector translator). `TantivyKeywordStore` (`[tantivy]`, tantivy-py 0.22+; RAM or persistent index; sync API wrapped in `asyncio.to_thread`; composite `_doc_id` field enables idempotent re-indexing via `delete_documents` before `add_document`; `_filter_tantivy.translate()` composes `boolean_query` w/ `term_query` + pairs `MustNot` with `all_query` so `Not(x)` composes stand-alone; metadata stashed as JSON blob for hydrate round-trip). **Per-field boosting is a backend configuration knob, not a per-call SPI parameter** (operator passes `field_boosts={"title": 3.0, "section_path": 2.0, "body": 1.0}` at construction time; ES applies via `multi_match.fields` with `type=best_fields`, tantivy applies via parser's `field_boosts` arg). Same indexed-field schema across both backends so swapping is a config change. SDKs imported lazily so `import rag_backends.keyword.{elasticsearch,tantivy}` without the extra still succeeds. 47 new tests: 13 ES filter-translator units + 14 tantivy filter-translator units (tantivy module stubbed at `sys.modules`) + 9 ES backend units (full SDK stubbed — initialize/bulk_index/search/mget/delete) + 11 tantivy backend units (full SDK stubbed — including idempotency + boost ranking). Policy-engine coverage allowlist extended for the two new files (call site, not the backend, consults `PolicyEngine`). `tantivy>=0.22` added to root `pyproject.toml` mypy overrides. Cross-platform: tantivy-py ships pre-built wheels for ubuntu/macos/windows (no Rust toolchain required at install time); ES SDK pure-Python. Deliberately deferred: rag.yaml → backend wiring (Step 3.5 corpus router), real-time index lag metrics (Step 5.1), custom highlighter / snippet generation (Step 2.8 context packer owns snippet extraction). New docs: [reference/backends.md](docs/reference/backends.md) (ES BM25 + tantivy sections), [architecture/keyword-backends.md](docs/architecture/keyword-backends.md) (per-field boosting as a backend config knob, indexed-field schema, tenant isolation pattern, FilterExpr translator contracts, tantivy `Not` quirk, when to pick which). | | 2.4 | Graph retrieval | ✅ | `build/phase-2/step-2.4-graph-retrieval` | [#84](https://github.com/officialCodeWork/AgentContextOS/pull/84) | Three new `GraphStore` backends + a structured multi-hop traversal primitive layered onto the existing SPI surface. New `NeighborResult` Pydantic frozen model in `rag_core.types` (node_id / labels / properties / distance / path). New `expand(ctx, seed_ids, *, hops, rel_types, direction, node_filter, edge_filter, limit) → list[NeighborResult]` method on `GraphRetrievalBackend` — added as a **non-abstract default method** that raises `NotImplementedError`, so the abstract SPI surface is unchanged and existing ABC consumers (NoopGraphStore aside, which we explicitly upgraded) keep compiling. `NoopGraphStore` upgraded with a reference BFS implementation supporting `FilterExpr` node/edge filters — the conformance oracle every concrete backend must agree with. **`FilterExpr` push-down on edges + nodes uses the existing AST — no new types, no new parameter shape.** `edge_filter` applies to **every relationship** along the traversal path (the Step 2.4 headline ACL use case); `node_filter` applies to the **final neighbor only**, mirroring Cypher `WHERE neighbor.foo = X` placed at the path endpoint rather than every intermediate node (intermediate-node filtering would silently drop reachability — almost always wrong). Three backends: `Neo4jGraphStore` (`[neo4j]`, neo4j async driver via Bolt + Cypher), `MemgraphGraphStore` (`[memgraph]`, same neo4j driver — Memgraph speaks the same protocol), `NetworkXGraphStore` (`[networkx]`, pure-Python in-process MultiDiGraph). Neo4j + Memgraph share `BoltCypherStore` so the Cypher composition + parameter handling + tenant isolation + relationship-type validation are written once. Schema convention shared across all three: nodes MERGE'd by `(id, tenant_id)`, labels stored as a property (avoids APOC / dynamic labels), every relationship carries `tenant_id` so multi-hop traversal blocks cross-tenant leaks at every edge, relationship types validated against `^[A-Za-z_][A-Za-z0-9_]*$` + backtick-quoted (Cypher has no relationship-type parameter — interpolation is unavoidable, but injection is not). New `rag_backends.graph._filter_cypher.translate(expr, *, variable, array_fields, param_prefix)` emits parameterised Cypher `WHERE` fragments + a parameter dict; `param_prefix` lets a caller compose multiple translator outputs into one Cypher statement without parameter collisions (used by `expand()` to combine edge + node filters). 65+ new tests: 13 conformance tests for `expand()` on `NoopGraphStore` (the oracle — one-hop / two-hop / seed exclusion / path recording / rel_types / direction / edge & node filters / limit / tenant isolation / hops≤0 / missing seed), 16 Cypher translator units, 21 Neo4j + Memgraph unit tests (neo4j driver stubbed at `sys.modules` — every Cypher statement captured + asserted on for shape + parameter scoping + injection guard), 14 NetworkX backend tests (real networkx pulled via the extra; results compared element-wise against the Noop reference oracle for portability). Policy-engine coverage allowlist extended for the four new backend files (call site, not the backend, consults `PolicyEngine`). `neo4j>=5.20` / `networkx>=3.2` added to root `pyproject.toml` mypy overrides. Schemas regenerated (`NeighborResult.json`). `query()` on `NetworkXGraphStore` deliberately raises `RetrievalError` — NetworkX has no native query language, callers should use `expand()` or reach into the exposed `.graph` MultiDiGraph property for direct NetworkX algorithm calls. Cross-platform: neo4j driver is pure-Python; networkx is pure-Python (~2 MB, no system deps). Deliberately deferred: GraphRAG / community detection (Step 2.9), per-edge weight scoring (Step 2.9), `rag.yaml` → backend wiring (Step 3.5 corpus router), per-tenant separate databases (Step 6.2 physical tenancy). New docs: [reference/backends.md](docs/reference/backends.md) (Neo4j / Memgraph / NetworkX sections including `expand()` Cypher composition + schema convention + extension points), [architecture/graph-backends.md](docs/architecture/graph-backends.md) (why `expand()` is the canonical primitive, tenant-isolation pattern, FilterExpr push-down on edges vs. node, NoopGraphStore as the conformance oracle, when to pick which backend). | -| 2.5 | Hybrid RRF fusion | 🚧 | `build/phase-2/step-2.5-hybrid-rrf-fusion` | — | New `rag-retrieval` package (`packages/retrieval/`, v0.1.0). Pure `rrf_fuse(rankings, *, weights, k=60, top_k=None) → list[ChunkRef]` primitive — score-free fusion using only 1-indexed source ranks via `Σ w_i / (k + rank_i)`; preserves earliest-source metadata; raises `RetrievalError` on cross-tenant inputs (defence-in-depth against upstream wiring bugs). `HybridRetriever` async orchestrator fans out to vector + keyword + graph backends in parallel via `asyncio.gather(return_exceptions=True)`; per-call source routing (a wired backend whose per-call input is `None` is silently skipped — leaves query-shape routing to Step 2.10); per-backend top-k defaults to `max(top_k * 3, 30)` so RRF sees the long tail; partial-failure tolerance (one degraded backend logs `hybrid_retriever.source_failed` + drops out of fusion; only when *all* sources fail does the orchestrator raise `RetrievalError`); canonical `read_chunk` policy call site via `policy_engine.filter_pushdown` `And`-merged with caller-supplied `FilterExpr`. `HybridWeights(vector, keyword, graph)` frozen dataclass (negative-weight validator). `GraphAdapter` async protocol + `default_graph_adapter` treating `node_id == chunk_id` for the chunk-graph projection — `NeighborResult` lists arrive sorted by ascending distance per the Step 2.4 SPI contract, so RRF's rank-based fusion gets the right ordering without re-sorting; richer adapter lands with GraphRAG (Step 2.9). `ragctl hybrid ` smoke command — seeds a 5-document toy corpus + tiny graph, runs all three paths against the noop SPIs, prints fused top-k with per-chunk RRF scores; flags for `--top-k`, `--w-vector` / `--w-keyword` / `--w-graph`, `--rrf-k`. 43 new tests under `tests/retrieval/` (19 pure-fusion units: formula correctness incl. multi-list contribution sum, weight scaling, weight-zero source disabling, k smoothing curve; edge cases: empty input, top_k truncate/zero, deterministic chunk_id tie-break, metadata-from-first-source; tenant safety: mixed-tenant raise; argument validation: invalid k/top_k/weights — plus 5 graph-adapter units and 19 orchestrator integration tests covering single-source happy paths, per-call source skip, fan-out fusion, weighting effects, parallel fan-out wall-time check, partial-source failure, caller-filter pass-through, end-to-end with real `NoopVectorStore` + `NoopKeywordStore` + `NoopGraphStore`) + 5 `ragctl hybrid` CLI tests. PolicyEngine coverage allowlist *not* extended — `HybridRetriever` IS the canonical `read_chunk` call site, not a bypasser. New docs: [reference/retrieval.md](docs/reference/retrieval.md) (rrf_fuse formula + HybridRetriever API + extension points), [architecture/hybrid-fusion.md](docs/architecture/hybrid-fusion.md) (why RRF over score normalisation or learned fusion, `k = 60` justification, graph adapter rationale, what's deferred to Steps 2.6–2.10, reviewer checklist). | +| 2.5 | Hybrid RRF fusion | ✅ | `build/phase-2/step-2.5-hybrid-rrf-fusion` | [#86](https://github.com/officialCodeWork/AgentContextOS/pull/86) | New `rag-retrieval` package (`packages/retrieval/`, v0.1.0). Pure `rrf_fuse(rankings, *, weights, k=60, top_k=None) → list[ChunkRef]` primitive — score-free fusion using only 1-indexed source ranks via `Σ w_i / (k + rank_i)`; preserves earliest-source metadata; raises `RetrievalError` on cross-tenant inputs (defence-in-depth against upstream wiring bugs). `HybridRetriever` async orchestrator fans out to vector + keyword + graph backends in parallel via `asyncio.gather(return_exceptions=True)`; per-call source routing (a wired backend whose per-call input is `None` is silently skipped — leaves query-shape routing to Step 2.10); per-backend top-k defaults to `max(top_k * 3, 30)` so RRF sees the long tail; partial-failure tolerance (one degraded backend logs `hybrid_retriever.source_failed` + drops out of fusion; only when *all* sources fail does the orchestrator raise `RetrievalError`); canonical `read_chunk` policy call site via `policy_engine.filter_pushdown` `And`-merged with caller-supplied `FilterExpr`. `HybridWeights(vector, keyword, graph)` frozen dataclass (negative-weight validator). `GraphAdapter` async protocol + `default_graph_adapter` treating `node_id == chunk_id` for the chunk-graph projection — `NeighborResult` lists arrive sorted by ascending distance per the Step 2.4 SPI contract, so RRF's rank-based fusion gets the right ordering without re-sorting; richer adapter lands with GraphRAG (Step 2.9). `ragctl hybrid ` smoke command — seeds a 5-document toy corpus + tiny graph, runs all three paths against the noop SPIs, prints fused top-k with per-chunk RRF scores; flags for `--top-k`, `--w-vector` / `--w-keyword` / `--w-graph`, `--rrf-k`. 43 new tests under `tests/retrieval/` (19 pure-fusion units: formula correctness incl. multi-list contribution sum, weight scaling, weight-zero source disabling, k smoothing curve; edge cases: empty input, top_k truncate/zero, deterministic chunk_id tie-break, metadata-from-first-source; tenant safety: mixed-tenant raise; argument validation: invalid k/top_k/weights — plus 5 graph-adapter units and 19 orchestrator integration tests covering single-source happy paths, per-call source skip, fan-out fusion, weighting effects, parallel fan-out wall-time check, partial-source failure, caller-filter pass-through, end-to-end with real `NoopVectorStore` + `NoopKeywordStore` + `NoopGraphStore`) + 5 `ragctl hybrid` CLI tests. PolicyEngine coverage allowlist *not* extended — `HybridRetriever` IS the canonical `read_chunk` call site, not a bypasser. New docs: [reference/retrieval.md](docs/reference/retrieval.md) (rrf_fuse formula + HybridRetriever API + extension points), [architecture/hybrid-fusion.md](docs/architecture/hybrid-fusion.md) (why RRF over score normalisation or learned fusion, `k = 60` justification, graph adapter rationale, what's deferred to Steps 2.6–2.10, reviewer checklist). | | 2.6 | Query understanding | ⏳ | — | — | Rewriter, decomposer, HyDE, synonym/glossary expansion; `query.understand` span | | 2.7 | Cross-encoder reranker | ⏳ | — | — | Cross-encoder + MMR diversity; `rerank` span with top_k_in/out | | 2.8 | Context packer | ⏳ | — | — | Dedup, reordering, conflict detection, token budget enforcement; `pack` span | @@ -250,6 +250,10 @@ | [#82](https://github.com/officialCodeWork/AgentContextOS/pull/82) | feat(backends): BM25 keyword retrieval — Elasticsearch + tantivy (Step 2.3) | `build/phase-2/step-2.3-bm25-keyword-retrieval` | ✅ Merged | 2026-05-26 | | [#83](https://github.com/officialCodeWork/AgentContextOS/pull/83) | chore(tracker): sync Step 2.3 → log PR #82 + history rows #77, #82 | `chore/tracker-sync-pr82` | ✅ Merged | 2026-05-26 | | [#84](https://github.com/officialCodeWork/AgentContextOS/pull/84) | feat(core,backends): graph retrieval — Neo4j + Memgraph + NetworkX + expand() (Step 2.4) | `build/phase-2/step-2.4-graph-retrieval` | ✅ Merged | 2026-05-26 | +| [#85](https://github.com/officialCodeWork/AgentContextOS/pull/85) | chore(tracker): sync Step 2.4 → log PR #84 + history rows #83, #84 | `chore/tracker-sync-pr84` | ✅ Merged | 2026-05-26 | +| [#86](https://github.com/officialCodeWork/AgentContextOS/pull/86) | feat(retrieval): hybrid RRF fusion — rrf_fuse + HybridRetriever (Step 2.5) | `build/phase-2/step-2.5-hybrid-rrf-fusion` | ✅ Merged | 2026-05-26 | +| [#87](https://github.com/officialCodeWork/AgentContextOS/pull/87) | fix(tooling): unblock mypy by removing duplicate package-local tests/__init__.py | `fix/mypy-duplicate-tests-module` | ✅ Merged | 2026-05-26 | +| [#88](https://github.com/officialCodeWork/AgentContextOS/pull/88) | fix(retrieval): cast policy_filter to restore FilterExpr across lazy boundary | `fix/mypy-no-any-return-hybrid-policy-filter` | ✅ Merged | 2026-05-26 | ---