chore(tracker): sync Step 2.3 → log PR #82 + history rows #77, #82

TRACKER.md

@@ -8,7 +8,7 @@
 > Use `Taskfile.yml` for task targets (cross-platform), `Makefile` for Unix convenience.
 > CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`.
 
-**Last updated:** 2026-05-26 (Step 2.3 — BM25 / keyword retrieval)
+**Last updated:** 2026-05-26 (tracker sync — PR #82)
 **Current phase:** Phase 2 — Retrieval Engine
 **Next action:** Phase 2 Step 2.4 — Graph retrieval: Neo4j, Memgraph, and in-memory NetworkX plugins satisfying the Step 1.1b `GraphRetrievalBackend` / `GraphIndexBackend` SPI; multi-hop neighbor expansion as the canonical traversal primitive; `FilterExpr` push-down on traversal edges (per-edge ACL filtering, no new SPI surface). Step 2.3 added the BM25 keyword pair (`ElasticsearchKeywordStore` + `TantivyKeywordStore`) on top of the Step 2.1 read-layer contract.
 
@@ -94,7 +94,7 @@
 |------|-------|--------|--------|----|-----------------|
 | 2.1 | Knowledge store read layer | ✅ | `build/phase-2/step-2.1-knowledge-store-read-layer` | [#74](https://github.com/officialCodeWork/AgentContextOS/pull/74) | `FilterExpr` AST + in-memory `evaluate()` reference semantics relocated from `rag-policy` into `rag_core.filter` so SPI signatures can take it directly without inverting the dependency graph (`rag_policy.filter` keeps re-exporting every symbol for back-compat). `VectorRetrievalBackend.retrieve_ids` switched from `dict[str, Any]` to typed `FilterExpr \| None`; `KeywordRetrievalBackend.retrieve_ids` gains the same parameter. New SPI signature gate `test_retrieve_ids_accepts_filter_expr` enforces the shape on every retrieval ABC. Noop stores (`NoopVectorStore`, `NoopKeywordStore`) apply the predicate via `evaluate()` against a per-chunk attribute view — the canonical oracle every translator must agree with. `PgVectorStore`: new `_filter_sql.translate()` produces parameterised `WHERE` fragments, `acl_labels TEXT[]` column + GIN index added (idempotent `ADD COLUMN IF NOT EXISTS` migration on `initialize()`), `bulk_index` persists ACL labels from `Embedding.acl_labels`. `QdrantVectorStore`: new `_filter_qdrant.translate()` builds `Filter(must / must_not / should)` merged with existing tenant/corpus `must` clauses, payload gains `acl_labels` on every write. Unsupported fields raise `RetrievalError` (no silent push-down loss). 75 new tests (16 `evaluate` units, 16 SQL translator units, 11 Qdrant translator units, 5 vector contract push-down, 3 keyword contract push-down, 1 signature gate, schema regen). New docs: [architecture/retrieval-read-layer.md](docs/architecture/retrieval-read-layer.md); `reference/rag-core.md` + `reference/rag-policy.md` updated. `rag-core` 0.13.0 → 0.14.0. Schemas regenerated (`Eq.json`, `AnyIn.json`, `And.json`, `Or.json`, `Not.json`, `TrueExpr.json`). |
 | 2.2 | Vector retrieval backends | ✅ | `build/phase-2/step-2.2-vector-retrieval-backends` | [#76](https://github.com/officialCodeWork/AgentContextOS/pull/76) | Three new vector backends behind their own optional extras: `WeaviateVectorStore` (`[weaviate]`, weaviate-client v4 async; single `RagEmbeddings` collection w/ `tenant_id` filter + UUID5 `(tenant, chunk)` IDs; `text[]` `acl_labels` property pushed down via `_filter_weaviate.translate()` that lazy-imports the SDK and composes `Filter.by_property(...).equal/contains_any` with `&`/`\|`/`~`), `PineconeVectorStore` (`[pinecone]`, PineconeAsyncio v5+; namespace=tenant primary isolation primitive plus belt-and-braces `tenant_id` filter; metadata: `tenant_id`/`chunk_id`/`corpus_id`/`model`/`acl_labels`; `_filter_pinecone.translate()` emits MongoDB-style `$eq`/`$in`/`$ne`/`$nin`/`$and`/`$or` and rewrites `Not` via De Morgan since Pinecone has no `$not`), `ElasticsearchDenseVectorStore` (`[elasticsearch]`, AsyncElasticsearch 8.x w/ `dense_vector` + native kNN; single index, `tenant_id` keyword filter, composite `_id="<tenant>::<chunk>"`; `_filter_elasticsearch.translate()` emits query-DSL `bool.filter`/`terms`/`match_all`/`match_none`). All five backends now consume `IndexHint` via the new shared `_index_hint.select_index_variant()` helper that centralises the ADR-0009 §1 size/recall → variant table — pgvector picks ivfflat vs HNSW DDL at `initialize()` time (with a new `_index_ddl_for` helper; PQ tier falls back to HNSW w/ larger `m` since pgvector has no PQ), Qdrant configures `HnswConfigDiff` + scalar `ScalarQuantization(INT8)` for the PQ tier (new `_qdrant_index_config` helper), Weaviate picks `Configure.VectorIndex.{flat,hnsw,hnsw+pq}` via a `_vector_index_config` helper, ES picks `index=false` (flat) vs HNSW `index_options.{m,ef_construction}` per tier, Pinecone records the advisory variant in logs (managed indexes). Each SDK is imported lazily inside `__init__` so importing the module without the extra still succeeds (constructing the class raises a clear `ImportError` pointing at the extra). 53 new tests: 11 `_index_hint` parametrised tier tests, 12 `_filter_pinecone` translator units, 11 `_filter_elasticsearch` translator units, 13 `_filter_weaviate` translator units (SDK stubbed at `sys.modules` so they run without the extra) + 6 integration suites per backend gated on `pytest.importorskip` + a service-reachability fixture (Weaviate via `/v1/.well-known/ready`, ES via `_cluster/health`, Pinecone via `TEST_PINECONE_API_KEY`). Policy-engine coverage allowlist extended for the three new files (the call site, not the engine, consults `PolicyEngine`). New extras + workspace wiring in `packages/backends/pyproject.toml` (`weaviate`/`pinecone`/`elasticsearch`) and mypy overrides in root `pyproject.toml` so CI without the extras stays green. Cross-platform: all three SDKs ship pure-Python or wheel distributions for ubuntu/macos/windows. Deliberately deferred: `ragctl ann tune` (Step 2.6 — query understanding owns the planner-side tuning surface) and per-corpus `dtype: int8` knob in `rag.yaml` (Step 3.5 — corpus router lands rag.yaml → backend wiring). New docs: [reference/backends.md](docs/reference/backends.md) (Weaviate / Pinecone / ES sections + IndexHint mapping tables per backend), [architecture/vector-backends.md](docs/architecture/vector-backends.md) (IndexHint → variant table, FilterExpr translator contracts, tenant-isolation patterns, extension points). |
-| 2.3 | BM25 / keyword retrieval | ✅ | `build/phase-2/step-2.3-bm25-keyword-retrieval` | — | Two new `KeywordStore` backends behind their own optional extras, on top of the Step 2.1 read-layer contract — no new SPI surface, per the Step 2.3 constraint. `ElasticsearchKeywordStore` (`[elasticsearch]`, AsyncElasticsearch 8.x; single `rag-keyword` index for all tenants w/ `tenant_id` keyword filter + composite `_id="<tenant>::<chunk>"`; BM25 over operator-selected analyzer on `body`/`title`/`section_path`; `acl_labels`/`trust_level`/`document_id`/`corpus_id` keyword-filterable; `metadata` stored as `object` w/ `enabled: false` so the mapping doesn't explode; new `rag_backends.keyword._filter_es.translate()` emits ES query-DSL `bool.filter`/`terms`/`match_all`/`match_none` with the broader keyword-side field set vs. the vector translator). `TantivyKeywordStore` (`[tantivy]`, tantivy-py 0.22+; RAM or persistent index; sync API wrapped in `asyncio.to_thread`; composite `_doc_id` field enables idempotent re-indexing via `delete_documents` before `add_document`; `_filter_tantivy.translate()` composes `boolean_query` w/ `term_query` + pairs `MustNot` with `all_query` so `Not(x)` composes stand-alone; metadata stashed as JSON blob for hydrate round-trip). **Per-field boosting is a backend configuration knob, not a per-call SPI parameter** (operator passes `field_boosts={"title": 3.0, "section_path": 2.0, "body": 1.0}` at construction time; ES applies via `multi_match.fields` with `type=best_fields`, tantivy applies via parser's `field_boosts` arg). Same indexed-field schema across both backends so swapping is a config change. SDKs imported lazily so `import rag_backends.keyword.{elasticsearch,tantivy}` without the extra still succeeds. 47 new tests: 13 ES filter-translator units + 14 tantivy filter-translator units (tantivy module stubbed at `sys.modules`) + 9 ES backend units (full SDK stubbed — initialize/bulk_index/search/mget/delete) + 11 tantivy backend units (full SDK stubbed — including idempotency + boost ranking). Policy-engine coverage allowlist extended for the two new files (call site, not the backend, consults `PolicyEngine`). `tantivy>=0.22` added to root `pyproject.toml` mypy overrides. Cross-platform: tantivy-py ships pre-built wheels for ubuntu/macos/windows (no Rust toolchain required at install time); ES SDK pure-Python. Deliberately deferred: rag.yaml → backend wiring (Step 3.5 corpus router), real-time index lag metrics (Step 5.1), custom highlighter / snippet generation (Step 2.8 context packer owns snippet extraction). New docs: [reference/backends.md](docs/reference/backends.md) (ES BM25 + tantivy sections), [architecture/keyword-backends.md](docs/architecture/keyword-backends.md) (per-field boosting as a backend config knob, indexed-field schema, tenant isolation pattern, FilterExpr translator contracts, tantivy `Not` quirk, when to pick which). |
+| 2.3 | BM25 / keyword retrieval | ✅ | `build/phase-2/step-2.3-bm25-keyword-retrieval` | [#82](https://github.com/officialCodeWork/AgentContextOS/pull/82) | Two new `KeywordStore` backends behind their own optional extras, on top of the Step 2.1 read-layer contract — no new SPI surface, per the Step 2.3 constraint. `ElasticsearchKeywordStore` (`[elasticsearch]`, AsyncElasticsearch 8.x; single `rag-keyword` index for all tenants w/ `tenant_id` keyword filter + composite `_id="<tenant>::<chunk>"`; BM25 over operator-selected analyzer on `body`/`title`/`section_path`; `acl_labels`/`trust_level`/`document_id`/`corpus_id` keyword-filterable; `metadata` stored as `object` w/ `enabled: false` so the mapping doesn't explode; new `rag_backends.keyword._filter_es.translate()` emits ES query-DSL `bool.filter`/`terms`/`match_all`/`match_none` with the broader keyword-side field set vs. the vector translator). `TantivyKeywordStore` (`[tantivy]`, tantivy-py 0.22+; RAM or persistent index; sync API wrapped in `asyncio.to_thread`; composite `_doc_id` field enables idempotent re-indexing via `delete_documents` before `add_document`; `_filter_tantivy.translate()` composes `boolean_query` w/ `term_query` + pairs `MustNot` with `all_query` so `Not(x)` composes stand-alone; metadata stashed as JSON blob for hydrate round-trip). **Per-field boosting is a backend configuration knob, not a per-call SPI parameter** (operator passes `field_boosts={"title": 3.0, "section_path": 2.0, "body": 1.0}` at construction time; ES applies via `multi_match.fields` with `type=best_fields`, tantivy applies via parser's `field_boosts` arg). Same indexed-field schema across both backends so swapping is a config change. SDKs imported lazily so `import rag_backends.keyword.{elasticsearch,tantivy}` without the extra still succeeds. 47 new tests: 13 ES filter-translator units + 14 tantivy filter-translator units (tantivy module stubbed at `sys.modules`) + 9 ES backend units (full SDK stubbed — initialize/bulk_index/search/mget/delete) + 11 tantivy backend units (full SDK stubbed — including idempotency + boost ranking). Policy-engine coverage allowlist extended for the two new files (call site, not the backend, consults `PolicyEngine`). `tantivy>=0.22` added to root `pyproject.toml` mypy overrides. Cross-platform: tantivy-py ships pre-built wheels for ubuntu/macos/windows (no Rust toolchain required at install time); ES SDK pure-Python. Deliberately deferred: rag.yaml → backend wiring (Step 3.5 corpus router), real-time index lag metrics (Step 5.1), custom highlighter / snippet generation (Step 2.8 context packer owns snippet extraction). New docs: [reference/backends.md](docs/reference/backends.md) (ES BM25 + tantivy sections), [architecture/keyword-backends.md](docs/architecture/keyword-backends.md) (per-field boosting as a backend config knob, indexed-field schema, tenant isolation pattern, FilterExpr translator contracts, tantivy `Not` quirk, when to pick which). |
 | 2.4 | Graph retrieval | ⏳ | — | — | Neo4j, Memgraph, in-memory NetworkX plugins; multi-hop neighbor expansion |
 | 2.5 | Hybrid RRF fusion | ⏳ | — | — | Reciprocal Rank Fusion across dense+sparse+graph; configurable weights |
 | 2.6 | Query understanding | ⏳ | — | — | Rewriter, decomposer, HyDE, synonym/glossary expansion; `query.understand` span |
@@ -246,6 +246,8 @@
 | [#74](https://github.com/officialCodeWork/AgentContextOS/pull/74) | feat(core,backends): knowledge store read layer + FilterExpr push-down (Step 2.1) | `build/phase-2/step-2.1-knowledge-store-read-layer` | ✅ Merged | 2026-05-25 |
 | [#75](https://github.com/officialCodeWork/AgentContextOS/pull/75) | chore(tracker): sync Step 2.1 → ✅ and log PRs #73, #74 | `chore/tracker-sync-pr74` | ✅ Merged | 2026-05-26 |
 | [#76](https://github.com/officialCodeWork/AgentContextOS/pull/76) | feat(backends): vector retrieval backends — Weaviate / Pinecone / Elasticsearch + IndexHint wiring (Step 2.2) | `build/phase-2/step-2.2-vector-retrieval-backends` | ✅ Merged | 2026-05-26 |
+| [#77](https://github.com/officialCodeWork/AgentContextOS/pull/77) | chore(tracker): sync Step 2.2 → log PR #76 + history rows #75, #76 | `chore/tracker-sync-pr76` | ✅ Merged | 2026-05-26 |
+| [#82](https://github.com/officialCodeWork/AgentContextOS/pull/82) | feat(backends): BM25 keyword retrieval — Elasticsearch + tantivy (Step 2.3) | `build/phase-2/step-2.3-bm25-keyword-retrieval` | ✅ Merged | 2026-05-26 |
 
 ---