From 17c472aa281c0abd23e1b8732bded9102d57d493 Mon Sep 17 00:00:00 2001 From: Deep Kumar Singh Kushwah Date: Mon, 25 May 2026 22:51:49 +0530 Subject: [PATCH] =?UTF-8?q?chore(tracker):=20sync=20Step=201.8=20=E2=86=92?= =?UTF-8?q?=20=E2=9C=85=20and=20log=20PRs=20#67,=20#68;=20advance=20to=201?= =?UTF-8?q?.10?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Flip Step 1.8 (embedder pipeline) from ⏳ to ✅; capture full deliverable (rag-embedders package, 3 plugins / 4 model families, shared BatchingEmbedder + RetryPolicy, dim normalization, ragctl embed, docs). - Phase 1 progress: 13/16 → 14/16 done (2 remaining) - Total: 26/84 → 27/84 done - Advance Next-action straight to Step 1.10 (write path + ingest API) — the CLI demo milestone. Step 1.9 (CDC connectors) is deferred to a follow-up since it's about streaming-source incremental sync, not needed for the one-shot folder-ingest demo. - Add PRs #67 (Step 1.7 tracker sync) and #68 (Step 1.8) to history. Co-Authored-By: Claude Opus 4.7 (1M context) --- TRACKER.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/TRACKER.md b/TRACKER.md index 5b945c9..9e8e6a6 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -10,7 +10,7 @@ **Last updated:** 2026-05-25 **Current phase:** Phase 1 — Ingestion + Knowledge Store -**Next action:** Phase 1 Step 1.8 — Embedder pipeline: OpenAI, Cohere, bge-large, E5 plugins; provider batching with rate-limit back-off; dimension normalization. Last piece before Step 1.10 wires the full ingest API together (the "CLI demo" milestone). +**Next action:** Phase 1 Step 1.10 — Write path & ingest API: wire the full pipeline end-to-end (connector → parse → chunk → enrich → PII → embed → store) and expose `POST /v1/ingest`. This is the **CLI demo milestone** — end-to-end ingest works from a folder of documents. Step 1.9 (CDC connectors) is deferred to a follow-up; it's about streaming-source incremental sync, which isn't on the critical path for the demo. > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md]. @@ -32,14 +32,14 @@ | Phase | Title | Steps | ✅ Done | Remaining | |-------|-------|------:|-------:|----------:| | 0 | Foundation | 13 | **13** | 0 | -| 1 | Ingestion + Knowledge Store | 16 | **13** | 3 | +| 1 | Ingestion + Knowledge Store | 16 | **14** | 2 | | 2 | Retrieval Engine | 11 | 0 | 11 | | 3 | Gateway & Agent Runtime | 11 | 0 | 11 | | 4 | Reliability | 6 | 0 | 6 | | 5 | Eval & Observability | 7 | 0 | 7 | | 6 | Governance & Tenancy | 10 | 0 | 10 | | 7 | Pilot, Harden, GA | 10 | 0 | 10 | -| **Total** | | **84** | **26** | **58** | +| **Total** | | **84** | **27** | **57** | --- @@ -80,7 +80,7 @@ | 1.5 | Structure-aware chunker | ✅ | `build/phase-1/step-1.5-structure-aware-chunker` | [#62](https://github.com/officialCodeWork/AgentContextOS/pull/62) | New `rag-chunker` package (`packages/chunker/`, v0.1.0): `HeadingAwareChunker` walks `ParsedDocument.blocks`, maintains a heading-level stack for `parent_id` semantics (h2 under h1 → parent=h1; new h2 sibling reparents under the same h1), accumulates body blocks under each heading up to a token budget (default 512 tokens, 64 overlap), splits oversized blocks on sentence boundaries (pure-Python regex), hard-splits sentences that still exceed the budget. Within-section overlap only; never bleeds across heading boundaries. Chunker SPI added: `rag_core.spi.chunker.Chunker` ABC (`async chunk(ctx, parsed) -> list[Chunk]`); `NoopChunker` one-chunk-per-block fallback; 7 contract tests + signature linter extension enforcing monotonic `position`, parent-refs-earlier-only, tenant/document propagation. `TokenCounter` Protocol + `TiktokenCounter` (default `cl100k_base`, swap encodings via constructor); tiktoken pulled in as default dep with wheels for all OS+arch combos. `ocr_result_to_parsed_document()` adapter turns Step 1.4 `OCRResult` into a synthetic `ParsedDocument` so OCR'd image pages flow through the same chunker (each `OCRRegion` → paragraph `Block` with `engine`/`ocr_confidence`/`bbox`/`page` in attributes). `ragctl chunk ` smoke command (`--max-tokens`, `--overlap-tokens`, `--mime`, `--chunks N`) — 5 CLI tests. 33 unit tests under `tests/chunker/` (sentence splitter + tokenizer + heading-aware + OCR adapter). `rag-core` 0.9.0 → 0.10.0. Cross-platform: pure-Python except `tiktoken` extension; unit tests use a `_WordCounter` stand-in so token-boundary assertions don't depend on BPE specifics. New docs: [reference/chunker.md](docs/reference/chunker.md), [architecture/chunker.md](docs/architecture/chunker.md). | | 1.6 | Metadata enricher | ✅ | `build/phase-1/step-1.6-metadata-enricher` | [#64](https://github.com/officialCodeWork/AgentContextOS/pull/64) | New `rag-enricher` package (`packages/enricher/`, v0.1.0): `DefaultEnricher` tags each chunk's metadata with `language` (langdetect, deterministic seed; skipped < 20 chars), `doc_type` (from `Document.mime_type` via `doc_type_from_mime()` — markdown/pdf/docx/pptx/xlsx/json/csv/yaml/html/text/ocr or first-of-mime fallback), `created_at`/`modified_at` (parser-extracted dates from `Document.metadata` override `Document.created_at`/`updated_at`), `author` and `title` when present, `source_uri`, `document_id`, `section_path` (root-first heading-text list from walking `parent_id` chain via the new `section_path()` helper; O(d) via per-doc lookup table; cycle-guarded), and `reading_level` (Flesch-Kincaid grade via textstat — gated on body chunks + English + ≥80 chars so we skip rather than emit a misleading "grade 0"). Enricher SPI added: `rag_core.spi.enricher.Enricher` ABC (`async enrich(ctx, chunks, parsed) -> list[Chunk]`); `NoopEnricher` passthrough; 4 contract tests + signature linter extension enforce length/order/identifying-fields preserved (only `metadata` differs). `LanguageDetector` Protocol + `LangdetectDetector` default (seeded `DetectorFactory.seed=0` so cache keys derived from chunk metadata are stable across runs); swappable via constructor. `ragctl enrich ` smoke command pipes parse → chunk → enrich and prints tag-coverage summary + first-N enriched chunks with language / reading level / section path — 4 CLI tests. 30 unit tests under `tests/enricher/` (doc_type, language, section_path, default_enricher integration). `rag-core` 0.10.0 → 0.11.0. Cross-platform: langdetect + textstat + pyphen all ship pure-Python wheels — no system binaries, no native extensions. New docs: [reference/enricher.md](docs/reference/enricher.md), [architecture/enricher.md](docs/architecture/enricher.md). | | 1.7 | PII detection | ✅ | `build/phase-1/step-1.7-pii-detection` | [#66](https://github.com/officialCodeWork/AgentContextOS/pull/66) | New `rag-pii` package (`packages/pii/`, v0.1.0): `RegexPIIDetector` (pure-Python, always available — EMAIL/PHONE/SSN/CREDIT_CARD/IP_ADDRESS with type-specific confidence scores and Luhn-validated credit cards; regex patterns mirror `rag_observability.events.check_pii` so the Step 0.7b log-leak CI gate and ingest detection share one definition of PII) + `PresidioPIIDetector` (optional `[presidio]` extra; lazy `AnalyzerEngine` init; overlap resolution matches the regex path). `PiiProcessor` applies `ctx.pii_policy.action` across the chunk list — **redact** (default, `` markers), **mask** (`*` of equal length, preserves layout), **block** (chunk dropped) — pre-filters spans by `policy.entities` + `policy.min_score`, and returns `(filtered_chunks, outcomes)` so callers see exactly what happened per chunk without re-running the detector. Pure helpers `redact_spans()` / `mask_spans()` exported for custom flows. Event protocol: one `pii.detected` `PiiEvent` per entity type per affected chunk, carrying only metadata (`field_name=chunk:`, `pii_type`, `action_taken`, plus `chunk_id`/`document_id`/`principal_id`/`span_count` in `metadata`) — never raw matched text, so the existing log-leak CI gate keeps passing. SPI evolution: `PIISpan` promoted from dataclass to Pydantic v2 frozen model in `rag_core.types` (validates `end >= start` + 0–1 score); schema exported (`PIISpan.json`). `ragctl pii ` smoke command (`--detector regex\|presidio`, `--action redact\|mask\|block`, `--chunks N`) — 5 CLI tests. 35 unit tests under `tests/pii/` (rewriters, regex incl. parametrised Luhn, processor, Presidio stubbed at `sys.modules`). 8 contract tests cover SPI + PIISpan validation. `rag-core` 0.11.0 → 0.12.0. Cross-platform: core pure-Python; `[presidio]` extra needs `spacy download en_core_web_sm` separately. New docs: [reference/pii.md](docs/reference/pii.md), [architecture/pii.md](docs/architecture/pii.md). | -| 1.8 | Embedder pipeline | ⏳ | — | — | OpenAI, Cohere, bge-large, E5 plugins; batching; rate-limit back-off; dimension normalization | +| 1.8 | Embedder pipeline | ✅ | `build/phase-1/step-1.8-embedder-pipeline` | [#68](https://github.com/officialCodeWork/AgentContextOS/pull/68) | New `rag-embedders` package (`packages/embedders/`, v0.1.0): **3 plugin classes covering 4 model families** behind the existing `Embedder` SPI. `OpenAIEmbedder` (`[openai]` extra) covers `text-embedding-3-{small,large}` + `ada-002` with server-side dim reduction for v3 models via the `dimensions` request parameter; `CohereEmbedder` (`[cohere]` extra) covers `embed-{english,multilingual}-v3.0` with SDK v4/v5 response-shape probing and `search_document` default `input_type`; `SentenceTransformersEmbedder` (`[sentence-transformers]` extra) covers any HuggingFace model, with `bge_large_en()` and `e5_large_v2()` factory helpers — E5 passage prefix auto-applied via `_PASSAGE_PREFIXES`; encoding runs in `asyncio.to_thread` so the sync API doesn't block the loop. All three subclass `BatchingEmbedder` for shared sub-batch splitting (OpenAI 2048 / Cohere 96 / local 32), exponential-back-off retry via `RetryPolicy` (`retry_on` tuple declares which provider exceptions are transient), and dimension normalization (truncate + L2 renorm). `normalize_dimension()` exposed as a pure helper. `ragctl embed ` smoke command — default `noop` backend exercises the full parse → chunk → enrich → PII → embed pipeline without provider credentials; `--backend openai\|cohere\|bge\|e5` selects real backends. 41 unit tests under `tests/embedders/` (dim + retry + base + 3 backend test files with each provider stubbed at `sys.modules`); 5 CLI tests. PolicyEngine coverage linter allowlist extended to cover `ragctl/main.py` (operator CLI, not production policy boundary). Cross-platform: core pure-Python; `[sentence-transformers]` extra pulls in PyTorch (~600 MB) but ships wheels for ubuntu/macos/windows. New docs: [reference/embedders.md](docs/reference/embedders.md), [architecture/embedders.md](docs/architecture/embedders.md). | | 1.9 | CDC connectors (incremental sync) | ⏳ | — | — | Postgres CDC (pgoutput), S3 event notifications, webhook receiver; `Document.fingerprint` dedup | | 1.10 | Write path & ingest API | ⏳ | — | — | Full ingest pipeline wired end-to-end: connector → parse → chunk → enrich → PII → embed → store; `POST /v1/ingest` | @@ -234,6 +234,8 @@ | [#64](https://github.com/officialCodeWork/AgentContextOS/pull/64) | feat(core,enricher): metadata enricher (Step 1.6) | `build/phase-1/step-1.6-metadata-enricher` | ✅ Merged | 2026-05-25 | | [#65](https://github.com/officialCodeWork/AgentContextOS/pull/65) | chore(tracker): sync Step 1.6 → ✅ and log PRs #63, #64 | `chore/tracker-sync-pr64` | ✅ Merged | 2026-05-25 | | [#66](https://github.com/officialCodeWork/AgentContextOS/pull/66) | feat(core,pii): PII detection + per-tenant policy enforcement (Step 1.7) | `build/phase-1/step-1.7-pii-detection` | ✅ Merged | 2026-05-25 | +| [#67](https://github.com/officialCodeWork/AgentContextOS/pull/67) | chore(tracker): sync Step 1.7 → ✅ and log PRs #65, #66 | `chore/tracker-sync-pr66` | ✅ Merged | 2026-05-25 | +| [#68](https://github.com/officialCodeWork/AgentContextOS/pull/68) | feat(embedders): embedder pipeline — OpenAI / Cohere / BGE / E5 (Step 1.8) | `build/phase-1/step-1.8-embedder-pipeline` | ✅ Merged | 2026-05-25 | ---