Skip to content

TRACKER.md

Latest commit

 

History

History
216 lines (176 loc) · 20.4 KB

TRACKER.md

File metadata and controls

216 lines (176 loc) · 20.4 KB
Raw

AgentContextOS — Build Tracker

How to use: Update Status and PR columns when a step lands on main. In a new session, read this file first to instantly know where we are.

Standing requirement (all steps): Every deliverable — scripts, tooling, CI, dev-stack commands — must work on Windows, macOS, and Linux without WSL. Use Taskfile.yml for task targets (cross-platform), Makefile for Unix convenience. CI matrix always includes ubuntu-22.04, macos-14, and windows-latest.

Last updated: 2026-05-24 Current phase: Phase 1 — Ingestion + Knowledge Store Next action: Phase 1 Step 1.1 — Storage backends

Status Legend

Symbol Meaning
✅ Complete — PR merged to main
🚧 In Progress — branch open, PR exists
⏳ Not Started
🔒 Blocked — depends on incomplete step

Milestone Summary

Phase Title Steps ✅ Done Remaining
0 Foundation 13 13 0
1 Ingestion + Knowledge Store 10 0 10
2 Retrieval Engine 10 0 10
3 Gateway & Agent Runtime 11 0 11
4 Reliability 6 0 6
5 Eval & Observability 7 0 7
6 Governance & Tenancy 10 0 10
7 Pilot, Harden, GA 10 0 10
Total 77 13 64

Phase 0 — Foundation (Weeks 0–4)

Step Title Status Branch PR Key Deliverables
0.1 Monorepo & build system ✅ build/phase-0/step-0.1-monorepo #3 pyproject.toml, Makefile, .pre-commit-config, CI workflows, docker-compose, ADR-0001, lockfiles
0.2 Core domain types & errors ✅ build/phase-0/step-0.2-core-types #4 types.py (11 Pydantic v2 models), errors.py (14-type exception hierarchy), gen_schemas.py (JSON Schema export), proto/core.proto
0.2c Cross-platform tooling ✅ build/phase-0/step-0.2c-cross-platform #6 Taskfile.yml (Windows/Mac/Linux task runner), scripts/bootstrap.ps1, scripts/bootstrap.sh updated, CI matrix expanded to windows-latest
0.3 Plugin SPI ✅ build/phase-0/step-0.3-plugin-spi #7 SPI interfaces in packages/core/src/rag_core/spi/: VectorStore, KeywordStore, GraphStore, Embedder, Reranker, LLM, Cache, Queue, Storage, Secrets, Auth, Telemetry, Connector, Parser, OCR, PIIDetector; noop implementations in spi/noop/; 79 conformance tests in tests/contract/
0.4 rag.yaml schema & loader ✅ build/phase-0/step-0.4-rag-yaml-schema-loader #8 JSON Schema for rag.yaml (dist/rag.schema.json), Pydantic v2 schema (packages/config/), env-var interpolation, ragctl config validate, polling hot-reload, 91 fixture tests, 5 example configs
0.5 CI/CD pipeline hardening ✅ build/phase-0/step-0.5-ci-cd-hardening #9 Path-filtered jobs, dependency caching, Docker layer cache, SBOM generation, cosign image signing, Dependabot
0.6 Local dev stack (Docker Compose) ✅ build/phase-0/step-0.6-local-dev-stack #26 Health-check scripts, seed data loader, make dev-reset, pgvector schema migration, Qdrant collection init
0.7 Tracing & metrics foundation ✅ build/phase-0/step-0.7-tracing-metrics #27 OTel SDK wired, TraceContext propagation, SpiMetrics counter+histogram, Grafana provisioned dashboards, ragctl traces
0.7b Structured logging foundation ✅ build/phase-0/step-0.7b-structured-logging #29, #30 Shared logger package (packages/core/src/rag_core/logging.py), event registry (events.py), RAG001 pre-commit hook + ruff T201 rule, schema+PII CI gates in tests/logs/ (50 tests; log-gates job now blocking). Follow-up (0.7b-observability-package): extracted packages/observability/ (rag-observability) with full 7-field JSON schema (ts, level, service, module, msg, env, version), set_log_context() contextvar-based context manager, _context.py; rag_core.logging and rag_core.events reduced to backwards-compat shims (391 tests pass).
0.7c Audit log skeleton ✅ build/phase-0/step-0.7c-audit-log-skeleton #31 AuditStore SPI (append/events/verify_chain), NoopAuditStore (SHA-256 hash chain), AuditWriter facade (store + structured log), 14 conformance tests
0.8 Eval skeleton ✅ build/phase-0/step-0.8-eval-skeleton #33 rag_core.eval domain types (GoldenSample, EvalMetrics, EvalReport), rag_config.eval metric functions (recall@k, MRR, citation_precision), RagasAdapter spike, ragctl eval run/show, 5-sample golden JSONL fixture, tests/eval/ harness (39 tests)
0.9 IaC foundation ✅ build/phase-0/step-0.9-iac-foundation #34 Terraform modules for Postgres/pgvector, Redis, Qdrant, Elasticsearch (Kubernetes-native, Helm provider); rag-platform Helm chart (Deployment, Service, ConfigMap, ServiceAccount, HPA, PDB, Ingress); dev + prod environments; task infra:* + task helm:* targets; ADR-0003
0.10 ragctl CLI scaffold ✅ build/phase-0/step-0.10-ragctl-cli-scaffold #36 packages/ragctl/ package (Typer 0.12+), root ragctl entry point, working config/eval/traces/version groups, scaffolded ingest/query/logs/tenant/plugin/secret groups (announce target step + exit 0), shell completion via --install-completion/--show-completion; 19 ragctl tests; docs/reference/ragctl.md + docs/guides/ragctl-quickstart.md; py.typed markers added to rag-config + rag-observability

Phase 1 — Ingestion + Knowledge Store (Weeks 4–8)

Step Title Status Branch PR Key Deliverables
1.1 Storage backends ⏳ — — Postgres/pgvector plugin, Qdrant plugin, Redis plugin, S3/GCS/Azure blob plugin; all pass VectorStore conformance suite
1.2 Connectors framework ⏳ — — Connector SPI implementation; built-in: filesystem, S3, GCS; crawler base class
1.3 Document parsers ⏳ — — PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection
1.4 OCR pipeline ⏳ — — Tesseract + PaddleOCR plugins; image region extraction; confidence scoring
1.5 Structure-aware chunker ⏳ — — Heading-based chunking, parent-child Chunk.parent_id links, sentence-boundary, overlap, size normalization
1.6 Metadata enricher ⏳ — — Auto-tag: language, doc type, created/modified dates, author, section path, reading level
1.7 PII detection ⏳ — — Presidio integration; per-tenant policy (block/redact/allow); pii.detected event; CI gate
1.8 Embedder pipeline ⏳ — — OpenAI, Cohere, bge-large, E5 plugins; batching; rate-limit back-off; dimension normalization
1.9 CDC connectors (incremental sync) ⏳ — — Postgres CDC (pgoutput), S3 event notifications, webhook receiver; Document.fingerprint dedup
1.10 Write path & ingest API ⏳ — — Full ingest pipeline wired end-to-end: connector → parse → chunk → enrich → PII → embed → store; POST /v1/ingest

Phase 2 — Retrieval Engine (Weeks 8–12)

Step Title Status Branch PR Key Deliverables
2.1 Knowledge store read layer ⏳ — — Read interfaces for all VectorStore, KeywordStore, GraphStore backends; filter push-down
2.2 Vector retrieval backends ⏳ — — Qdrant, pgvector, Weaviate, Pinecone, Elasticsearch dense-vector plugins
2.3 BM25 / keyword retrieval ⏳ — — Elasticsearch BM25 plugin; tantivy in-process fallback; field boosting
2.4 Graph retrieval ⏳ — — Neo4j, Memgraph, in-memory NetworkX plugins; multi-hop neighbor expansion
2.5 Hybrid RRF fusion ⏳ — — Reciprocal Rank Fusion across dense+sparse+graph; configurable weights
2.6 Query understanding ⏳ — — Rewriter, decomposer, HyDE, synonym/glossary expansion; query.understand span
2.7 Cross-encoder reranker ⏳ — — Cross-encoder + MMR diversity; rerank span with top_k_in/out
2.8 Context packer ⏳ — — Dedup, reordering, conflict detection, token budget enforcement; pack span
2.9 GraphRAG ⏳ — — Community detection, sub-graph summarization, graph-aware retrieval
2.10 Retrieval router ⏳ — — Per-query backend selection; degradation detection; fallback to BM25-only

Phase 3 — Gateway & Agent Runtime (Weeks 12–18)

Step Title Status Branch PR Key Deliverables
3.1 Gateway service (REST + OpenAPI) ⏳ — — FastAPI gateway, /v1/query, /v1/retrieve, /v1/ingest, /v1/corpora, streaming SSE, OpenAPI 3.1
3.2 gRPC service ⏳ — — proto/rag.proto, bidirectional streaming, buf lint gate
3.3 MCP server ⏳ — — apps/gateway/mcp/ exposing retrieve/query/ingest tools; npx @ragplatform/mcp distribution
3.4 OpenAI-compatible endpoints ⏳ — — /v1/embeddings, /v1/chat/completions with retrieval pre-fetch
3.5 Corpus router ⏳ — — Static rules + learned classifier + federated fan-out; routing in trace + audit
3.6 Agent loop ⏳ — — State-machine iterative retrieval; budget controls (iter/token/wall/dollar); tool use; resumability; SSE events
3.7 Official SDKs ⏳ — — Python, TypeScript, .NET, Java, Go SDKs generated from OpenAPI + Proto
3.8 Framework adapters ⏳ — — LangChain, LlamaIndex, Semantic Kernel, Haystack, LangGraph, CrewAI, AutoGen, DSPy
3.9 Webhooks ⏳ — — Outbound webhooks for ingest.completed, drift.detected, eval.regression, audit.policy_violation
3.10 Admin UI v0 ⏳ — — Next.js 14 admin: tenant/corpus mgmt, connector setup, API-key mgmt, glossary editor, audit log, config viewer
3.11 Status & Metrics GUI (MVP) ⏳ — — Home, Live Status, Metrics, Logs tail, Connectors pages; WebSocket + SSE; ≤3-click drilldown

Phase 4 — Reliability (Weeks 18–22)

Step Title Status Branch PR Key Deliverables
4.1 Semantic cache (L0/L1/L2) ⏳ — — In-memory LRU (L0), Redis exact (L1), embedding-similarity (L2); cache-hit events
4.2 Fallback chain ⏳ — — hybrid → BM25-only → keyword → "no answer"; fallback.engaged event
4.3 Hallucination guard ⏳ — — NLI faithfulness scoring per claim; guard.claim_blocked event; per-tenant threshold
4.4 Circuit breakers ⏳ — — Per-backend breaker (closed/open/half-open); breaker.opened event; GUI one-click force-close
4.5 Quotas & rate limiting ⏳ — — Per-tenant QPS, token, cost, storage quotas; Redis-backed sliding window
4.6 Latency tuning & load test ⏳ — — Gateway p99 ≤ 30 ms gate; Locust load test; async pipeline profiling

Phase 5 — Eval & Observability (Weeks 22–28)

Step Title Status Branch PR Key Deliverables
5.1 Per-query tracing & provenance ⏳ — — All pipeline spans with stable attributes; HMAC-signed provenance; GET /v1/query/{id}/trace
5.2 Offline eval harness (golden set) ⏳ — — 500 queries / 5 domains; Recall@k, MRR, nDCG, Faithfulness, Citation Precision metrics; HTML report
5.3 CI eval gate ⏳ — — eval-gate.yml comparing metrics vs tests/eval/baselines/main.json; PR comment with diff table
5.4 Online metrics & feedback ⏳ — — POST /v1/feedback; implicit signals; per-tenant dashboard; PII-redacted feedback store
5.5 Drift monitors ⏳ — — 5 monitors: query distribution, embedding PSI, retrieval score, citation-clickthrough, faithfulness
5.6 Status & Metrics GUI (full build) ⏳ — — All 12 GUI pages complete; Grafana dashboards; regression bisector; drift alerts; cost anomaly; cross-links
5.7 A/B testing & shadow mode ⏳ — — Shadow N% of live queries; A/B routing; statistical analyzer with lift + CI

Phase 6 — Governance & Tenancy (Weeks 28–34)

Step Title Status Branch PR Key Deliverables
6.1 Logical multi-tenancy ⏳ — — Namespace isolation; per-tenant config, quotas, ACLs in rag.yaml
6.2 Physical tenancy (dedicated index) ⏳ — — Dedicated vector index per tenant; cross-tenant probe gate
6.3 ACL push-down at retrieval ⏳ — — ACL filter injected into every vector/BM25/graph query; acl.egress_denied event
6.4 ACL egress verifier ⏳ — — Post-retrieval re-check; defense-in-depth; zero ACL violation rate gate
6.5 PII policies ⏳ — — Per-tenant PII enforcement: block/redact/allow; egress redaction; pii.egress_blocked event
6.6 Immutable audit log ⏳ — — Hash-chain audit log; WORM export; tamper-evident verification; GET /v1/audit
6.7 BYOK (Bring Your Own Key) ⏳ — — KMS integration (AWS KMS, GCP KMS, HashiCorp Vault); envelope encryption for embeddings
6.8 SSO / SCIM ⏳ — — OIDC + SAML IdP federation; SCIM 2.0 user provisioning; per-tenant IdP config
6.9 Air-gapped install bundle ⏳ — — Signed tarball with all images + Helm chart; offline bootstrap; cosign verification
6.10 Compliance posture ⏳ — — SOC 2 Type II control mapping; GDPR data-residency config; data-retention policies

Phase 7 — Pilot, Harden, GA (Weeks 34–40)

Step Title Status Branch PR Key Deliverables
7.1 Load testing ⏳ — — Locust suite; 1000 QPS sustained; p99 < 500 ms; chaos under load
7.2 Chaos engineering ⏳ — — Chaos Monkey / LitmusChaos; kill each backend; verify fallback chain holds
7.3 Red-team ⏳ — — Prompt injection, ACL bypass, PII egress, tenant escape probes
7.4 Design partner onboarding ⏳ — — 2–3 design partners; feedback incorporated; case study documented
7.5 Documentation site ⏳ — — Docusaurus/MkDocs site; API reference generated from OpenAPI; quickstart guides
7.6 Marketplace listings ⏳ — — AWS Marketplace, Azure Marketplace, GCP Marketplace AMI/Helm listings
7.7 Packaging & distribution ⏳ — — PyPI publish (rag-platform), npm publish (@ragplatform/sdk), Docker Hub images
7.8 Support & SLA ⏳ — — Support tiers defined; SLA dashboards; PagerDuty integration; runbooks
7.9 Billing integration ⏳ — — Stripe metered billing; usage export API; invoice generation
7.10 GA cutover ⏳ — — main tag v1.0.0; release notes; all Phase 7 exit gates passed

PR & Branch History

PR Title Branch Status Merged
#1 Problems catalog planning/V1/problems-1 ✅ Merged 2026-05-22
#2 V1 execution plan + logging standard + GUI spec + doc restructure planning/V1/execution-plan ✅ Merged 2026-05-22
#3 Phase 0 Step 0.1 — monorepo and build system build/phase-0/step-0.1-monorepo ✅ Merged 2026-05-22
#4 Phase 0 Step 0.2 — core domain types & errors build/phase-0/step-0.2-core-types ✅ Merged 2026-05-22
#5 Backwards no-op (duplicate of #4) main ❌ Closed 2026-05-22
#6 Phase 0 Step 0.2c — cross-platform support build/phase-0/step-0.2c-cross-platform ✅ Merged 2026-05-22
#7 Phase 0 Step 0.3 — Plugin SPI interfaces build/phase-0/step-0.3-plugin-spi ✅ Merged 2026-05-22
#8 Phase 0 Step 0.4 — rag.yaml schema & loader build/phase-0/step-0.4-rag-yaml-schema-loader ✅ Merged 2026-05-22
#9 Phase 0 Step 0.5 — CI/CD pipeline hardening build/phase-0/step-0.5-ci-cd-hardening ✅ Merged 2026-05-22
#10–#24 Dependabot dependency bumps (GH Actions, Docker, npm, Python dev deps) various ✅ Merged 2026-05-23
#25 chore: release main (Release Please auto-PR) release-please--branches--main ✅ Merged 2026-05-23
#26 Phase 0 Step 0.6 — local dev stack build/phase-0/step-0.6-local-dev-stack ✅ Merged 2026-05-23
#27 feat(core): OTel tracing + SPI metrics foundation (Step 0.7) build/phase-0/step-0.7-tracing-metrics ✅ Merged 2026-05-23
#28 chore: release main (Release Please auto-PR) release-please--branches--main 🟡 Open —
#29 feat(core): structured logging foundation (Step 0.7b) build/phase-0/step-0.7b-structured-logging ✅ Merged 2026-05-23
#30 refactor(observability): extract packages/observability with full log schema (Step 0.7b follow-up) build/phase-0/step-0.7b-observability-package ✅ Merged 2026-05-23
#31 feat(core): audit log skeleton — AuditStore SPI + hash-chain noop + AuditWriter (Step 0.7c) build/phase-0/step-0.7c-audit-log-skeleton ✅ Merged 2026-05-23
#32 docs: add CLAUDE.md with project context and per-PR documentation requirement build/phase-0/step-0.7c-audit-log-skeleton ✅ Merged 2026-05-23
#33 feat(eval): eval skeleton — golden-set schema, metrics, RAGAS spike, ragctl eval (Step 0.8) build/phase-0/step-0.8-eval-skeleton ✅ Merged 2026-05-23
#34 feat(infra): IaC foundation — Terraform modules + Helm chart skeleton (Step 0.9) build/phase-0/step-0.9-iac-foundation ✅ Merged 2026-05-23
#35 chore(tracker): sync PR links for steps 0.7b–0.9 and history #28–#34 chore/tracker-sync-pr28-34 ✅ Merged 2026-05-23
#36 feat(ragctl): consolidated control-plane CLI scaffold (Step 0.10) build/phase-0/step-0.10-ragctl-cli-scaffold ✅ Merged 2026-05-23
#38 ci: wire RAG001 logging check into ci.yml on all OSes fix/ci-rag001-gate ✅ Merged 2026-05-23

Branch Naming Convention

build/phase-{N}/step-{N}.{M}-{short-slug}

Examples:

  • build/phase-0/step-0.2-core-types
  • build/phase-1/step-1.5-chunker
  • build/phase-3/step-3.1-gateway-rest