feat(core): Pipeline + Batcher primitives (Step 1.1d)

[officialCodeWork]

Summary

• Adds Pipeline (rag_core.pipeline) — composable async DAG with bounded queues, per-stage worker counts, and natural backpressure. Joiner tasks per stage transition cleanly handle different upstream/downstream worker counts; on_error="fail" / "skip" policies; optional error_handler callback.
• Adds Batcher[Req, Resp] (rag_core.batcher) — DataLoader-pattern middleware that coalesces concurrent load(req) calls into one batched batch_fn(reqs) invocation. Size trigger (max_batch_size) + time trigger (max_wait_ms, default 5 ms). Per-request asyncio.Future for isolated error propagation; flush() for deterministic shutdown.
• Both use PEP-695 generic syntax. CancelledError is intentionally allowed to propagate everywhere so structured cancellation tears the pipeline down cleanly.
• rag-core 0.5.0 → 0.6.0.

Step 1.1d of the Phase 1 refactor window (1.1a–1.1f). Unblocks Step 1.10 (write path), Step 1.2 (connectors framework), and the future Embedder / Reranker / LLM adapters that will sit under the Batcher per the performance.md reviewer checklist.

Tests

• 22 new tests: 8 covering Batcher (coalescing, size/time triggers, error propagation, length-mismatch, flush, validation) and 14 covering Pipeline (one-to-one, none-drops, list fan-out, multi-stage chain, per-stage concurrency, bounded-queue backpressure, fail/skip error policies, source-exception surfacing, duplicate-name rejection, Stage validation, default queue sizing, early-break cleanup).
• Full repo: 549 passed, 1 skipped.
• task lint clean; mypy --strict clean for new files (two pre-existing rag_observability import-resolution warnings in rag_core.logging / rag_core.events shims are unchanged from main).

Documentation

Per the per-PR docs requirement (CLAUDE.md):

• New: docs/architecture/pipeline-batcher.md — design, usage, internals, error semantics, extension points (public + technical sections).
• Updated: docs/reference/rag-core.md (Concurrency primitives section), docs/README.md (architecture + reference index entries), docs/architecture/performance.md (Related cross-link).

Tracker

• Step 1.1d moved ⏳ → 🚧 with branch + deliverables filled in.
• Next action set to Step 1.1e (Cache SPI split + perf discipline + async telemetry).
• PR chore(tracker): sync PR links for steps 1.1a/1.1c + log #46–#48 #49 logged in the PR history table.

Test plan

• uv run pytest packages/core/tests/test_batcher.py packages/core/tests/test_pipeline.py -x
• uv run pytest tests/ packages/ -k "not integration" (549 passed)
• uv run ruff check . clean
• uv run mypy packages/core/ apps/gateway/ clean for new files
• Schema regeneration (PYTHONPATH=packages/core/src:packages/observability/src uv run python -m rag_core.gen_schemas dist/schemas/) — no drift

🤖 Generated with Claude Code