diff --git a/TRACKER.md b/TRACKER.md index 5630525..6983758 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -10,7 +10,7 @@ **Last updated:** 2026-05-24 **Current phase:** Phase 1 — Ingestion + Knowledge Store -**Next action:** Phase 1 Step 1.3 — Document parsers: PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers with MIME detection. The Connectors framework (1.2) lands the discovery/fetch side; 1.3 fills in the parse step that turns raw bytes into structured chunks. +**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring. (Step 1.3 — Document parsers — is in progress: see PR for `build/phase-1/step-1.3-document-parsers`.) > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md]. @@ -75,7 +75,7 @@ | 1.1e | Cache SPI split + perf discipline + async telemetry | ✅ | `build/phase-1/step-1.1e-cache-split-perf-telemetry` | [#51](https://github.com/officialCodeWork/AgentContextOS/pull/51) | `EmbeddingCache` (`rag_core.spi.embedding_cache`, key tenant+model_id+model_version+text_hash, `invalidate_model`), `RetrievalCache` (`rag_core.spi.retrieval_cache`, key tenant+plan_hash+corpus_version, `invalidate_corpus`), `AnswerCache` (`rag_core.spi.answer_cache`, key tenant+plan_hash+corpus_version+policy_version, `invalidate_policy`). All ctx-first; signature linter extended. Noop in-memory impls + 18 conformance tests covering hit/miss/version-partition/invalidate/tenant-isolation. `AsyncTelemetrySink` in `rag-observability` — bounded buffer (default 10_000), non-blocking `submit()`, per-kind drop + exporter-error counters, idempotent start/stop, drain-timeout-cancel; 9 unit tests. Hot-path discipline doc in `performance.md` expanded with concrete hot/cold call-site table. New `docs/reference/rag-observability.md`. `caching.md` SPI shapes updated with ctx-first signatures. `rag-core` 0.6.0 → 0.7.0. | | 1.1f | ADRs 0005–0009 + reviewer checklist | ✅ | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). | | 1.2 | Connectors framework | ✅ | `build/phase-1/step-1.2-connectors-framework` | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | `Connector` SPI evolved to `(Document, ConnectorState)` async-iterator with resumable watermark (`ConnectorState` frozen model: connector_id, tenant_id, cursor, last_seen_at, extra). Internal `Crawler[Raw]` base class in `rag_backends.connectors._base` (iteration scaffold, error policy, monotonic watermark). Built-in connectors: `FilesystemConnector` (cross-platform pathlib, sha256 hashing, sorted resume cursor), `S3Connector` (aioboto3, ListObjectsV2 continuation token, MinIO-compatible), `GCSConnector` (gcloud-aio-storage, REST nextPageToken, `[gcs]` extra). Policy boundary documented — connectors are upstream of `PolicyEngine`; the ingest pipeline (Step 1.10) is the enforcement point. 9 contract tests (NoopConnector resume + watermark + tenant mismatch), 9 unit tests (FilesystemConnector), MinIO + GCS integration tests skip-if-no-service. `ConnectorState` schema added to `dist/schemas/`. New docs: [reference/connectors.md](docs/reference/connectors.md), [architecture/connectors.md](docs/architecture/connectors.md). | -| 1.3 | Document parsers | ⏳ | — | — | PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection | +| 1.3 | Document parsers | 🚧 | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse ` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). | | 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring | | 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization | | 1.6 | Metadata enricher | ⏳ | — | — | Auto-tag: language, doc type, created/modified dates, author, section path, reading level | diff --git a/dist/schemas/Block.json b/dist/schemas/Block.json new file mode 100644 index 0000000..7ea5168 --- /dev/null +++ b/dist/schemas/Block.json @@ -0,0 +1,57 @@ +{ + "$defs": { + "BlockType": { + "description": "Coarse structural type of a parsed block.\n\nEmitted by `Parser` implementations (Step 1.3) so the structure-aware\nchunker (Step 1.5) can find natural boundaries without re-parsing. The\ntaxonomy is deliberately small \u2014 parsers map library-native node types\nonto one of these labels.", + "enum": [ + "heading", + "paragraph", + "list_item", + "table_row", + "code", + "quote", + "caption", + "other" + ], + "title": "BlockType", + "type": "string" + } + }, + "description": "One structural unit produced by a Parser (Step 1.3).\n\nBlocks preserve enough document structure for the heading-aware chunker\n(Step 1.5) to find natural boundaries without re-parsing the source\nbytes. Each block carries the extracted text, its 0-based start offset\ninto ``ParsedDocument.text`` (the end is ``start + len(text)``), an\noptional 1-6 heading level (only for ``BlockType.heading``), and a\nfree-form attributes dict for parser-specific extras such as\n``{\"page\": 3}`` for PDFs or ``{\"sheet\": \"Q1\", \"row\": 5}`` for XLSX.", + "properties": { + "type": { + "$ref": "#/$defs/BlockType" + }, + "text": { + "title": "Text", + "type": "string" + }, + "start": { + "title": "Start", + "type": "integer" + }, + "level": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Level" + }, + "attributes": { + "additionalProperties": true, + "title": "Attributes", + "type": "object" + } + }, + "required": [ + "type", + "text", + "start" + ], + "title": "Block", + "type": "object" +} diff --git a/dist/schemas/ParsedDocument.json b/dist/schemas/ParsedDocument.json new file mode 100644 index 0000000..0622053 --- /dev/null +++ b/dist/schemas/ParsedDocument.json @@ -0,0 +1,220 @@ +{ + "$defs": { + "ACL": { + "description": "Single access-control entry: who can do what on which resource.", + "properties": { + "principal_id": { + "title": "Principal Id", + "type": "string" + }, + "action": { + "$ref": "#/$defs/ACLAction" + }, + "resource": { + "title": "Resource", + "type": "string" + } + }, + "required": [ + "principal_id", + "action", + "resource" + ], + "title": "ACL", + "type": "object" + }, + "ACLAction": { + "enum": [ + "read", + "write", + "delete", + "admin" + ], + "title": "ACLAction", + "type": "string" + }, + "Block": { + "description": "One structural unit produced by a Parser (Step 1.3).\n\nBlocks preserve enough document structure for the heading-aware chunker\n(Step 1.5) to find natural boundaries without re-parsing the source\nbytes. Each block carries the extracted text, its 0-based start offset\ninto ``ParsedDocument.text`` (the end is ``start + len(text)``), an\noptional 1-6 heading level (only for ``BlockType.heading``), and a\nfree-form attributes dict for parser-specific extras such as\n``{\"page\": 3}`` for PDFs or ``{\"sheet\": \"Q1\", \"row\": 5}`` for XLSX.", + "properties": { + "type": { + "$ref": "#/$defs/BlockType" + }, + "text": { + "title": "Text", + "type": "string" + }, + "start": { + "title": "Start", + "type": "integer" + }, + "level": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Level" + }, + "attributes": { + "additionalProperties": true, + "title": "Attributes", + "type": "object" + } + }, + "required": [ + "type", + "text", + "start" + ], + "title": "Block", + "type": "object" + }, + "BlockType": { + "description": "Coarse structural type of a parsed block.\n\nEmitted by `Parser` implementations (Step 1.3) so the structure-aware\nchunker (Step 1.5) can find natural boundaries without re-parsing. The\ntaxonomy is deliberately small \u2014 parsers map library-native node types\nonto one of these labels.", + "enum": [ + "heading", + "paragraph", + "list_item", + "table_row", + "code", + "quote", + "caption", + "other" + ], + "title": "BlockType", + "type": "string" + }, + "Document": { + "description": "A single ingested source document, before chunking.", + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "tenant_id": { + "title": "Tenant Id", + "type": "string" + }, + "corpus_id": { + "title": "Corpus Id", + "type": "string" + }, + "source_uri": { + "title": "Source Uri", + "type": "string" + }, + "content_hash": { + "title": "Content Hash", + "type": "string" + }, + "title": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Title" + }, + "mime_type": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Mime Type" + }, + "status": { + "$ref": "#/$defs/DocumentStatus", + "default": "pending" + }, + "acls": { + "items": { + "$ref": "#/$defs/ACL" + }, + "title": "Acls", + "type": "array" + }, + "metadata": { + "additionalProperties": true, + "title": "Metadata", + "type": "object" + }, + "created_at": { + "format": "date-time", + "title": "Created At", + "type": "string" + }, + "updated_at": { + "format": "date-time", + "title": "Updated At", + "type": "string" + } + }, + "required": [ + "tenant_id", + "corpus_id", + "source_uri", + "content_hash" + ], + "title": "Document", + "type": "object" + }, + "DocumentStatus": { + "enum": [ + "pending", + "processing", + "ready", + "failed" + ], + "title": "DocumentStatus", + "type": "string" + } + }, + "description": "A ``Document`` plus its extracted text and structural blocks.\n\nReturned by ``Parser.parse``. The chunker (Step 1.5) consumes\n``blocks`` for structure-aware boundary detection; parsers that cannot\nrecover structure (plain text, opaque blobs) emit a single\n``BlockType.paragraph`` block spanning the full text.\n\n``detected_mime`` carries the parser's normalized MIME type, which may\ndiffer from the caller-supplied hint when sniffing reveals a more\nspecific content type (e.g. a ``.txt`` file that is actually JSON).", + "properties": { + "document": { + "$ref": "#/$defs/Document" + }, + "text": { + "title": "Text", + "type": "string" + }, + "blocks": { + "items": { + "$ref": "#/$defs/Block" + }, + "title": "Blocks", + "type": "array" + }, + "detected_mime": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Detected Mime" + } + }, + "required": [ + "document", + "text" + ], + "title": "ParsedDocument", + "type": "object" +} diff --git a/docs/README.md b/docs/README.md index 2612b2e..26beeff 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,6 +15,7 @@ | [iac.md](architecture/iac.md) | IaC overview: Terraform module design, Helm chart structure, dev/prod environments, extension points | | [storage-backends.md](architecture/storage-backends.md) | Storage backend architecture: PgVector, Qdrant, Redis, S3/MinIO, tenant isolation, integration test strategy | | [connectors.md](architecture/connectors.md) | Connectors framework: SPI shape, `ConnectorState` watermark, `Crawler` base class, policy boundary, CDC interaction | +| [parsers.md](architecture/parsers.md) | Document parsers: `Block` taxonomy, `ParsedDocument` return shape, MIME detection, policy boundary, library choices | ## reference/ @@ -26,6 +27,7 @@ | [rag-observability.md](reference/rag-observability.md) | `rag-observability` — structured logger, event registry, `AsyncTelemetrySink` (bounded, drop-on-overflow) | | [rag-policy.md](reference/rag-policy.md) | `rag-policy` reference — `PolicyEngine`, `PolicyWriter`, `PolicyDecision`, `PolicyResult`, `FilterExpr` | | [connectors.md](reference/connectors.md) | `rag-backends` connectors — `FilesystemConnector`, `S3Connector`, `GCSConnector`, resumable crawls via `ConnectorState` | +| [parsers.md](reference/parsers.md) | `rag-parsers` reference — `Parser` SPI return shape, `default_registry()`, `detect_mime()`, format coverage, `ragctl parse` | ## guides/ diff --git a/docs/architecture/parsers.md b/docs/architecture/parsers.md new file mode 100644 index 0000000..d5fe8e6 --- /dev/null +++ b/docs/architecture/parsers.md @@ -0,0 +1,161 @@ +# Document parsers — architecture + +> Step 1.3 — Document parsers (PDF, DOCX, PPTX, XLSX, HTML, Markdown, +> plain text, JSON, CSV, YAML) + MIME detection. + +## Overview + +`rag-parsers` turns raw bytes into a `ParsedDocument` — extracted text +plus a list of structural `Block` records. It sits between the +connectors (Step 1.2 — fetch bytes) and the chunker (Step 1.5 — +boundary-aware splitting) in the ingest pipeline: + +``` +Connector → Parser → Chunker → Enricher → PII → Embedder → Store + (1.2) (1.3) (1.5) (1.6) (1.7) (1.8) (1.1) +``` + +The full pipeline is wired in Step 1.10. + +## Why a structural Block model + +A naïve "give me the text" Parser SPI would force the chunker to +re-parse every binary format to recover headings, list items, and table +rows for structure-aware chunking. That doubles the work and couples +the chunker to parser library internals. + +Instead, parsers emit a `Block` list alongside the flat text: + +```python +class Block(BaseModel): + type: BlockType # heading / paragraph / list_item / table_row / … + text: str + start: int # offset into ParsedDocument.text + level: int | None # 1–6 for headings, else None + attributes: dict # {"page": 3}, {"sheet": "Q1", "row": 5}, … +``` + +`BlockType` is deliberately small (8 values) — every parser maps its +library-native node types onto one of them so the chunker can apply +uniform rules regardless of source format. + +Decisions baked into the taxonomy: + +- **Headings carry levels (1–6).** Step 1.5's heading-based chunker + needs the level to decide where to break. PDFs do not carry heading + semantics, so PDFs emit no heading blocks; the chunker falls back to + paragraph-window heuristics there. +- **Tables flatten to `table_row` blocks.** Each row is a block; the + chunker can group rows back together when they fit a budget. Wide + tables would otherwise be a single overflowing block. +- **PPTX titles are headings at level 2, not level 1.** Decks are + one-level-down trees under an implicit deck-level title — keeping the + level-1 slot lets the chunker promote the deck name when one is + known. +- **PPTX speaker notes are `caption` blocks with + `attributes.kind == "notes"`.** Retrieval can opt notes in or out + per-corpus without us inventing a new block type. +- **XLSX sheet names are headings.** Same reason as PPTX titles. + +## Cross-platform constraint + +Every library in the dependency closure is pure-Python or wheel-only: + +| Library | Why this one | +|---------|--------------| +| `puremagic` | Pure-Python content sniffing; no system `libmagic` install on Windows | +| `markdown-it-py` | Maintained, used by mkdocs; clean token API for offset recovery | +| `beautifulsoup4` + stdlib `html.parser` | No `lxml` system dep; perf is sufficient at ingest speeds | +| `PyYAML` | Standard; wheels everywhere | +| `pypdf` | Pure-Python; `pdfplumber`/`pdfminer` rejected for being slower and larger | +| `python-docx`, `python-pptx`, `openpyxl` | Maintained by `python-openxml`; pure-Python | + +Per the standing constraint, dev setup must work on Windows / macOS / +Linux without WSL; CI matrix includes `windows-latest`. + +## MIME detection strategy + +`detect_mime(data, filename=..., hint=...)` resolves in priority order: + +1. **Extension** (when `filename` is given) — Office and PDF extensions + are deterministic, skip sniffing. +2. **Content sniff** via `puremagic` on the first 4 KiB. OOXML files + look like generic ZIP to `puremagic`; we inspect the ZIP central + directory entries (`word/`, `ppt/`, `xl/`) to disambiguate. +3. **Caller hint** (e.g. HTTP `Content-Type` header). +4. `application/octet-stream` so the registry can return a single + `ParseError` from one place instead of via `KeyError`. + +This three-tier approach is deliberately conservative: we trust +extensions for formats that have unambiguous file signatures +(docx/pptx/xlsx/pdf) and only sniff when the extension is ambiguous or +missing (e.g. content from a webhook with no filename). + +## Policy boundary + +Like connectors (Step 1.2), parsers are **upstream of the +`PolicyEngine`**. The PDP cannot enforce ACL or PII rules on bytes it +has not yet seen as a `Chunk`. The enforcement points stay in the +ingest pipeline (Step 1.10): + +- Per-document quota check **before** the parser runs (`PolicyEngine` + decision `ingest_doc`). +- Per-chunk ACL labeling **after** the chunker runs, before chunks are + written to the vector / keyword / graph stores. +- PII detection / redaction **after** the chunker, before embedding. + +This keeps the parser SPI free of governance plumbing and lets us swap +parsers without re-auditing the policy surface. + +## Error handling + +- `ParseError` (subclass of `IngestionError`) is raised for malformed + input. Step 1.10's pipeline catches it and marks the document + `DocumentStatus.failed` with a structured error log; the connector's + watermark still advances so the bad doc doesn't block subsequent ones. +- Empty input returns an empty `ParsedDocument`. The chunker is + responsible for deciding whether to drop empty documents — making + parsers raise on empty bytes would conflate "couldn't parse" with + "nothing to parse." +- Encrypted PDFs are rejected today. Password-protected ingestion + belongs with the secret-store work in Step 6.7. + +## Performance discipline + +Per the [hot-path discipline](performance.md) standing rule, parsers +are SPI-boundary code: Pydantic is fine for the return value +(`ParsedDocument`), but parser internals use plain dicts / dataclasses / +generators. Block lists are built incrementally; `text` is built with +a `list[str]` and joined at the end to avoid quadratic string +concatenation on large documents. + +`Batcher` (Step 1.1d) wraps the parser in Step 1.10 so concurrent +connector pulls coalesce into one parse-per-document — important for +PDF and XLSX where parsing dominates ingest latency. + +## Extension points + +To add a new format: + +1. Implement `rag_core.spi.Parser` and return a `ParsedDocument`. +2. Use `rag_parsers._base.build_document` so `content_hash` / + `tenant_id` / `status` / `mime_type` are wired the same way as the + built-ins. +3. Register via `ParserRegistry.register(MyParser())` — first match + wins. Specific MIME types should be registered before fallbacks. +4. Add MIME → extension mapping in `rag_parsers.mime._EXT_MIME` if a + file extension exists. +5. Mirror the conformance tests in `tests/parsers/`. + +## Related work + +- [Step 1.2 — connectors](connectors.md): upstream of parsers; emits + `(Document, ConnectorState)` pairs. +- [Step 1.5 — structure-aware chunker]: downstream consumer of `blocks`. +- [Step 1.10 — write path & ingest API]: wires connector → parser → + chunker → enricher → PII → embedder → store end-to-end. +- [ADR-0005 — PolicyEngine](../adr/ADR-0005-policy-engine.md): the + enforcement boundary for parser output. +- [ADR-0007 — tiered storage](../adr/ADR-0007-tiered-storage.md): + parsers return inline text today; large extracted blobs will move to + `BlobRef` when the chunker offloads them. diff --git a/docs/reference/parsers.md b/docs/reference/parsers.md new file mode 100644 index 0000000..182ea3d --- /dev/null +++ b/docs/reference/parsers.md @@ -0,0 +1,209 @@ +# rag-parsers — reference + +Built-in document parsers for AgentContextOS. Step 1.3 delivers ten +format-specific parsers plus a content-type detector that the ingest +pipeline (Step 1.10) will wire up after the chunker (Step 1.5) lands. + +## Overview + +A `Parser` consumes raw bytes and returns a `ParsedDocument` — + +- the embedded `Document` with `content_hash`, `tenant_id`, `corpus_id`, + `mime_type`, and `status=DocumentStatus.ready` set +- the extracted plain-text `text` +- a list of structural `Block` records (heading / paragraph / list_item / + table_row / code / quote / caption / other) with offsets into `text`, + optional heading level, and parser-specific `attributes` like + `{"page": 3}` or `{"sheet": "Q1", "row": 5}` +- the `detected_mime` — the parser's normalized MIME if it sniffed past + the caller-supplied hint + +The downstream chunker (Step 1.5) consumes `blocks` for structure-aware +boundary detection. Parsers that cannot recover structure (plain text, +opaque blobs) emit a single `BlockType.paragraph` spanning the full +extracted text. + +## Supported formats + +| MIME | Parser | Library | Heading detection | +|------|--------|---------|-------------------| +| `text/plain` and other `text/*` | `PlainTextParser` | stdlib | none — blank-line paragraph split | +| `text/markdown` | `MarkdownParser` | `markdown-it-py` | `#`–`######` → level 1–6 | +| `text/html`, `application/xhtml+xml` | `HtmlParser` | `beautifulsoup4` + stdlib `html.parser` | `

`–`

` | +| `application/json`, `application/x-ndjson` | `JsonParser` | stdlib | none — re-emitted as indented text | +| `text/csv`, `text/tab-separated-values` | `CsvParser` | stdlib | first row treated as header (`attributes.header=True`) | +| `application/yaml`, `text/yaml`, `text/x-yaml`, `application/x-yaml` | `YamlParser` | `PyYAML` (`safe_load`) | none | +| `application/pdf` | `PdfParser` | `pypdf` | none — one paragraph block per page | +| `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | `DocxParser` | `python-docx` | `Heading 1`–`Heading 6` styles | +| `application/vnd.openxmlformats-officedocument.presentationml.presentation` | `PptxParser` | `python-pptx` | slide title → level-2 heading | +| `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | `XlsxParser` | `openpyxl` (read-only) | sheet name → level-2 heading | + +All libraries are pure Python (or distributed as wheels), so the package +installs identically on Windows, macOS, and Linux without WSL. + +## Usage + +```python +from rag_core.types import ( + CorpusId, DocumentId, Principal, PrincipalId, PrincipalKind, + RequestContext, TenantId, +) +from rag_parsers import default_registry, detect_mime + +tenant = TenantId("acme") +ctx = RequestContext( + tenant_id=tenant, + principal=Principal( + id=PrincipalId("svc"), + kind=PrincipalKind.service, + display_name="svc", + tenant_id=tenant, + ), +) + +with open("memo.docx", "rb") as f: + data = f.read() + +mime = detect_mime(data, filename="memo.docx") +parser = default_registry().select(mime) +parsed = await parser.parse( + ctx, data, mime, + DocumentId("d1"), CorpusId("c1"), "file:///memo.docx", +) + +print(parsed.text[:200]) +for block in parsed.blocks[:5]: + print(block.type, block.level, block.text[:60]) +``` + +### MIME detection + +`detect_mime(data=..., filename=..., hint=...)` accepts any combination +of bytes / filename / caller hint. Resolution order: + +1. **Extension** — if `filename` is given, the lowercase extension wins + for known formats. Office and PDF extensions are deterministic, so + no content sniffing is needed. +2. **Content sniffing** via `puremagic` on the first 4 KiB. When + `puremagic` returns the generic `application/zip` for an OOXML file, + the ZIP central directory is inspected to distinguish `docx` / `pptx` + / `xlsx`. +3. **Caller hint** (e.g. an HTTP `Content-Type` header) as a last-resort + fallback. +4. `application/octet-stream` if all three fail. + +### Registry + +`default_registry()` returns a `ParserRegistry` pre-populated with every +built-in parser in the order **specific → generic**: Markdown, HTML, JSON, +CSV, YAML, PDF, DOCX, PPTX, XLSX, PlainText. Use `registry.select(mime)` +to resolve a parser; it raises `ParseError` when no registered parser +accepts the MIME type. + +You can also build a custom registry to override default ordering or to +inject your own `Parser` implementation: + +```python +from rag_parsers import ParserRegistry, PlainTextParser +registry = ParserRegistry() +registry.register(MyHtmlParser()) # takes precedence over the default +registry.register(PlainTextParser()) +``` + +### `ragctl parse` + +A smoke-test command ships with the CLI: + +``` +$ ragctl parse memo.docx +file: memo.docx +mime: application/vnd.openxmlformats-officedocument.wordprocessingml.document +parser: DocxParser +bytes: 24,816 +sha256: a1b2c3... +text: 1,847 chars +blocks: 18 (heading=3, paragraph=12, table_row=3) + +first blocks: + [heading h1] @0: Section One + [paragraph] @13: First body paragraph. + ... +``` + +Useful for design-partner demos and triaging parse problems before the +full ingest pipeline lands in Step 1.10. + +## Error handling + +- `ParseError` is raised when input bytes are malformed (corrupt PDF, + invalid JSON, broken OOXML, undecodable bytes). +- Empty input returns an empty `ParsedDocument` — `text=""`, `blocks=[]`, + `document.content_hash` still set (sha256 of `b""`). Callers should + decide whether to skip empty documents downstream rather than have + parsers raise. +- Encrypted PDFs are rejected with a `ParseError` — password-protected + ingestion is deferred to Step 6.7 (BYOK / secrets). + +## Internals + +### Block taxonomy + +`BlockType` is deliberately small: + +- `heading` — carries `level` 1–6 +- `paragraph` — generic flowing text +- `list_item` — bullet / numbered list element +- `table_row` — one row of a table (HTML ``, DOCX table, CSV/TSV row, + XLSX row) +- `code` — preformatted code block +- `quote` — `
` / Markdown `>` quote +- `caption` — figure caption (HTML `
`) or PPTX speaker notes +- `other` — escape hatch for parser-specific fragments + +Parsers normalize text inside blocks (collapsed whitespace, stripped +leading/trailing spaces) but do not merge across structural boundaries — +the chunker owns higher-level grouping decisions. + +### Offsets and reproducibility + +Block `start` offsets index into `ParsedDocument.text`, the same string +that was used to compute `document.content_hash`-adjacent text length +metrics. Parsers guarantee: + +- `text[block.start : block.start + len(block.text)] == block.text` + for plain-text / Markdown / HTML, where reading order matches input + order. +- For tabular formats (CSV / DOCX tables / XLSX) and OOXML where text is + re-assembled by the parser, offsets index into the re-assembled + `text`, not the source bytes. The chunker should treat `start` as an + intra-document anchor, not a byte offset. + +### Performance + +Parsers are designed to be cheap to construct (registry-friendly) and to +do real work inside `parse()`. Tests on the developer laptop (M1): + +- text-family: < 5 ms per file +- PDF (single-page): ~30 ms +- DOCX / PPTX / XLSX: ~50–150 ms +- 10-page PDF: ~200 ms + +Step 1.5's chunker is expected to dominate; Step 1.10's pipeline will +wrap parsers in a `Batcher` so connector-triggered concurrent fetches +coalesce into batched parses. + +## Extension points + +Adding a new parser: + +1. Implement `rag_core.spi.Parser` returning a `ParsedDocument`. +2. Use `rag_parsers._base.build_document` to assemble the embedded + `Document` with `content_hash` set correctly. +3. Register via `ParserRegistry.register(MyParser())` — first match wins. +4. Add the new MIME to `rag_parsers.mime._EXT_MIME` if a file-extension + shortcut is appropriate. +5. Add a conformance test under `tests/parsers/` and update + `docs/reference/parsers.md`. + +See [docs/architecture/parsers.md](../architecture/parsers.md) for the +design rationale and the policy / pipeline integration plan. diff --git a/packages/core/pyproject.toml b/packages/core/pyproject.toml index a9f3c8f..a3c1672 100644 --- a/packages/core/pyproject.toml +++ b/packages/core/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "rag-core" -version = "0.7.0" +version = "0.8.0" description = "AgentContextOS — core domain types, error hierarchy, and plugin SPIs" readme = "README.md" requires-python = ">=3.12" diff --git a/packages/core/src/rag_core/__init__.py b/packages/core/src/rag_core/__init__.py index a7f9edb..6d3aa2a 100644 --- a/packages/core/src/rag_core/__init__.py +++ b/packages/core/src/rag_core/__init__.py @@ -94,6 +94,8 @@ AuditEvent, AuditOutcome, BlobRef, + Block, + BlockType, Budget, Chunk, ChunkId, @@ -108,6 +110,7 @@ Embedding, EmbeddingDtype, IndexHint, + ParsedDocument, PiiAction, PiiPolicy, PlanNode, @@ -128,7 +131,7 @@ WriteVolume, ) -__version__ = "0.7.0" +__version__ = "0.8.0" __all__ = [ # eval @@ -167,6 +170,8 @@ "AuditEvent", "AuditOutcome", "BlobRef", + "Block", + "BlockType", "Budget", "Chunk", "ChunkId", @@ -181,6 +186,7 @@ "Embedding", "EmbeddingDtype", "IndexHint", + "ParsedDocument", "PiiAction", "PiiPolicy", "PlanNode", diff --git a/packages/core/src/rag_core/gen_schemas.py b/packages/core/src/rag_core/gen_schemas.py index ae68146..b5a0162 100644 --- a/packages/core/src/rag_core/gen_schemas.py +++ b/packages/core/src/rag_core/gen_schemas.py @@ -19,6 +19,7 @@ ACL, AuditEvent, BlobRef, + Block, Budget, Chunk, ChunkRef, @@ -28,6 +29,7 @@ Document, Embedding, IndexHint, + ParsedDocument, PiiPolicy, PlanNode, Principal, @@ -50,6 +52,8 @@ Tenant, BlobRef, Document, + Block, + ParsedDocument, ConnectorState, Chunk, Embedding, diff --git a/packages/core/src/rag_core/spi/noop/parser.py b/packages/core/src/rag_core/spi/noop/parser.py index 5eecd83..9afc993 100644 --- a/packages/core/src/rag_core/spi/noop/parser.py +++ b/packages/core/src/rag_core/spi/noop/parser.py @@ -1,15 +1,24 @@ -"""In-memory noop Parser — treats all bytes as plain text.""" +"""In-memory noop Parser — decodes bytes as UTF-8, one paragraph block.""" from __future__ import annotations import hashlib from rag_core.spi.parser import Parser -from rag_core.types import CorpusId, Document, DocumentId, DocumentStatus, RequestContext +from rag_core.types import ( + Block, + BlockType, + CorpusId, + Document, + DocumentId, + DocumentStatus, + ParsedDocument, + RequestContext, +) class NoopParser(Parser): - """Decodes bytes as UTF-8 text and returns a ready Document.""" + """Decodes bytes as UTF-8 text and returns a single-paragraph ParsedDocument.""" def supports(self, mime_type: str) -> bool: return True # accepts everything @@ -22,9 +31,10 @@ async def parse( document_id: DocumentId, corpus_id: CorpusId, source_uri: str, - ) -> Document: + ) -> ParsedDocument: content_hash = hashlib.sha256(data).hexdigest() - return Document( + text = data.decode("utf-8", errors="replace") + document = Document( id=document_id, tenant_id=ctx.tenant_id, corpus_id=corpus_id, @@ -33,6 +43,13 @@ async def parse( mime_type=mime_type, status=DocumentStatus.ready, ) + blocks = [Block(type=BlockType.paragraph, text=text, start=0)] if text else [] + return ParsedDocument( + document=document, + text=text, + blocks=blocks, + detected_mime=mime_type, + ) async def health(self) -> bool: return True diff --git a/packages/core/src/rag_core/spi/parser.py b/packages/core/src/rag_core/spi/parser.py index f36dd5c..aab51eb 100644 --- a/packages/core/src/rag_core/spi/parser.py +++ b/packages/core/src/rag_core/spi/parser.py @@ -1,18 +1,25 @@ -"""Parser SPI — raw bytes → Document content.""" +"""Parser SPI — raw bytes → ParsedDocument (extracted text + structural blocks).""" from __future__ import annotations import abc from rag_core.spi._base import HealthCheckMixin -from rag_core.types import CorpusId, Document, DocumentId, RequestContext +from rag_core.types import CorpusId, DocumentId, ParsedDocument, RequestContext class Parser(HealthCheckMixin, abc.ABC): """Abstract document parser (PDF, DOCX, HTML, Markdown, …). - Each implementation handles one or more MIME types. The pipeline selects - the appropriate parser via ``supports()``. + Step 1.3 evolved the contract from returning a bare ``Document`` to a + ``ParsedDocument`` carrying the extracted text plus structural + ``Block`` list, so the downstream chunker (Step 1.5) can find natural + boundaries without re-parsing the source bytes. + + Each implementation handles one or more MIME types; the ingest + pipeline selects a parser via ``supports()``. Implementations are + expected to be cheap to construct (registry-friendly) and to do the + real work inside ``parse()``. """ @abc.abstractmethod @@ -28,11 +35,13 @@ async def parse( document_id: DocumentId, corpus_id: CorpusId, source_uri: str, - ) -> Document: - """Parse ``data`` and return a ``Document`` with ``content_hash`` set. + ) -> ParsedDocument: + """Parse ``data`` and return its extracted text + structural blocks. - The returned ``Document.status`` should be ``DocumentStatus.ready``. - ``tenant_id`` is taken from ``ctx``. + The embedded ``Document`` has ``content_hash`` set (sha256 of the + raw bytes), ``status == DocumentStatus.ready``, and ``tenant_id`` + taken from ``ctx``. Parsers that cannot recover structure emit a + single ``BlockType.paragraph`` block spanning the full text. Raises: rag_core.errors.ParseError: If the bytes cannot be parsed. diff --git a/packages/core/src/rag_core/types.py b/packages/core/src/rag_core/types.py index bfbdba9..31e23bb 100644 --- a/packages/core/src/rag_core/types.py +++ b/packages/core/src/rag_core/types.py @@ -140,6 +140,25 @@ class StageEventKind(StrEnum): skipped = "skipped" +class BlockType(StrEnum): + """Coarse structural type of a parsed block. + + Emitted by `Parser` implementations (Step 1.3) so the structure-aware + chunker (Step 1.5) can find natural boundaries without re-parsing. The + taxonomy is deliberately small — parsers map library-native node types + onto one of these labels. + """ + + heading = "heading" + paragraph = "paragraph" + list_item = "list_item" + table_row = "table_row" + code = "code" + quote = "quote" + caption = "caption" + other = "other" + + # --------------------------------------------------------------------------- # TraceContext # --------------------------------------------------------------------------- @@ -342,6 +361,54 @@ class Document(BaseModel): updated_at: datetime = Field(default_factory=_utcnow) +# --------------------------------------------------------------------------- +# Block — one structural unit inside a parsed document +# --------------------------------------------------------------------------- +class Block(BaseModel): + """One structural unit produced by a Parser (Step 1.3). + + Blocks preserve enough document structure for the heading-aware chunker + (Step 1.5) to find natural boundaries without re-parsing the source + bytes. Each block carries the extracted text, its 0-based start offset + into ``ParsedDocument.text`` (the end is ``start + len(text)``), an + optional 1-6 heading level (only for ``BlockType.heading``), and a + free-form attributes dict for parser-specific extras such as + ``{"page": 3}`` for PDFs or ``{"sheet": "Q1", "row": 5}`` for XLSX. + """ + + model_config = {"frozen": True} + + type: BlockType + text: str + start: int + level: int | None = None + attributes: dict[str, Any] = Field(default_factory=dict) + + +# --------------------------------------------------------------------------- +# ParsedDocument — what a Parser returns +# --------------------------------------------------------------------------- +class ParsedDocument(BaseModel): + """A ``Document`` plus its extracted text and structural blocks. + + Returned by ``Parser.parse``. The chunker (Step 1.5) consumes + ``blocks`` for structure-aware boundary detection; parsers that cannot + recover structure (plain text, opaque blobs) emit a single + ``BlockType.paragraph`` block spanning the full text. + + ``detected_mime`` carries the parser's normalized MIME type, which may + differ from the caller-supplied hint when sniffing reveals a more + specific content type (e.g. a ``.txt`` file that is actually JSON). + """ + + model_config = {"frozen": True} + + document: Document + text: str + blocks: list[Block] = Field(default_factory=list) + detected_mime: str | None = None + + # --------------------------------------------------------------------------- # ConnectorState — watermark threaded through Connector.list_documents # --------------------------------------------------------------------------- diff --git a/packages/parsers/README.md b/packages/parsers/README.md new file mode 100644 index 0000000..900d448 --- /dev/null +++ b/packages/parsers/README.md @@ -0,0 +1,41 @@ +# rag-parsers + +Built-in document parsers for AgentContextOS — turn raw bytes into a `ParsedDocument` +(text + structural blocks) so downstream chunking, embedding, and retrieval can work +uniformly across formats. + +## Formats + +| MIME | Parser | Library | +|------|--------|---------| +| `text/plain` | `PlainTextParser` | stdlib | +| `text/markdown` | `MarkdownParser` | `markdown-it-py` | +| `text/html` | `HtmlParser` | `beautifulsoup4` (stdlib `html.parser`) | +| `application/json` | `JsonParser` | stdlib | +| `text/csv` | `CsvParser` | stdlib | +| `application/yaml`, `text/yaml` | `YamlParser` | `PyYAML` | +| `application/pdf` | `PdfParser` | `pypdf` | +| `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | `DocxParser` | `python-docx` | +| `application/vnd.openxmlformats-officedocument.presentationml.presentation` | `PptxParser` | `python-pptx` | +| `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | `XlsxParser` | `openpyxl` | + +## Usage + +```python +from rag_parsers import default_registry, detect_mime + +with open("doc.pdf", "rb") as f: + data = f.read() + +mime = detect_mime(data, filename="doc.pdf") +parser = default_registry().select(mime) +parsed = await parser.parse(ctx, data, mime, document_id, corpus_id, "file:///doc.pdf") + +print(parsed.text[:200]) +for block in parsed.blocks[:5]: + print(block.type, block.level, block.text[:60]) +``` + +See `docs/reference/parsers.md` for the full API surface and +`docs/architecture/parsers.md` for the design rationale (block taxonomy, +heading detection, cross-platform constraints, error handling). diff --git a/packages/parsers/pyproject.toml b/packages/parsers/pyproject.toml new file mode 100644 index 0000000..c71f7dc --- /dev/null +++ b/packages/parsers/pyproject.toml @@ -0,0 +1,40 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "rag-parsers" +version = "0.1.0" +description = "AgentContextOS — document parsers (PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML) with MIME detection" +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + "rag-core", + # MIME detection — pure-Python, no system libmagic dependency. + "puremagic>=1.27", + # Text-family parsers. + "markdown-it-py>=3.0", + "beautifulsoup4>=4.12", + "pyyaml>=6.0", + # Binary parsers — all pure-Python or wheel-distributed; no system deps. + "pypdf>=5.0", + "python-docx>=1.1", + "python-pptx>=1.0", + "openpyxl>=3.1", +] + +[project.optional-dependencies] +dev = [ + "pytest>=9.0", + "pytest-asyncio>=1.3", + "mypy>=2.1", +] + +[tool.hatch.build.targets.wheel] +packages = ["src/rag_parsers"] + +[tool.hatch.metadata] +allow-direct-references = true + +[tool.uv.sources] +rag-core = { workspace = true } diff --git a/packages/parsers/src/rag_parsers/__init__.py b/packages/parsers/src/rag_parsers/__init__.py new file mode 100644 index 0000000..2aead78 --- /dev/null +++ b/packages/parsers/src/rag_parsers/__init__.py @@ -0,0 +1,42 @@ +"""rag-parsers — built-in document parsers for AgentContextOS. + +Each parser is a concrete ``rag_core.spi.Parser`` returning a +``ParsedDocument`` (the embedded ``Document`` plus extracted text and a +list of structural ``Block`` records the chunker can consume in Step 1.5). + +See ``docs/reference/parsers.md`` for the full API surface and +``docs/architecture/parsers.md`` for the block taxonomy + design notes. +""" + +from __future__ import annotations + +from rag_parsers.csv_parser import CsvParser +from rag_parsers.docx import DocxParser +from rag_parsers.html import HtmlParser +from rag_parsers.json_parser import JsonParser +from rag_parsers.markdown import MarkdownParser +from rag_parsers.mime import detect_mime +from rag_parsers.pdf import PdfParser +from rag_parsers.pptx import PptxParser +from rag_parsers.registry import ParserRegistry, default_registry +from rag_parsers.text import PlainTextParser +from rag_parsers.xlsx import XlsxParser +from rag_parsers.yaml_parser import YamlParser + +__version__ = "0.1.0" + +__all__ = [ + "CsvParser", + "DocxParser", + "HtmlParser", + "JsonParser", + "MarkdownParser", + "ParserRegistry", + "PdfParser", + "PlainTextParser", + "PptxParser", + "XlsxParser", + "YamlParser", + "default_registry", + "detect_mime", +] diff --git a/packages/parsers/src/rag_parsers/_base.py b/packages/parsers/src/rag_parsers/_base.py new file mode 100644 index 0000000..b890a58 --- /dev/null +++ b/packages/parsers/src/rag_parsers/_base.py @@ -0,0 +1,56 @@ +"""Shared helpers for built-in parsers.""" + +from __future__ import annotations + +import hashlib + +from rag_core.types import ( + CorpusId, + Document, + DocumentId, + DocumentStatus, + ParsedDocument, + RequestContext, +) + + +def build_document( + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, +) -> Document: + """Assemble the embedded ``Document`` for a ``ParsedDocument`` return value. + + Centralises the sha256 content hash + tenant/corpus/status wiring so each + parser implementation only has to think about extracting text and blocks. + """ + return Document( + id=document_id, + tenant_id=ctx.tenant_id, + corpus_id=corpus_id, + source_uri=source_uri, + content_hash=hashlib.sha256(data).hexdigest(), + mime_type=mime_type, + status=DocumentStatus.ready, + ) + + +def empty_parsed( + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + detected_mime: str | None = None, +) -> ParsedDocument: + """Construct a ``ParsedDocument`` with empty text and no blocks.""" + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text="", + blocks=[], + detected_mime=detected_mime or mime_type, + ) diff --git a/packages/parsers/src/rag_parsers/csv_parser.py b/packages/parsers/src/rag_parsers/csv_parser.py new file mode 100644 index 0000000..a4b322e --- /dev/null +++ b/packages/parsers/src/rag_parsers/csv_parser.py @@ -0,0 +1,101 @@ +"""CSV / TSV parser — one ``table_row`` block per row, with header attribution.""" + +from __future__ import annotations + +import csv +import io + +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = {"text/csv", "text/tab-separated-values"} + + +class CsvParser(Parser): + """Parses CSV / TSV into row-per-block ``ParsedDocument``s. + + The first row is treated as the header; downstream block ``attributes`` + record both the row index (1-based, header excluded) and the header tuple + so the chunker / retrieval can re-emit the row in a denormalised form. + """ + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + raw = data.decode("utf-8-sig") + except UnicodeDecodeError as exc: + raise ParseError(f"CSV bytes are not valid UTF-8: {exc}") from exc + + if not raw.strip(): + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + delimiter = "\t" if mime_type == "text/tab-separated-values" else "," + reader = csv.reader(io.StringIO(raw), delimiter=delimiter) + try: + rows = list(reader) + except csv.Error as exc: + raise ParseError(f"Invalid CSV: {exc}") from exc + + if not rows: + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + header = tuple(rows[0]) + blocks: list[Block] = [] + text_parts: list[str] = [] + cursor = 0 + # Emit header row as its own block so consumers can spot the schema. + header_line = delimiter.join(header) + blocks.append( + Block( + type=BlockType.table_row, + text=header_line, + start=cursor, + attributes={"row": 0, "header": True}, + ) + ) + text_parts.append(header_line) + cursor += len(header_line) + 1 # +1 for the newline we'll join with + + for i, row in enumerate(rows[1:], start=1): + line = delimiter.join(row) + blocks.append( + Block( + type=BlockType.table_row, + text=line, + start=cursor, + attributes={"row": i, "header": False, "columns": list(header)}, + ) + ) + text_parts.append(line) + cursor += len(line) + 1 + + text = "\n".join(text_parts) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True diff --git a/packages/parsers/src/rag_parsers/docx.py b/packages/parsers/src/rag_parsers/docx.py new file mode 100644 index 0000000..429d344 --- /dev/null +++ b/packages/parsers/src/rag_parsers/docx.py @@ -0,0 +1,105 @@ +"""DOCX parser — python-docx, heading-aware paragraph + table extraction.""" + +from __future__ import annotations + +import io +import re +from typing import Any + +from docx import Document as DocxDocument +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = { + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", +} + +_HEADING_STYLE_RE = re.compile(r"^heading\s*(\d+)$", re.IGNORECASE) +_LIST_STYLES = {"list paragraph", "list bullet", "list number"} + + +class DocxParser(Parser): + """Extracts paragraphs (with heading levels), list items, and table rows from DOCX.""" + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + doc = DocxDocument(io.BytesIO(data)) + except Exception as exc: # noqa: BLE001 — python-docx raises broadly + raise ParseError(f"Unable to read DOCX: {exc}") from exc + + blocks: list[Block] = [] + text_parts: list[str] = [] + cursor = 0 + + for para in doc.paragraphs: + line = (para.text or "").strip() + if not line: + continue + block_type, level = _classify_paragraph(para) + blocks.append(Block(type=block_type, text=line, start=cursor, level=level)) + text_parts.append(line) + cursor += len(line) + 1 + + for table_idx, table in enumerate(doc.tables, start=1): + for row_idx, row in enumerate(table.rows, start=1): + cells = [cell.text.strip() for cell in row.cells] + line = " | ".join(cells) + if not line.strip(): + continue + blocks.append( + Block( + type=BlockType.table_row, + text=line, + start=cursor, + attributes={"table": table_idx, "row": row_idx}, + ) + ) + text_parts.append(line) + cursor += len(line) + 1 + + if not blocks: + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + text = "\n".join(text_parts) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True + + +def _classify_paragraph(para: Any) -> tuple[BlockType, int | None]: + """Return (block_type, heading_level) for a python-docx paragraph.""" + style_name = (getattr(para.style, "name", "") or "").strip() + heading_match = _HEADING_STYLE_RE.match(style_name) + if heading_match: + level = int(heading_match.group(1)) + return BlockType.heading, max(1, min(6, level)) + if style_name.lower() in _LIST_STYLES: + return BlockType.list_item, None + return BlockType.paragraph, None diff --git a/packages/parsers/src/rag_parsers/html.py b/packages/parsers/src/rag_parsers/html.py new file mode 100644 index 0000000..b931430 --- /dev/null +++ b/packages/parsers/src/rag_parsers/html.py @@ -0,0 +1,106 @@ +"""HTML parser — BeautifulSoup + stdlib ``html.parser`` (no lxml system dep).""" + +from __future__ import annotations + +from bs4 import BeautifulSoup +from bs4.element import Tag +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = {"text/html", "application/xhtml+xml"} + +# Tags we drop entirely — never carry user-facing text. +_DROPPED_TAGS = {"script", "style", "noscript", "template", "head"} + +# Tag → BlockType mapping for terminal text containers. +_TAG_TO_BLOCK: dict[str, BlockType] = { + "p": BlockType.paragraph, + "li": BlockType.list_item, + "blockquote": BlockType.quote, + "pre": BlockType.code, + "code": BlockType.code, + "tr": BlockType.table_row, + "figcaption": BlockType.caption, +} + + +class HtmlParser(Parser): + """Extracts headings + paragraphs + lists + tables from HTML using BeautifulSoup.""" + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + # html.parser is pure-Python and bundled with the stdlib — keeps the + # dependency footprint identical on Windows, macOS, and Linux. + soup = BeautifulSoup(data, "html.parser") + except Exception as exc: # noqa: BLE001 — bs4 raises a variety + raise ParseError(f"Invalid HTML: {exc}") from exc + + for tag_name in _DROPPED_TAGS: + for node in soup.find_all(tag_name): + node.decompose() + + blocks: list[Block] = [] + text_parts: list[str] = [] + cursor = 0 + + for tag in soup.find_all(_block_tags()): + block_text = _normalized_text(tag) + if not block_text: + continue + block_type, level = _classify(tag) + blocks.append(Block(type=block_type, text=block_text, start=cursor, level=level)) + text_parts.append(block_text) + cursor += len(block_text) + 1 # +1 for the joining newline + + if not blocks: + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + text = "\n".join(text_parts) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True + + +def _block_tags() -> list[str]: + return ["h1", "h2", "h3", "h4", "h5", "h6", *_TAG_TO_BLOCK.keys()] + + +def _classify(tag: Tag) -> tuple[BlockType, int | None]: + name = tag.name.lower() + if name in {"h1", "h2", "h3", "h4", "h5", "h6"}: + return BlockType.heading, int(name[1]) + block_type = _TAG_TO_BLOCK.get(name, BlockType.other) + return block_type, None + + +def _normalized_text(tag: Tag) -> str: + """Collapse runs of whitespace inside the tag and strip leading/trailing space.""" + raw = tag.get_text(separator=" ", strip=True) + return " ".join(raw.split()) diff --git a/packages/parsers/src/rag_parsers/json_parser.py b/packages/parsers/src/rag_parsers/json_parser.py new file mode 100644 index 0000000..38f13be --- /dev/null +++ b/packages/parsers/src/rag_parsers/json_parser.py @@ -0,0 +1,65 @@ +"""JSON parser — validate + pretty-print so the text is grep-friendly.""" + +from __future__ import annotations + +import json + +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = {"application/json", "application/x-ndjson"} + + +class JsonParser(Parser): + """Validates JSON / NDJSON, returns indent=2 rendered text as one paragraph block.""" + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + raw = data.decode("utf-8") + except UnicodeDecodeError as exc: + raise ParseError(f"JSON bytes are not valid UTF-8: {exc}") from exc + + if not raw.strip(): + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + try: + if mime_type == "application/x-ndjson": + lines = [json.loads(line) for line in raw.splitlines() if line.strip()] + rendered = "\n".join(json.dumps(obj, indent=2, sort_keys=True) for obj in lines) + else: + parsed = json.loads(raw) + rendered = json.dumps(parsed, indent=2, sort_keys=True) + except json.JSONDecodeError as exc: + raise ParseError(f"Invalid JSON: {exc}") from exc + + block = Block(type=BlockType.paragraph, text=rendered, start=0) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=rendered, + blocks=[block], + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True diff --git a/packages/parsers/src/rag_parsers/markdown.py b/packages/parsers/src/rag_parsers/markdown.py new file mode 100644 index 0000000..ab66e72 --- /dev/null +++ b/packages/parsers/src/rag_parsers/markdown.py @@ -0,0 +1,184 @@ +"""Markdown parser — markdown-it-py tokens → structural blocks.""" + +from __future__ import annotations + +from markdown_it import MarkdownIt +from markdown_it.token import Token +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = {"text/markdown", "text/x-markdown"} + +# Map a markdown-it container token name (without the ``_open`` suffix) to a +# block type. Headings are special-cased to recover their level. +_CONTAINER_TO_BLOCK: dict[str, BlockType] = { + "paragraph": BlockType.paragraph, + "list_item": BlockType.list_item, + "blockquote": BlockType.quote, +} + + +class MarkdownParser(Parser): + """Parses CommonMark + GFM tables into headings, paragraphs, lists, code, quotes.""" + + def __init__(self) -> None: + self._md = MarkdownIt("commonmark", {"html": False}).enable("table") + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + text = data.decode("utf-8") + except UnicodeDecodeError as exc: + raise ParseError(f"Markdown bytes are not valid UTF-8: {exc}") from exc + + if not text.strip(): + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + tokens = self._md.parse(text) + blocks = _tokens_to_blocks(tokens, text) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True + + +def _tokens_to_blocks(tokens: list[Token], text: str) -> list[Block]: + """Walk markdown-it tokens and emit one Block per top-level container. + + ``Token.map`` carries the source line range, which we use to compute the + block's start offset in the original text. When a token has no map + (rare — e.g. autolink inline tokens promoted to the root), we fall back to + locating the rendered text via ``str.find`` starting from the previous + cursor so offsets stay monotonic. + """ + blocks: list[Block] = [] + line_starts = _line_start_offsets(text) + cursor = 0 + i = 0 + while i < len(tokens): + tok = tokens[i] + + if tok.type == "heading_open": + close_idx = _find_close(tokens, i, "heading_close") + inline = _inline_text(tokens[i + 1 : close_idx]) + level = int(tok.tag[1:]) if tok.tag.startswith("h") else 1 + start = _start_offset(tok, line_starts, text, inline, cursor) + blocks.append(Block(type=BlockType.heading, text=inline, start=start, level=level)) + cursor = start + len(inline) + i = close_idx + 1 + continue + + if tok.type == "fence" or tok.type == "code_block": + content = tok.content.rstrip("\n") + start = _start_offset(tok, line_starts, text, content, cursor) + attributes = {"info": tok.info} if tok.info else {} + blocks.append( + Block( + type=BlockType.code, + text=content, + start=start, + attributes=attributes, + ) + ) + cursor = start + len(content) + i += 1 + continue + + # Generic container open: paragraph / list_item / blockquote. + if tok.type.endswith("_open"): + container = tok.type[: -len("_open")] + block_type = _CONTAINER_TO_BLOCK.get(container) + if block_type is not None: + close_idx = _find_close(tokens, i, f"{container}_close") + inline = _inline_text(tokens[i + 1 : close_idx]) + if inline.strip(): + start = _start_offset(tok, line_starts, text, inline, cursor) + blocks.append(Block(type=block_type, text=inline, start=start)) + cursor = start + len(inline) + i = close_idx + 1 + continue + i += 1 + return blocks + + +def _line_start_offsets(text: str) -> list[int]: + offsets = [0] + for i, ch in enumerate(text): + if ch == "\n": + offsets.append(i + 1) + return offsets + + +def _start_offset( + tok: Token, + line_starts: list[int], + text: str, + needle: str, + cursor: int, +) -> int: + """Compute the character offset of ``tok`` in ``text``. + + Uses ``Token.map`` (line range) when available; otherwise falls back to a + forward-search for ``needle`` starting from ``cursor``. The fallback + keeps offsets monotonic even when markdown-it elides line maps. + """ + if tok.map is not None and 0 <= tok.map[0] < len(line_starts): + return line_starts[tok.map[0]] + if needle: + found = text.find(needle, cursor) + if found >= 0: + return found + return cursor + + +def _find_close(tokens: list[Token], start_idx: int, close_type: str) -> int: + depth = 0 + open_type = tokens[start_idx].type + for j in range(start_idx, len(tokens)): + if tokens[j].type == open_type: + depth += 1 + elif tokens[j].type == close_type: + depth -= 1 + if depth == 0: + return j + return len(tokens) - 1 + + +def _inline_text(inline_tokens: list[Token]) -> str: + """Extract the rendered text from a slice of tokens between open/close markers.""" + parts: list[str] = [] + for tok in inline_tokens: + if tok.type == "inline" and tok.children is not None: + for child in tok.children: + if child.type == "text" or child.type == "code_inline": + parts.append(child.content) + elif child.type == "softbreak" or child.type == "hardbreak": + parts.append("\n") + elif tok.content: + parts.append(tok.content) + return "".join(parts).strip() diff --git a/packages/parsers/src/rag_parsers/mime.py b/packages/parsers/src/rag_parsers/mime.py new file mode 100644 index 0000000..a52114c --- /dev/null +++ b/packages/parsers/src/rag_parsers/mime.py @@ -0,0 +1,156 @@ +"""MIME / content-type detection for built-in parsers. + +Uses :mod:`puremagic` for content sniffing — pure-Python, no system +``libmagic`` dependency, so dev setup is identical on Windows, macOS, and +Linux without WSL (per the standing cross-platform constraint). + +Detection rules, in order: + +1. If the caller passes ``filename``, normalize the extension first. Office + and PDF extensions are deterministic, so we don't waste cycles on + content sniffing for those. +2. Sniff the first 4 KiB of bytes via :mod:`puremagic`; map its + often-OOXML-generic return value to the specific Office MIME using the + ZIP central-directory entry names. +3. Fall back to ``application/octet-stream`` so the registry can return a + ``ParseError`` from a single place rather than via ``KeyError``. +""" + +from __future__ import annotations + +import io +import zipfile +from pathlib import Path + +import puremagic + +# Extension → canonical MIME. Lowercase, no leading dot. +_EXT_MIME: dict[str, str] = { + "txt": "text/plain", + "log": "text/plain", + "md": "text/markdown", + "markdown": "text/markdown", + "rst": "text/plain", + "html": "text/html", + "htm": "text/html", + "xhtml": "application/xhtml+xml", + "json": "application/json", + "ndjson": "application/x-ndjson", + "jsonl": "application/x-ndjson", + "csv": "text/csv", + "tsv": "text/tab-separated-values", + "yaml": "application/yaml", + "yml": "application/yaml", + "pdf": "application/pdf", + "docx": ("application/vnd.openxmlformats-officedocument.wordprocessingml.document"), + "pptx": ("application/vnd.openxmlformats-officedocument.presentationml.presentation"), + "xlsx": ("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"), +} + +# OOXML root entry → MIME, used to disambiguate puremagic's generic ZIP hit. +_OOXML_ROOT_TO_MIME: dict[str, str] = { + "word/": ("application/vnd.openxmlformats-officedocument.wordprocessingml.document"), + "ppt/": ("application/vnd.openxmlformats-officedocument.presentationml.presentation"), + "xl/": ("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"), +} + +_SNIFF_BYTES = 4096 + + +def detect_mime( + data: bytes | None = None, + *, + filename: str | None = None, + hint: str | None = None, +) -> str: + """Detect a canonical MIME type for ``data`` and/or ``filename``. + + Parameters + ---------- + data: + Optional raw bytes — the first ``_SNIFF_BYTES`` are inspected. + filename: + Optional file name or path — only the extension is consulted. + hint: + Optional caller-supplied MIME (e.g. an HTTP ``Content-Type`` header). + Used only when extension + content sniffing both fail. + + At least one of ``data`` or ``filename`` must be provided. The function + is deterministic and side-effect-free. + """ + if data is None and filename is None: + raise ValueError("detect_mime requires data or filename") + + if filename is not None: + ext = Path(filename).suffix.lower().lstrip(".") + if ext in _EXT_MIME: + return _EXT_MIME[ext] + + if data: + sniffed = _sniff(data) + if sniffed: + return sniffed + + if hint: + return hint + + return "application/octet-stream" + + +def _sniff(data: bytes) -> str | None: + """Best-effort content sniffing using puremagic + OOXML disambiguation.""" + if not data: + return None + head = data[:_SNIFF_BYTES] + + # OOXML files start with the ZIP magic ``PK\x03\x04`` — peek inside to + # distinguish docx/pptx/xlsx from a generic .zip. + if head[:4] == b"PK\x03\x04": + ooxml = _sniff_ooxml(data) + if ooxml is not None: + return ooxml + + try: + candidates = puremagic.magic_string(head) + except puremagic.PureError: + return None + + if not candidates: + return None + + # puremagic returns confidence-sorted matches with a leading-dot extension + # and a mime_type attribute; favour the first non-empty mime_type, then + # fall back to mapping the extension through ``_EXT_MIME``. + for candidate in candidates: + mime = (candidate.mime_type or "").strip() + if mime: + return _normalize_sniffed_mime(mime) + ext = (candidate.extension or "").lstrip(".").lower() + if ext in _EXT_MIME: + return _EXT_MIME[ext] + return None + + +def _sniff_ooxml(data: bytes) -> str | None: + """Inspect a ZIP's central directory to identify docx/pptx/xlsx.""" + try: + with zipfile.ZipFile(io.BytesIO(data)) as zf: + names = zf.namelist() + except zipfile.BadZipFile: + return None + for prefix, mime in _OOXML_ROOT_TO_MIME.items(): + if any(name.startswith(prefix) for name in names): + return mime + return None + + +def _normalize_sniffed_mime(mime: str) -> str: + """Collapse puremagic synonyms onto our canonical set.""" + # puremagic occasionally returns ``application/x-yaml`` for YAML and + # ``application/x-markdown`` for Markdown; normalise to the modern names. + return { + "application/x-yaml": "application/yaml", + "text/x-yaml": "application/yaml", + "application/x-markdown": "text/markdown", + "text/x-markdown": "text/markdown", + }.get(mime, mime) diff --git a/packages/parsers/src/rag_parsers/pdf.py b/packages/parsers/src/rag_parsers/pdf.py new file mode 100644 index 0000000..704f796 --- /dev/null +++ b/packages/parsers/src/rag_parsers/pdf.py @@ -0,0 +1,91 @@ +"""PDF parser — pypdf, one paragraph block per page.""" + +from __future__ import annotations + +import io + +from pypdf import PdfReader +from pypdf.errors import PdfReadError +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = {"application/pdf"} + + +class PdfParser(Parser): + """Extracts page-by-page text using pypdf (pure-Python, cross-platform). + + Each page becomes one ``BlockType.paragraph`` block with + ``attributes={"page": <1-based index>}``. The 1.5 chunker can break + long pages into smaller chunks; we keep the page as the natural unit + here because most PDFs are typeset with page-scoped semantics. + """ + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + reader = PdfReader(io.BytesIO(data)) + except (PdfReadError, ValueError, OSError) as exc: + raise ParseError(f"Unable to read PDF: {exc}") from exc + + if reader.is_encrypted: + # Refuse encrypted PDFs in this PR — Step 1.4 (OCR) and the + # secret-store work in Step 6.7 are the right places to handle + # password-protected ingestion. + raise ParseError("Encrypted PDFs are not supported by PdfParser") + + blocks: list[Block] = [] + text_parts: list[str] = [] + cursor = 0 + for i, page in enumerate(reader.pages, start=1): + try: + page_text = page.extract_text() or "" + except Exception as exc: # noqa: BLE001 — pypdf raises broadly + raise ParseError(f"Failed to extract text from PDF page {i}: {exc}") from exc + page_text = page_text.strip() + if not page_text: + continue + blocks.append( + Block( + type=BlockType.paragraph, + text=page_text, + start=cursor, + attributes={"page": i}, + ) + ) + text_parts.append(page_text) + cursor += len(page_text) + 2 # +2 for the "\n\n" page separator + + if not blocks: + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + text = "\n\n".join(text_parts) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True diff --git a/packages/parsers/src/rag_parsers/pptx.py b/packages/parsers/src/rag_parsers/pptx.py new file mode 100644 index 0000000..5e3842e --- /dev/null +++ b/packages/parsers/src/rag_parsers/pptx.py @@ -0,0 +1,143 @@ +"""PPTX parser — python-pptx, slide-by-slide title + body extraction.""" + +from __future__ import annotations + +import io +from typing import Any + +from pptx import Presentation +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = { + "application/vnd.openxmlformats-officedocument.presentationml.presentation", +} + + +class PptxParser(Parser): + """Extracts a slide title (as a heading) + paragraph text for each slide. + + The slide title becomes a ``BlockType.heading`` with ``level=2`` so that + deck-level heuristics in the chunker (Step 1.5) can treat decks as + one-level-down trees under an implicit deck title. Speaker notes are + surfaced as ``BlockType.caption`` blocks with + ``attributes={"kind": "notes"}`` so retrieval can opt in or out. + """ + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + deck = Presentation(io.BytesIO(data)) + except Exception as exc: # noqa: BLE001 — python-pptx raises broadly + raise ParseError(f"Unable to read PPTX: {exc}") from exc + + blocks: list[Block] = [] + text_parts: list[str] = [] + cursor = 0 + + for slide_idx, slide in enumerate(deck.slides, start=1): + title = _slide_title(slide) + if title: + blocks.append( + Block( + type=BlockType.heading, + text=title, + start=cursor, + level=2, + attributes={"slide": slide_idx}, + ) + ) + text_parts.append(title) + cursor += len(title) + 1 + + for shape in slide.shapes: + if shape == _title_shape(slide): + continue + if not shape.has_text_frame: + continue + for para in shape.text_frame.paragraphs: + body = (para.text or "").strip() + if not body: + continue + blocks.append( + Block( + type=BlockType.paragraph, + text=body, + start=cursor, + attributes={"slide": slide_idx}, + ) + ) + text_parts.append(body) + cursor += len(body) + 1 + + notes = _slide_notes(slide) + if notes: + blocks.append( + Block( + type=BlockType.caption, + text=notes, + start=cursor, + attributes={"slide": slide_idx, "kind": "notes"}, + ) + ) + text_parts.append(notes) + cursor += len(notes) + 1 + + if not blocks: + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + text = "\n".join(text_parts) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True + + +def _title_shape(slide: Any) -> Any: + try: + return slide.shapes.title + except AttributeError: + return None + + +def _slide_title(slide: Any) -> str | None: + shape = _title_shape(slide) + if shape is None: + return None + title = (getattr(shape, "text", "") or "").strip() + return title or None + + +def _slide_notes(slide: Any) -> str | None: + if not getattr(slide, "has_notes_slide", False): + return None + notes_frame = getattr(slide.notes_slide, "notes_text_frame", None) + if notes_frame is None: + return None + text = (notes_frame.text or "").strip() + return text or None diff --git a/packages/parsers/src/rag_parsers/py.typed b/packages/parsers/src/rag_parsers/py.typed new file mode 100644 index 0000000..e69de29 diff --git a/packages/parsers/src/rag_parsers/registry.py b/packages/parsers/src/rag_parsers/registry.py new file mode 100644 index 0000000..f56fffb --- /dev/null +++ b/packages/parsers/src/rag_parsers/registry.py @@ -0,0 +1,79 @@ +"""Parser registry — MIME → ``Parser`` resolution for the built-in parsers.""" + +from __future__ import annotations + +from collections.abc import Iterator + +from rag_core.errors import ParseError +from rag_core.spi import Parser + + +class ParserRegistry: + """Ordered registry mapping MIME types to ``Parser`` implementations. + + The first registered parser whose ``supports(mime)`` returns True wins. + Insertion order is preserved, so more specific parsers should be + registered before generic fallbacks. + """ + + def __init__(self) -> None: + self._parsers: list[Parser] = [] + + def register(self, parser: Parser) -> None: + """Append a parser to the resolution order.""" + self._parsers.append(parser) + + def select(self, mime_type: str) -> Parser: + """Return the first registered parser that supports ``mime_type``. + + Raises: + ParseError: If no registered parser accepts the MIME type. + """ + for parser in self._parsers: + if parser.supports(mime_type): + return parser + raise ParseError(f"No registered parser handles MIME type {mime_type!r}") + + def __len__(self) -> int: + return len(self._parsers) + + def __iter__(self) -> Iterator[Parser]: + return iter(self._parsers) + + +def default_registry() -> ParserRegistry: + """Return a ``ParserRegistry`` pre-populated with every built-in parser. + + Order: text-family (cheap, deterministic, no heavy deps) first, then + OOXML (PDF/DOCX/PPTX/XLSX). Specific MIME types come before generic + fallbacks so the most precise handler wins. + """ + # Imported here so a consumer that only needs the registry type doesn't pay + # the cost of importing every parser module up-front. + from rag_parsers.csv_parser import CsvParser + from rag_parsers.docx import DocxParser + from rag_parsers.html import HtmlParser + from rag_parsers.json_parser import JsonParser + from rag_parsers.markdown import MarkdownParser + from rag_parsers.pdf import PdfParser + from rag_parsers.pptx import PptxParser + from rag_parsers.text import PlainTextParser + from rag_parsers.xlsx import XlsxParser + from rag_parsers.yaml_parser import YamlParser + + registry = ParserRegistry() + # Text-family — register before PlainTextParser so JSON/CSV/YAML/MD/HTML + # claim their specific MIME types before the catch-all text branch. + registry.register(MarkdownParser()) + registry.register(HtmlParser()) + registry.register(JsonParser()) + registry.register(CsvParser()) + registry.register(YamlParser()) + # OOXML + PDF. + registry.register(PdfParser()) + registry.register(DocxParser()) + registry.register(PptxParser()) + registry.register(XlsxParser()) + # PlainText is the generic text/* fallback. + registry.register(PlainTextParser()) + return registry diff --git a/packages/parsers/src/rag_parsers/text.py b/packages/parsers/src/rag_parsers/text.py new file mode 100644 index 0000000..dfa0e0d --- /dev/null +++ b/packages/parsers/src/rag_parsers/text.py @@ -0,0 +1,88 @@ +"""Plain-text parser — UTF-8 decode with BOM handling and paragraph splitting.""" + +from __future__ import annotations + +import re + +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +# Splits on a run of two or more newlines (with optional intervening whitespace). +_PARAGRAPH_RE = re.compile(r"\n[ \t]*\n+") + +_SUPPORTED_PREFIXES = ("text/",) +_SUPPORTED_EXACT = {"application/octet-stream"} + + +class PlainTextParser(Parser): + """Catch-all for ``text/*`` content — decodes UTF-8 and splits on blank lines.""" + + def supports(self, mime_type: str) -> bool: + return mime_type.startswith(_SUPPORTED_PREFIXES) or mime_type in _SUPPORTED_EXACT + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + text = _decode(data) + if not text.strip(): + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + blocks = _split_paragraphs(text) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True + + +def _decode(data: bytes) -> str: + """UTF-8 decode with BOM stripping; fall back to latin-1 on errors. + + The fallback is intentional: text files in the wild often come in with + legacy encodings, and replacing characters tends to corrupt downstream + chunk hashes. latin-1 round-trips every byte and lets the chunker make + its own decision. + """ + if data.startswith(b"\xef\xbb\xbf"): + data = data[3:] + try: + return data.decode("utf-8") + except UnicodeDecodeError as exc: + try: + return data.decode("latin-1") + except UnicodeDecodeError: + raise ParseError(f"Unable to decode text bytes: {exc}") from exc + + +def _split_paragraphs(text: str) -> list[Block]: + """Split text into paragraph blocks on blank lines, preserving offsets.""" + blocks: list[Block] = [] + cursor = 0 + for match in _PARAGRAPH_RE.finditer(text): + chunk = text[cursor : match.start()] + if chunk.strip(): + blocks.append(Block(type=BlockType.paragraph, text=chunk, start=cursor)) + cursor = match.end() + tail = text[cursor:] + if tail.strip(): + blocks.append(Block(type=BlockType.paragraph, text=tail, start=cursor)) + return blocks diff --git a/packages/parsers/src/rag_parsers/xlsx.py b/packages/parsers/src/rag_parsers/xlsx.py new file mode 100644 index 0000000..3d94821 --- /dev/null +++ b/packages/parsers/src/rag_parsers/xlsx.py @@ -0,0 +1,110 @@ +"""XLSX parser — openpyxl, one ``table_row`` block per non-empty row.""" + +from __future__ import annotations + +import io + +from openpyxl import load_workbook +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = { + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", +} + + +class XlsxParser(Parser): + """Reads an XLSX workbook in read-only mode and emits one row per block. + + Each row's ``attributes`` carries the sheet name and 1-based row index. + Cell values are rendered with ``str(value)`` — formatting (currency, + dates) is intentionally not preserved because retrieval cares about + the underlying value, not the visual representation. + """ + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + workbook = load_workbook(io.BytesIO(data), read_only=True, data_only=True) + except Exception as exc: # noqa: BLE001 — openpyxl raises broadly + raise ParseError(f"Unable to read XLSX: {exc}") from exc + + blocks: list[Block] = [] + text_parts: list[str] = [] + cursor = 0 + + try: + for sheet_name in workbook.sheetnames: + ws = workbook[sheet_name] + # Emit the sheet name as a level-2 heading so a downstream + # chunker can keep rows grouped under their sheet. + heading = sheet_name + blocks.append( + Block( + type=BlockType.heading, + text=heading, + start=cursor, + level=2, + attributes={"sheet": sheet_name}, + ) + ) + text_parts.append(heading) + cursor += len(heading) + 1 + + for row_idx, row in enumerate(ws.iter_rows(values_only=True), start=1): + cells = [_render(value) for value in row] + if not any(cell for cell in cells): + continue + line = " | ".join(cells) + blocks.append( + Block( + type=BlockType.table_row, + text=line, + start=cursor, + attributes={"sheet": sheet_name, "row": row_idx}, + ) + ) + text_parts.append(line) + cursor += len(line) + 1 + finally: + workbook.close() + + if not blocks: + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + text = "\n".join(text_parts) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=text, + blocks=blocks, + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True + + +def _render(value: object) -> str: + if value is None: + return "" + return str(value) diff --git a/packages/parsers/src/rag_parsers/yaml_parser.py b/packages/parsers/src/rag_parsers/yaml_parser.py new file mode 100644 index 0000000..53cc8e9 --- /dev/null +++ b/packages/parsers/src/rag_parsers/yaml_parser.py @@ -0,0 +1,71 @@ +"""YAML parser — safe_load validate, dump back for normalized text.""" + +from __future__ import annotations + +from typing import Any + +import yaml +from rag_core.errors import ParseError +from rag_core.spi import Parser +from rag_core.types import ( + Block, + BlockType, + CorpusId, + DocumentId, + ParsedDocument, + RequestContext, +) + +from rag_parsers._base import build_document, empty_parsed + +_SUPPORTED = {"application/yaml", "text/yaml", "text/x-yaml", "application/x-yaml"} + + +class YamlParser(Parser): + """Validates YAML via ``safe_load`` and renders one paragraph block. + + ``safe_load`` is mandatory — full ``load`` would allow arbitrary Python + construction and is a known RCE vector on untrusted ingest input. + """ + + def supports(self, mime_type: str) -> bool: + return mime_type in _SUPPORTED + + async def parse( + self, + ctx: RequestContext, + data: bytes, + mime_type: str, + document_id: DocumentId, + corpus_id: CorpusId, + source_uri: str, + ) -> ParsedDocument: + try: + raw = data.decode("utf-8") + except UnicodeDecodeError as exc: + raise ParseError(f"YAML bytes are not valid UTF-8: {exc}") from exc + + if not raw.strip(): + return empty_parsed(ctx, data, mime_type, document_id, corpus_id, source_uri) + + try: + docs: list[Any] = list(yaml.safe_load_all(raw)) + except yaml.YAMLError as exc: + raise ParseError(f"Invalid YAML: {exc}") from exc + + rendered_parts = [ + yaml.safe_dump(doc, sort_keys=True, default_flow_style=False).rstrip("\n") + for doc in docs + if doc is not None + ] + rendered = "\n---\n".join(rendered_parts) + block = Block(type=BlockType.paragraph, text=rendered, start=0) + return ParsedDocument( + document=build_document(ctx, data, mime_type, document_id, corpus_id, source_uri), + text=rendered, + blocks=[block] if rendered else [], + detected_mime=mime_type, + ) + + async def health(self) -> bool: + return True diff --git a/packages/ragctl/pyproject.toml b/packages/ragctl/pyproject.toml index e43ce0e..1110960 100644 --- a/packages/ragctl/pyproject.toml +++ b/packages/ragctl/pyproject.toml @@ -13,12 +13,14 @@ dependencies = [ "rag-core>=0.1.0", "rag-config>=0.1.0", "rag-observability>=0.1.0", + "rag-parsers>=0.1.0", ] [tool.uv.sources] rag-core = { workspace = true } rag-config = { workspace = true } rag-observability = { workspace = true } +rag-parsers = { workspace = true } [project.scripts] ragctl = "ragctl.main:app" diff --git a/packages/ragctl/src/ragctl/main.py b/packages/ragctl/src/ragctl/main.py index 8375dff..c316cd7 100644 --- a/packages/ragctl/src/ragctl/main.py +++ b/packages/ragctl/src/ragctl/main.py @@ -8,6 +8,7 @@ | traces | 0.7 | working | | eval | 0.8 | working | | version | 0.10 | working | +| parse | 1.3 | working | | ingest | 1.10 | scaffold | | plugin | 1.1 | scaffold | | query | 3.1 | scaffold | @@ -60,6 +61,109 @@ def version() -> None: typer.echo(f"ragctl {__version__}") +# --------------------------------------------------------------------------- +# parse — Step 1.3 (smoke-test built-in document parsers) +# --------------------------------------------------------------------------- + + +@app.command("parse") +def parse( + path: Path = typer.Argument( + ..., + exists=True, + file_okay=True, + dir_okay=False, + readable=True, + help="File to parse.", + ), + mime_type: str | None = typer.Option( + None, + "--mime", + help="Override MIME detection (e.g. 'text/markdown').", + ), + show_blocks: int = typer.Option( + 5, + "--blocks", + "-n", + help="Number of structural blocks to preview (0 to skip).", + ), +) -> None: + """Detect MIME, parse a file, and print a structural summary. + + Intended as a smoke-test / demo entry point for the built-in parsers + delivered in Step 1.3. The full ingest pipeline (connector → parse → + chunk → enrich → embed → store) lands in Step 1.10 via ``ragctl ingest``. + """ + import asyncio + + from rag_core.types import ( + CorpusId, + DocumentId, + Principal, + PrincipalId, + PrincipalKind, + RequestContext, + TenantId, + ) + from rag_parsers import default_registry, detect_mime + + data = path.read_bytes() + mime = mime_type or detect_mime(data, filename=path.name) + try: + parser = default_registry().select(mime) + except Exception as exc: # noqa: BLE001 — surface a clean CLI message + typer.echo(f"ERROR: no parser for MIME {mime!r}: {exc}", err=True) + raise typer.Exit(1) from exc + + tenant = TenantId("ragctl-local") + ctx = RequestContext( + tenant_id=tenant, + principal=Principal( + id=PrincipalId("ragctl"), + kind=PrincipalKind.service, + display_name="ragctl", + tenant_id=tenant, + ), + ) + + parsed = asyncio.run( + parser.parse( + ctx, + data, + mime, + DocumentId(f"local-{path.name}"), + CorpusId("ragctl-local"), + f"file://{path.resolve()}", + ) + ) + + typer.echo(f"file: {path}") + typer.echo(f"mime: {mime}") + typer.echo(f"parser: {type(parser).__name__}") + typer.echo(f"bytes: {len(data):,}") + typer.echo(f"sha256: {parsed.document.content_hash}") + typer.echo(f"text: {len(parsed.text):,} chars") + typer.echo(f"blocks: {len(parsed.blocks)} ({_block_summary(parsed.blocks)})") + + if show_blocks > 0 and parsed.blocks: + typer.echo("") + typer.echo("first blocks:") + for block in parsed.blocks[:show_blocks]: + preview = block.text.replace("\n", " ") + if len(preview) > 70: + preview = preview[:67] + "..." + level_str = f" h{block.level}" if block.level else "" + typer.echo(f" [{block.type.value}{level_str}] @{block.start}: {preview}") + + +def _block_summary(blocks: list) -> str: # type: ignore[type-arg] + """Compact per-type tally, e.g. ``heading=2, paragraph=10, table_row=15``.""" + counts: dict[str, int] = {} + for b in blocks: + counts[b.type.value] = counts.get(b.type.value, 0) + 1 + return ", ".join(f"{k}={v}" for k, v in sorted(counts.items())) or "—" + + # --------------------------------------------------------------------------- # config — Step 0.4 # --------------------------------------------------------------------------- diff --git a/packages/ragctl/tests/test_main.py b/packages/ragctl/tests/test_main.py index 5179eeb..a99d338 100644 --- a/packages/ragctl/tests/test_main.py +++ b/packages/ragctl/tests/test_main.py @@ -18,6 +18,7 @@ def test_root_help_lists_all_groups() -> None: "config", "eval", "traces", + "parse", "ingest", "query", "logs", diff --git a/packages/ragctl/tests/test_parse.py b/packages/ragctl/tests/test_parse.py new file mode 100644 index 0000000..be2ac35 --- /dev/null +++ b/packages/ragctl/tests/test_parse.py @@ -0,0 +1,52 @@ +"""Tests for ``ragctl parse`` — Step 1.3.""" + +from __future__ import annotations + +from pathlib import Path + +from ragctl.main import app +from typer.testing import CliRunner + +runner = CliRunner() + + +def test_parse_plain_text(tmp_path: Path) -> None: + sample = tmp_path / "hello.txt" + sample.write_text("first paragraph.\n\nsecond paragraph.\n") + result = runner.invoke(app, ["parse", str(sample)]) + assert result.exit_code == 0, result.output + assert "PlainTextParser" in result.output + assert "blocks: 2" in result.output + assert "paragraph=2" in result.output + + +def test_parse_markdown_blocks_preview(tmp_path: Path) -> None: + sample = tmp_path / "doc.md" + sample.write_text("# Heading\n\nA paragraph.\n\n- item\n") + result = runner.invoke(app, ["parse", str(sample), "-n", "3"]) + assert result.exit_code == 0, result.output + assert "MarkdownParser" in result.output + assert "heading" in result.output + assert "first blocks:" in result.output + + +def test_parse_mime_override(tmp_path: Path) -> None: + # File extension says .txt; --mime forces JSON parsing of a JSON body. + sample = tmp_path / "data.txt" + sample.write_text('{"a": 1}') + result = runner.invoke(app, ["parse", str(sample), "--mime", "application/json"]) + assert result.exit_code == 0, result.output + assert "JsonParser" in result.output + + +def test_parse_unknown_mime_exits_nonzero(tmp_path: Path) -> None: + sample = tmp_path / "weird.bin" + sample.write_bytes(b"\x00\x01\x02opaque") + result = runner.invoke(app, ["parse", str(sample), "--mime", "application/x-not-a-real-type"]) + assert result.exit_code != 0 + assert "no parser for MIME" in (result.output + (result.stderr or "")) + + +def test_parse_missing_file_exits_nonzero(tmp_path: Path) -> None: + result = runner.invoke(app, ["parse", str(tmp_path / "nope.txt")]) + assert result.exit_code != 0 diff --git a/pyproject.toml b/pyproject.toml index db45933..1805750 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,6 +9,7 @@ members = [ "packages/policy", "packages/ragctl", "packages/backends", + "packages/parsers", "apps/gateway", ] @@ -68,7 +69,7 @@ plugins = ["pydantic.mypy"] module = "tests.*" disallow_untyped_defs = false -# Third-party libraries without py.typed or stubs (backends package only). +# Third-party libraries without py.typed or stubs (backends + parsers packages). # gcloud.aio.* is behind the optional [gcs] extra — CI runs `uv sync` without # it, so mypy needs to tolerate the module being absent. [[tool.mypy.overrides]] @@ -80,6 +81,14 @@ module = [ "botocore.*", "qdrant_client.*", "gcloud.*", + # Parser libraries + "puremagic.*", + "markdown_it.*", + "bs4.*", + "pypdf.*", + "docx.*", + "pptx.*", + "openpyxl.*", ] ignore_missing_imports = true @@ -89,7 +98,7 @@ ignore_missing_imports = true [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["tests", "packages"] -pythonpath = ["packages/core/src", "packages/config/src", "packages/observability/src", "packages/policy/src", "packages/ragctl/src", "packages/backends/src", "apps/gateway/src"] +pythonpath = ["packages/core/src", "packages/config/src", "packages/observability/src", "packages/policy/src", "packages/ragctl/src", "packages/backends/src", "packages/parsers/src", "apps/gateway/src"] # spi_signature.py is the SPI signature linter (Step 1.1a); referenced by name in # docs/architecture/request-context.md. Collected alongside test_*.py files. # coverage.py is the PolicyEngine coverage linter (Step 1.1c). diff --git a/tests/contract/test_parser.py b/tests/contract/test_parser.py index f0157e8..c368fb4 100644 --- a/tests/contract/test_parser.py +++ b/tests/contract/test_parser.py @@ -2,7 +2,7 @@ import pytest from rag_core.spi.noop import NoopParser -from rag_core.types import CorpusId, DocumentId, DocumentStatus, RequestContext +from rag_core.types import BlockType, CorpusId, DocumentId, DocumentStatus, RequestContext pytestmark = pytest.mark.contract @@ -16,10 +16,10 @@ async def test_supports_any_mime(parser: NoopParser) -> None: assert parser.supports(mime) is True -async def test_parse_returns_document( +async def test_parse_returns_parsed_document( parser: NoopParser, ctx: RequestContext, corpus_id: CorpusId ) -> None: - doc = await parser.parse( + parsed = await parser.parse( ctx, b"hello world", "text/plain", @@ -27,10 +27,14 @@ async def test_parse_returns_document( corpus_id, "file:///test.txt", ) - assert doc.status == DocumentStatus.ready - assert doc.content_hash != "" - assert doc.tenant_id == ctx.tenant_id - assert doc.corpus_id == corpus_id + assert parsed.document.status == DocumentStatus.ready + assert parsed.document.content_hash != "" + assert parsed.document.tenant_id == ctx.tenant_id + assert parsed.document.corpus_id == corpus_id + assert parsed.text == "hello world" + # Empty-body parsers may emit zero blocks; non-empty must emit at least one. + assert len(parsed.blocks) >= 1 + assert all(b.start >= 0 for b in parsed.blocks) async def test_parse_content_hash_deterministic( @@ -42,6 +46,40 @@ async def test_parse_content_hash_deterministic( corpus_id=corpus_id, source_uri="file:///a.txt", ) - doc1 = await parser.parse(ctx, b"same content", **kwargs) - doc2 = await parser.parse(ctx, b"same content", **kwargs) - assert doc1.content_hash == doc2.content_hash + p1 = await parser.parse(ctx, b"same content", **kwargs) + p2 = await parser.parse(ctx, b"same content", **kwargs) + assert p1.document.content_hash == p2.document.content_hash + + +async def test_parse_empty_input( + parser: NoopParser, ctx: RequestContext, corpus_id: CorpusId +) -> None: + parsed = await parser.parse( + ctx, + b"", + "text/plain", + DocumentId("doc-empty"), + corpus_id, + "file:///empty.txt", + ) + assert parsed.text == "" + assert parsed.blocks == [] + assert parsed.document.content_hash != "" + + +async def test_parse_block_offsets_within_text( + parser: NoopParser, ctx: RequestContext, corpus_id: CorpusId +) -> None: + parsed = await parser.parse( + ctx, + b"hello world", + "text/plain", + DocumentId("doc-2"), + corpus_id, + "file:///test.txt", + ) + for block in parsed.blocks: + end = block.start + len(block.text) + assert end <= len(parsed.text) + if block.type == BlockType.heading: + assert block.level is not None and 1 <= block.level <= 6 diff --git a/tests/parsers/__init__.py b/tests/parsers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/parsers/conftest.py b/tests/parsers/conftest.py new file mode 100644 index 0000000..a1ad2ee --- /dev/null +++ b/tests/parsers/conftest.py @@ -0,0 +1,42 @@ +"""Shared fixtures for rag-parsers unit tests.""" + +from __future__ import annotations + +import pytest +from rag_core.types import ( + CorpusId, + DocumentId, + Principal, + PrincipalId, + PrincipalKind, + RequestContext, + TenantId, +) + + +@pytest.fixture +def tenant_id() -> TenantId: + return TenantId("t-test") + + +@pytest.fixture +def corpus_id() -> CorpusId: + return CorpusId("c-test") + + +@pytest.fixture +def document_id() -> DocumentId: + return DocumentId("d-test") + + +@pytest.fixture +def ctx(tenant_id: TenantId) -> RequestContext: + return RequestContext( + tenant_id=tenant_id, + principal=Principal( + id=PrincipalId("alice"), + kind=PrincipalKind.user, + display_name="alice", + tenant_id=tenant_id, + ), + ) diff --git a/tests/parsers/test_binary_parsers.py b/tests/parsers/test_binary_parsers.py new file mode 100644 index 0000000..54368fc --- /dev/null +++ b/tests/parsers/test_binary_parsers.py @@ -0,0 +1,228 @@ +"""Unit tests for binary-format parsers: PDF, DOCX, PPTX, XLSX. + +Each test generates its fixture in memory using the same library the parser +consumes — keeps the suite hermetic without committing binary blobs. These +are slow-ish (~50–200 ms each) and gated behind the default marker only. +""" + +from __future__ import annotations + +import io + +import pytest +from docx import Document as DocxDocument +from openpyxl import Workbook +from pptx import Presentation +from pypdf import PdfWriter +from pypdf.generic import DictionaryObject, NameObject, NumberObject, StreamObject +from rag_core.errors import ParseError +from rag_core.types import BlockType, CorpusId, DocumentId, RequestContext +from rag_parsers import DocxParser, PdfParser, PptxParser, XlsxParser + +# --------------------------------------------------------------------------- +# Fixture builders +# --------------------------------------------------------------------------- + + +def _make_pdf(text: str = "Hello PDF") -> bytes: + """Assemble a minimal one-page PDF that round-trips through pypdf.""" + writer = PdfWriter() + page = writer.add_blank_page(width=612, height=792) + + ops = f"BT\n/F1 24 Tf\n100 700 Td\n({text}) Tj\nET".encode() + stream = StreamObject() + stream._data = ops + stream[NameObject("/Length")] = NumberObject(len(ops)) + stream_ref = writer._add_object(stream) + page[NameObject("/Contents")] = stream_ref + + font_dict = DictionaryObject( + { + NameObject("/Type"): NameObject("/Font"), + NameObject("/Subtype"): NameObject("/Type1"), + NameObject("/BaseFont"): NameObject("/Helvetica"), + } + ) + font_ref = writer._add_object(font_dict) + resources = DictionaryObject() + resources[NameObject("/Font")] = DictionaryObject({NameObject("/F1"): font_ref}) + page[NameObject("/Resources")] = resources + + buf = io.BytesIO() + writer.write(buf) + return buf.getvalue() + + +def _make_docx() -> bytes: + doc = DocxDocument() + doc.add_heading("Section One", level=1) + doc.add_paragraph("First body paragraph.") + doc.add_heading("Section Two", level=2) + doc.add_paragraph("Second body paragraph.") + table = doc.add_table(rows=2, cols=2) + table.cell(0, 0).text = "A" + table.cell(0, 1).text = "B" + table.cell(1, 0).text = "1" + table.cell(1, 1).text = "2" + buf = io.BytesIO() + doc.save(buf) + return buf.getvalue() + + +def _make_pptx() -> bytes: + prs = Presentation() + layout = prs.slide_layouts[1] # Title and Content + slide = prs.slides.add_slide(layout) + slide.shapes.title.text = "Deck Title" + body = slide.placeholders[1] + body.text_frame.text = "First bullet" + body.text_frame.add_paragraph().text = "Second bullet" + # Notes + slide.notes_slide.notes_text_frame.text = "Speaker notes here." + buf = io.BytesIO() + prs.save(buf) + return buf.getvalue() + + +def _make_xlsx() -> bytes: + wb = Workbook() + ws = wb.active + ws.title = "Q1" + ws.append(["name", "score"]) + ws.append(["alice", 90]) + ws.append(["bob", 85]) + ws2 = wb.create_sheet("Q2") + ws2.append(["name", "score"]) + ws2.append(["carol", 77]) + buf = io.BytesIO() + wb.save(buf) + return buf.getvalue() + + +# --------------------------------------------------------------------------- +# PDF +# --------------------------------------------------------------------------- + + +async def test_pdf_supports() -> None: + assert PdfParser().supports("application/pdf") + assert not PdfParser().supports("text/plain") + + +async def test_pdf_extracts_pages( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + data = _make_pdf("Hello PDF World") + parsed = await PdfParser().parse( + ctx, data, "application/pdf", document_id, corpus_id, "file:///x.pdf" + ) + assert "Hello PDF World" in parsed.text + assert len(parsed.blocks) >= 1 + assert all(b.type == BlockType.paragraph for b in parsed.blocks) + assert parsed.blocks[0].attributes["page"] == 1 + + +async def test_pdf_invalid_bytes_raises( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + with pytest.raises(ParseError): + await PdfParser().parse( + ctx, + b"not a pdf", + "application/pdf", + document_id, + corpus_id, + "file:///x.pdf", + ) + + +# --------------------------------------------------------------------------- +# DOCX +# --------------------------------------------------------------------------- + + +async def test_docx_headings_and_table( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + data = _make_docx() + parsed = await DocxParser().parse( + ctx, + data, + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + document_id, + corpus_id, + "file:///x.docx", + ) + heading_levels = [b.level for b in parsed.blocks if b.type == BlockType.heading] + assert 1 in heading_levels + assert 2 in heading_levels + assert any(b.type == BlockType.paragraph for b in parsed.blocks) + table_rows = [b for b in parsed.blocks if b.type == BlockType.table_row] + assert len(table_rows) == 2 + assert table_rows[0].attributes == {"table": 1, "row": 1} + + +async def test_docx_invalid_bytes_raises( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + with pytest.raises(ParseError): + await DocxParser().parse( + ctx, + b"not docx", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + document_id, + corpus_id, + "file:///x.docx", + ) + + +# --------------------------------------------------------------------------- +# PPTX +# --------------------------------------------------------------------------- + + +async def test_pptx_title_body_notes( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + data = _make_pptx() + parsed = await PptxParser().parse( + ctx, + data, + "application/vnd.openxmlformats-officedocument.presentationml.presentation", + document_id, + corpus_id, + "file:///x.pptx", + ) + headings = [b for b in parsed.blocks if b.type == BlockType.heading] + assert headings, "expected at least one slide title as a heading" + assert headings[0].text == "Deck Title" + assert headings[0].level == 2 + bodies = [b for b in parsed.blocks if b.type == BlockType.paragraph] + assert any("First bullet" in b.text for b in bodies) + notes = [b for b in parsed.blocks if b.type == BlockType.caption] + assert notes and notes[0].attributes["kind"] == "notes" + + +# --------------------------------------------------------------------------- +# XLSX +# --------------------------------------------------------------------------- + + +async def test_xlsx_multi_sheet( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + data = _make_xlsx() + parsed = await XlsxParser().parse( + ctx, + data, + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + document_id, + corpus_id, + "file:///x.xlsx", + ) + sheet_headings = [b for b in parsed.blocks if b.type == BlockType.heading] + assert {h.text for h in sheet_headings} == {"Q1", "Q2"} + rows = [b for b in parsed.blocks if b.type == BlockType.table_row] + # Q1: header + 2 data rows = 3; Q2: header + 1 data row = 2. + assert len(rows) == 5 + assert {r.attributes["sheet"] for r in rows} == {"Q1", "Q2"} diff --git a/tests/parsers/test_mime.py b/tests/parsers/test_mime.py new file mode 100644 index 0000000..f838e9c --- /dev/null +++ b/tests/parsers/test_mime.py @@ -0,0 +1,56 @@ +"""Tests for the MIME detection helper.""" + +from __future__ import annotations + +import pytest +from rag_parsers.mime import detect_mime + + +def test_requires_data_or_filename() -> None: + with pytest.raises(ValueError): + detect_mime() + + +def test_extension_only() -> None: + assert detect_mime(filename="report.pdf") == "application/pdf" + assert detect_mime(filename="notes.md") == "text/markdown" + assert detect_mime(filename="data.csv") == "text/csv" + assert detect_mime(filename="rows.tsv") == "text/tab-separated-values" + assert detect_mime(filename="data.json") == "application/json" + assert detect_mime(filename="config.yaml") == "application/yaml" + assert detect_mime(filename="config.yml") == "application/yaml" + + +def test_extension_office() -> None: + docx = detect_mime(filename="memo.docx") + pptx = detect_mime(filename="deck.pptx") + xlsx = detect_mime(filename="sheet.xlsx") + assert docx.endswith("wordprocessingml.document") + assert pptx.endswith("presentationml.presentation") + assert xlsx.endswith("spreadsheetml.sheet") + + +def test_pdf_magic_sniff() -> None: + # %PDF-1.4 header without filename. + pdf_bytes = b"%PDF-1.4\n%\xe2\xe3\xcf\xd3\n" + b"0" * 200 + assert detect_mime(pdf_bytes) == "application/pdf" + + +def test_html_sniff() -> None: + html = b"

hi

" + mime = detect_mime(html, filename="x") + # Either extension lookup (none for "x") or content sniff should land here. + assert mime in {"text/html", "application/octet-stream"} + # With a filename hint, extension wins. + assert detect_mime(html, filename="x.html") == "text/html" + + +def test_hint_fallback() -> None: + # No data, unknown extension → hint is used. + assert ( + detect_mime(filename="weird.unknown", hint="application/x-thing") == "application/x-thing" + ) + + +def test_default_octet_stream() -> None: + assert detect_mime(b"\x00\x01\x02\x03random", filename="x") == "application/octet-stream" diff --git a/tests/parsers/test_registry.py b/tests/parsers/test_registry.py new file mode 100644 index 0000000..7aa8dea --- /dev/null +++ b/tests/parsers/test_registry.py @@ -0,0 +1,51 @@ +"""Tests for ParserRegistry and default_registry().""" + +from __future__ import annotations + +import pytest +from rag_core.errors import ParseError +from rag_parsers import ( + HtmlParser, + JsonParser, + ParserRegistry, + PdfParser, + PlainTextParser, + default_registry, +) + + +def test_default_registry_resolves_known_mimes() -> None: + registry = default_registry() + cases: list[tuple[str, type]] = [ + ("text/plain", PlainTextParser), + ("text/markdown", type(registry.select("text/markdown"))), + ("application/json", JsonParser), + ("text/html", HtmlParser), + ("application/pdf", PdfParser), + ] + for mime, expected_type in cases: + parser = registry.select(mime) + assert isinstance(parser, expected_type) + + +def test_default_registry_unknown_raises() -> None: + registry = default_registry() + with pytest.raises(ParseError): + registry.select("application/x-totally-unknown-format") + + +def test_registry_first_match_wins() -> None: + registry = ParserRegistry() + # PlainTextParser is a wildcard for text/* — register it first. + registry.register(PlainTextParser()) + registry.register(HtmlParser()) + # text/html accepts both, but PlainTextParser was registered first. + parser = registry.select("text/html") + assert isinstance(parser, PlainTextParser) + + +def test_registry_len_and_iter() -> None: + registry = default_registry() + assert len(registry) >= 10 # 10 built-in parsers + parsers = list(registry) + assert len(parsers) == len(registry) diff --git a/tests/parsers/test_text_family.py b/tests/parsers/test_text_family.py new file mode 100644 index 0000000..da7a2cd --- /dev/null +++ b/tests/parsers/test_text_family.py @@ -0,0 +1,278 @@ +"""Unit tests for text-family parsers: plain text, Markdown, JSON, CSV, YAML, HTML.""" + +from __future__ import annotations + +import pytest +from rag_core.errors import ParseError +from rag_core.types import BlockType, CorpusId, DocumentId, RequestContext +from rag_parsers import ( + CsvParser, + HtmlParser, + JsonParser, + MarkdownParser, + PlainTextParser, + YamlParser, +) + + +async def _parse(parser, ctx, mime, data, document_id, corpus_id, source_uri="file:///x"): + return await parser.parse(ctx, data, mime, document_id, corpus_id, source_uri) + + +# --------------------------------------------------------------------------- +# Plain text +# --------------------------------------------------------------------------- + + +async def test_plain_text_supports() -> None: + parser = PlainTextParser() + assert parser.supports("text/plain") + assert parser.supports("text/anything") + assert not parser.supports("application/json") + + +async def test_plain_text_paragraph_split( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + body = "first paragraph.\n\nsecond paragraph.\n\nthird." + parsed = await _parse( + PlainTextParser(), ctx, "text/plain", body.encode(), document_id, corpus_id + ) + assert parsed.text == body + assert len(parsed.blocks) == 3 + assert all(b.type == BlockType.paragraph for b in parsed.blocks) + for block in parsed.blocks: + assert parsed.text[block.start : block.start + len(block.text)] == block.text + + +async def test_plain_text_bom_stripped( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + PlainTextParser(), + ctx, + "text/plain", + b"\xef\xbb\xbfhello", + document_id, + corpus_id, + ) + assert parsed.text == "hello" + + +async def test_plain_text_empty( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse(PlainTextParser(), ctx, "text/plain", b"", document_id, corpus_id) + assert parsed.text == "" + assert parsed.blocks == [] + + +# --------------------------------------------------------------------------- +# Markdown +# --------------------------------------------------------------------------- + + +async def test_markdown_heading_levels( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + src = "# H1\n\n## H2\n\nA paragraph.\n\n- bullet one\n- bullet two\n\n```\nlet x = 1;\n```\n" + parsed = await _parse( + MarkdownParser(), ctx, "text/markdown", src.encode(), document_id, corpus_id + ) + by_type: dict[BlockType, list] = {} + for block in parsed.blocks: + by_type.setdefault(block.type, []).append(block) + headings = by_type[BlockType.heading] + assert [h.level for h in headings] == [1, 2] + assert headings[0].text == "H1" + assert any(b.type == BlockType.list_item for b in parsed.blocks) + assert any(b.type == BlockType.code for b in parsed.blocks) + + +async def test_markdown_empty_input( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse(MarkdownParser(), ctx, "text/markdown", b"", document_id, corpus_id) + assert parsed.text == "" + assert parsed.blocks == [] + + +async def test_markdown_invalid_utf8( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + with pytest.raises(ParseError): + await _parse(MarkdownParser(), ctx, "text/markdown", b"\xff\xfe", document_id, corpus_id) + + +# --------------------------------------------------------------------------- +# JSON +# --------------------------------------------------------------------------- + + +async def test_json_pretty_printed( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + JsonParser(), + ctx, + "application/json", + b'{"b":2,"a":1}', + document_id, + corpus_id, + ) + # Sorted keys + indent=2. + assert '"a": 1' in parsed.text + assert '"b": 2' in parsed.text + assert parsed.text.index('"a"') < parsed.text.index('"b"') + assert len(parsed.blocks) == 1 + assert parsed.blocks[0].type == BlockType.paragraph + + +async def test_json_ndjson( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + JsonParser(), + ctx, + "application/x-ndjson", + b'{"a":1}\n{"a":2}\n', + document_id, + corpus_id, + ) + assert parsed.text.count('"a"') == 2 + + +async def test_json_invalid_raises( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + with pytest.raises(ParseError): + await _parse( + JsonParser(), + ctx, + "application/json", + b"{not json", + document_id, + corpus_id, + ) + + +# --------------------------------------------------------------------------- +# CSV +# --------------------------------------------------------------------------- + + +async def test_csv_rows_and_header( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + csv_bytes = b"name,age\nalice,30\nbob,40\n" + parsed = await _parse(CsvParser(), ctx, "text/csv", csv_bytes, document_id, corpus_id) + assert len(parsed.blocks) == 3 # header + 2 rows + assert parsed.blocks[0].attributes == {"row": 0, "header": True} + assert parsed.blocks[1].attributes["row"] == 1 + assert parsed.blocks[1].attributes["columns"] == ["name", "age"] + + +async def test_tsv_delimiter( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + CsvParser(), + ctx, + "text/tab-separated-values", + b"a\tb\n1\t2\n", + document_id, + corpus_id, + ) + assert parsed.text == "a\tb\n1\t2" + + +# --------------------------------------------------------------------------- +# YAML +# --------------------------------------------------------------------------- + + +async def test_yaml_roundtrip( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + YamlParser(), + ctx, + "application/yaml", + b"name: alice\nage: 30\n", + document_id, + corpus_id, + ) + assert "name: alice" in parsed.text + assert "age: 30" in parsed.text + assert len(parsed.blocks) == 1 + + +async def test_yaml_multi_doc( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + YamlParser(), + ctx, + "application/yaml", + b"a: 1\n---\nb: 2\n", + document_id, + corpus_id, + ) + assert "---" in parsed.text + + +async def test_yaml_invalid_raises( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + with pytest.raises(ParseError): + await _parse( + YamlParser(), + ctx, + "application/yaml", + b"key: [unterminated", + document_id, + corpus_id, + ) + + +# --------------------------------------------------------------------------- +# HTML +# --------------------------------------------------------------------------- + + +async def test_html_extracts_blocks( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + html = b"""x + +

Title

+

Some paragraph.

+
  • one
  • two
+ 
let x = 1;
+ """ + parsed = await _parse(HtmlParser(), ctx, "text/html", html, document_id, corpus_id) + by_type: dict[BlockType, list] = {} + for block in parsed.blocks: + by_type.setdefault(block.type, []).append(block) + assert BlockType.heading in by_type + assert by_type[BlockType.heading][0].text == "Title" + assert by_type[BlockType.heading][0].level == 1 + assert BlockType.paragraph in by_type + assert BlockType.list_item in by_type + # script content must be dropped. + assert "alert(1)" not in parsed.text + + +async def test_html_empty_body( + ctx: RequestContext, corpus_id: CorpusId, document_id: DocumentId +) -> None: + parsed = await _parse( + HtmlParser(), + ctx, + "text/html", + b"", + document_id, + corpus_id, + ) + assert parsed.text == "" + assert parsed.blocks == [] diff --git a/uv.lock b/uv.lock index e7d0219..0477d17 100644 --- a/uv.lock +++ b/uv.lock @@ -23,6 +23,7 @@ members = [ "rag-core", "rag-gateway", "rag-observability", + "rag-parsers", "rag-policy", "rag-ragctl", ] @@ -392,6 +393,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/73/b6e24bd22e6720ca8ee9a85a0c4a2971af8497d8f3193fa05390cbd46e09/backoff-2.2.1-py3-none-any.whl", hash = "sha256:63579f9a0628e06278f7e47b7d7d5b6ce20dc65c5e96a6f3ca99a6adca0396e8", size = 15148, upload-time = "2022-10-05T19:19:30.546Z" }, ] +[[package]] +name = "beautifulsoup4" +version = "4.14.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "soupsieve" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c3/b0/1c6a16426d389813b48d95e26898aff79abbde42ad353958ad95cc8c9b21/beautifulsoup4-4.14.3.tar.gz", hash = "sha256:6292b1c5186d356bba669ef9f7f051757099565ad9ada5dd630bd9de5fa7fb86", size = 627737, upload-time = "2025-11-30T15:08:26.084Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1a/39/47f9197bdd44df24d67ac8893641e16f386c984a0619ef2ee4c51fbbc019/beautifulsoup4-4.14.3-py3-none-any.whl", hash = "sha256:0918bfe44902e6ad8d57732ba310582e98da931428d231a5ecb9e7c703a735bb", size = 107721, upload-time = "2025-11-30T15:08:24.087Z" }, +] + [[package]] name = "boolean-py" version = "5.0" @@ -869,6 +883,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a7/5f/ed01f9a3cdffbd5a008556fc7b2a08ddb1cc6ace7effa7340604b1d16699/docstring_parser-0.18.0-py3-none-any.whl", hash = "sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b", size = 22484, upload-time = "2026-04-14T04:09:18.638Z" }, ] +[[package]] +name = "et-xmlfile" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d3/38/af70d7ab1ae9d4da450eeec1fa3918940a5fafb9055e934af8d6eb0c2313/et_xmlfile-2.0.0.tar.gz", hash = "sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54", size = 17234, upload-time = "2024-10-25T17:25:40.039Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/8b/5fe2cc11fee489817272089c4203e679c63b570a5aaeb18d852ae3cbba6a/et_xmlfile-2.0.0-py3-none-any.whl", hash = "sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa", size = 18059, upload-time = "2024-10-25T17:25:39.051Z" }, +] + [[package]] name = "fastapi" version = "0.136.1" @@ -1727,6 +1750,86 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/af/40/791891d4c0c4dab4c5e187c17261cedc26285fd41541577f900470a45a4d/license_expression-30.4.4-py3-none-any.whl", hash = "sha256:421788fdcadb41f049d2dc934ce666626265aeccefddd25e162a26f23bcbf8a4", size = 120615, upload-time = "2025-07-22T11:13:31.217Z" }, ] +[[package]] +name = "lxml" +version = "6.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/3b/aab6728cae887456f409b4d75e8a01856e4f04bd510de38052a47768b680/lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40", size = 4197430, upload-time = "2026-05-18T19:19:06.424Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/6e/c4add832b6fc1e887125b96f880d7b9b70aae5248718e046b1704bcac4b9/lxml-6.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:104c09bda8d2a562824c0e319d0768ce26a779b7601e0931d33b09b53c392ef7", size = 8570821, upload-time = "2026-05-18T19:17:42.068Z" }, + { url = "https://files.pythonhosted.org/packages/22/00/ff3009c88e65de8011630acf8ab5a09cb2becd2aaf47fba2f3449f6224e9/lxml-6.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25c6997a9a534e016695a0ba06b2f07945de682731ff01065b6d5a4474179da1", size = 4624252, upload-time = "2026-05-18T19:17:47.897Z" }, + { url = "https://files.pythonhosted.org/packages/42/95/bb63f0fd62e554fe078e1fb3c8fe9083c14ddc7ad7fa178d10e57e071ac7/lxml-6.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c921ba5c51e4e9f63b8b00267d06566e1f63407408a0496da2d1d0bfc819c7fc", size = 4930746, upload-time = "2026-05-18T19:18:29.637Z" }, + { url = "https://files.pythonhosted.org/packages/eb/99/0013e8d9b5960f4f041cf0b73e2f80c23eb5205b1f7bfb20203243651359/lxml-6.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:54a7f95e4de5fb94e2f9f4b9055c6ba33bf3d628fd77a1d647c5923caa2cdcdc", size = 5093723, upload-time = "2026-05-18T19:18:34.168Z" }, + { url = "https://files.pythonhosted.org/packages/29/91/317b332636bfc7bddcff828d41b3307f50043f4b237e40849c333d80fa1a/lxml-6.1.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f2ec43df44b1f76249ee0a615334f9b5b060e1c8bd90e706dad2d14d02f383", size = 5005557, upload-time = "2026-05-18T19:18:39.798Z" }, + { url = "https://files.pythonhosted.org/packages/42/2f/cc9bf06afe70f9c9093ae60855d9759da9db601ec4080f7473319666ffd7/lxml-6.1.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:70ef8a7e102a1508f8121aae5b0867abd663f72c14f0a9c937e6554cb4587b7b", size = 5631036, upload-time = "2026-05-18T19:18:44.858Z" }, + { url = "https://files.pythonhosted.org/packages/08/f6/af32e23e563971ffb0fb86be52bc5be5c2c118858ffc119bf6a9039b173d/lxml-6.1.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ebe6af670449830d6d9b752c256a983291c766a1365ba5d5460048f9e33a7818", size = 5240367, upload-time = "2026-05-18T19:18:49.217Z" }, + { url = "https://files.pythonhosted.org/packages/78/83/8555d40948b09ce86f1bd0c68a7ac31d07b1929f92cc1b074006c97ef2d2/lxml-6.1.1-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:27acc820660aaffa4f7c087f29120e12980f7779d56d8492d263170111284740", size = 5350171, upload-time = "2026-05-18T19:18:52.779Z" }, + { url = "https://files.pythonhosted.org/packages/63/75/5d92da93729b7bad783689e6496049fa40927b45bec7bf183c981de3ca70/lxml-6.1.1-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:1db753c9115ec7100d073b744d17e25e88a8f90f5c39b2f5dd878149af59671f", size = 4694874, upload-time = "2026-05-18T19:18:55.139Z" }, + { url = "https://files.pythonhosted.org/packages/c5/b5/3aad415a9a25b822e783f15deeb4dffccf5113030f1afa2222dd929313d9/lxml-6.1.1-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4f469aebd783bb741c2ecb2a681008fd26bfe5c16a9a72ed5467f834e810df2", size = 5244492, upload-time = "2026-05-18T19:19:01.28Z" }, + { url = "https://files.pythonhosted.org/packages/f1/a1/5fcf7eb9904b80086aa47dcf0027de07b1bb990afad2e6823144c368ae04/lxml-6.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:766b010012d59470072c1816b5b6c69f1d243e5db36ea5968e94accf430a4635", size = 5048232, upload-time = "2026-05-18T19:18:12.67Z" }, + { url = "https://files.pythonhosted.org/packages/77/74/1f601b63c7a69fcdf10fa9b148c81da8442204194f6c55509cc485c786b9/lxml-6.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b8d812c6011c08b8111a15e54dd990b8923692d80adf35488bee34026c35accf", size = 4777023, upload-time = "2026-05-18T19:18:15.928Z" }, + { url = "https://files.pythonhosted.org/packages/a2/b9/7a78f51aec95b1bf780d78e12705a9f6533284f8693dc5c0e6724fa53d3f/lxml-6.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:fe0306bd29505a9177aac19f1877174b0e7422c222a59f70b2cd41633448c3dc", size = 5645773, upload-time = "2026-05-18T19:18:23.223Z" }, + { url = "https://files.pythonhosted.org/packages/a5/6e/98a7b7ad54e4e74fa1f20fff776913980619d0ebe5558232d7da6580bdd8/lxml-6.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5ba186ad207446c65d3bb3d3e0412b032b1d9f595e59861e2354798c5703d955", size = 5233088, upload-time = "2026-05-18T19:18:31.433Z" }, + { url = "https://files.pythonhosted.org/packages/65/d1/bc0ed2427bf609f2ee10da303a6a226f9c8bce94f945dc29a32ce55de6e4/lxml-6.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aa366a1e55b8ebfe8ca8ddc3cfe75c8ebade181aeb0f661d0cb05986b647f72a", size = 5260995, upload-time = "2026-05-18T19:18:37.091Z" }, + { url = "https://files.pythonhosted.org/packages/69/8b/6772e1a4b513fc50a8d931f19edde0e13ae6918510a1e13ff67864f3e5ed/lxml-6.1.1-cp312-cp312-win32.whl", hash = "sha256:126c93f7f56f0eda92f6d8c619edc463a4f23d9252f1c9d0405a76f25fa9f11a", size = 3596382, upload-time = "2026-05-18T19:17:18.37Z" }, + { url = "https://files.pythonhosted.org/packages/1b/89/45198e9624762af2dfd2cb8782598477ceb29f6e59caab560388ae1f4ec1/lxml-6.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:26e6eda8d38c1fcab1090dd196ee87cbd13788e531937610e2589085de074e77", size = 3997255, upload-time = "2026-05-18T19:17:56.781Z" }, + { url = "https://files.pythonhosted.org/packages/90/a9/7a54b6834088d9ae528a7b780584ba6a39a9457b0ac330479f20ffbc9449/lxml-6.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:6540377fbd53fe1b629172288c464fb18db11ce1fa7dc15891da10aa9dcc3e7f", size = 3659610, upload-time = "2026-05-19T19:22:50.843Z" }, + { url = "https://files.pythonhosted.org/packages/a5/eb/7e6f37c5584ccbb2ff267f56fd0339016938c1c8684cfefab9b33ffc2f36/lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736", size = 8559780, upload-time = "2026-05-18T19:17:57.661Z" }, + { url = "https://files.pythonhosted.org/packages/a1/36/587c2521cf23a2cd6c9c22108aa7528f683a1f195ed7ccd23a4b1786ad36/lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9", size = 4618006, upload-time = "2026-05-18T19:18:04.452Z" }, + { url = "https://files.pythonhosted.org/packages/6e/ca/ab7bfe2bf4c972af5e7878262845ead3a24a929a9b04bc11c7c1ece6c82a/lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354", size = 4924139, upload-time = "2026-05-18T19:19:04.873Z" }, + { url = "https://files.pythonhosted.org/packages/6b/55/a0c72851dfee5ecc689f949723a73dea457758912542cb955b108eaf0d8f/lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca", size = 5082329, upload-time = "2026-05-18T19:19:09.728Z" }, + { url = "https://files.pythonhosted.org/packages/f0/b6/0608f7d61a3b96cc67e5648a3d906e31a5082093e10e7be65b3886289938/lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099", size = 4993564, upload-time = "2026-05-18T19:19:13.608Z" }, + { url = "https://files.pythonhosted.org/packages/4c/66/ae227524b066d29d55bf0b453d93d2d793c40218657d643dcbbca13b8faf/lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6", size = 5613467, upload-time = "2026-05-18T19:19:16.228Z" }, + { url = "https://files.pythonhosted.org/packages/a6/76/dbe4a00b50385e40194231dcfe5a12c059de7cf90e89c83407d2b085b719/lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085", size = 5228304, upload-time = "2026-05-18T19:19:19.354Z" }, + { url = "https://files.pythonhosted.org/packages/1c/01/00b1b8442ed2041793336868ba0b9ea4b13d7da7c085c6404c207a63bf79/lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e", size = 5341607, upload-time = "2026-05-18T19:19:22.297Z" }, + { url = "https://files.pythonhosted.org/packages/63/36/1ad29931e9a4638bb707869f01d423a6c815f82152138d1a40dfcfde2b95/lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f", size = 4700168, upload-time = "2026-05-18T19:19:25.133Z" }, + { url = "https://files.pythonhosted.org/packages/3c/d1/a9536cecf9be18a0dc72d32bead283a2332d1ffebd2dd3ac70ce444686e5/lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c", size = 5232487, upload-time = "2026-05-18T19:19:28.603Z" }, + { url = "https://files.pythonhosted.org/packages/0e/77/b4fb1e03bf5d130e879214d3100092e386418807fb74dd0adc4b0a48f351/lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b", size = 5044231, upload-time = "2026-05-18T19:18:42.246Z" }, + { url = "https://files.pythonhosted.org/packages/26/4c/d00daeeb0a5530c4028a9232aa1b93db3ef4ed2158c116ea73c79a9765b3/lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2", size = 4769450, upload-time = "2026-05-18T19:18:48.013Z" }, + { url = "https://files.pythonhosted.org/packages/ed/6a/715a3a8d156ce42f29cf014706f5410c2ff3b02267774110fc23266409fe/lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5", size = 5635874, upload-time = "2026-05-18T19:18:51.914Z" }, + { url = "https://files.pythonhosted.org/packages/45/37/0544bc21dde2a88f3a17b504e6fc79c0e01d25a33c2f6079724e9e72b9c7/lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785", size = 5223987, upload-time = "2026-05-18T19:18:59.715Z" }, + { url = "https://files.pythonhosted.org/packages/4d/f8/f6a5e8185bcb28c2befae3d31f8e3df3b811cb0f47746517a81279fcafe1/lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947", size = 5250276, upload-time = "2026-05-18T19:19:03.834Z" }, + { url = "https://files.pythonhosted.org/packages/c7/f2/1a2b9f1b7a49d45495369be7ef9ad05b262930f2eab3e3145706fca8083f/lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca", size = 3596903, upload-time = "2026-05-18T19:17:29.863Z" }, + { url = "https://files.pythonhosted.org/packages/e6/99/f4ffb024f238eec2131aaa09f3278fb6129cf892741bf68e1fc1afb8c100/lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660", size = 3995869, upload-time = "2026-05-18T19:18:02.596Z" }, + { url = "https://files.pythonhosted.org/packages/d1/53/70eb8c5c6037f27448f1e3c54ebede9545a801ae63f0a7254afca4fe8e45/lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc", size = 3658490, upload-time = "2026-05-19T19:22:53.846Z" }, + { url = "https://files.pythonhosted.org/packages/13/e2/2e325795566de01d0d7c3bb57d3c370616b2d07b01214e84eec5d3b10963/lxml-6.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:19b7ab10b210b0b3ad7985d9ac4eb66ab09a90b20fe6e2f7ba55d01a234345d0", size = 8577146, upload-time = "2026-05-18T19:18:17.765Z" }, + { url = "https://files.pythonhosted.org/packages/93/cf/5630b5e4be7d2e6bee8efe83865c925221103cf0221303b104ce134b01e2/lxml-6.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c08e5c694306507275f2290073350c4f32e383db15213b2c69e7ff39c1193840", size = 4623866, upload-time = "2026-05-18T19:18:30.669Z" }, + { url = "https://files.pythonhosted.org/packages/d2/51/3904907c063451cf8d4a5c9fe0cad95fa1f4ec57f4e3884fa0731bd7a305/lxml-6.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:74a9717fd0d82effef5c2854f0d917231d5324b5a3eb7275c43ac9fa32f97a14", size = 4950022, upload-time = "2026-05-18T19:19:31.958Z" }, + { url = "https://files.pythonhosted.org/packages/94/cd/9c7611a51c37a2830928405817cc5d56a97f64fab83cc3f628748b135749/lxml-6.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:efe0374196335f93b53269acd811b944f2e6bdc88e8894f214bd636455484909", size = 5086695, upload-time = "2026-05-18T19:19:34.764Z" }, + { url = "https://files.pythonhosted.org/packages/da/d6/24e3b5906abb0b674ff2ae195bc3ce59708df2bcd17cf17703b2d7dd643a/lxml-6.1.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac931cdc9442c1763b8a8f6cd62c0c938737eafc5be75eff88df55fc73bc0d00", size = 5031642, upload-time = "2026-05-18T19:19:37.771Z" }, + { url = "https://files.pythonhosted.org/packages/2d/db/6ec54f99019838bff54785c51da07f189eb4676861c5f2730962b0d8d665/lxml-6.1.1-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:aee395f5d0927f947758b4ec119fd5fc8ec71f07a1c5c52077b30b04c0fa6955", size = 5647338, upload-time = "2026-05-18T19:19:40.553Z" }, + { url = "https://files.pythonhosted.org/packages/42/3d/ef4dcfffd22d27a61805d8ed9f7fb888495bc6aa88648fa07c1eaa5586b6/lxml-6.1.1-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9395002973c827b3ed67db77e6ec09f092919a587022174554096a269378fb13", size = 5239528, upload-time = "2026-05-18T19:19:43.657Z" }, + { url = "https://files.pythonhosted.org/packages/62/bb/37fb3f0dff146bdcfa78eec47879273820b2a0bf350ec236ce14bd0b1c26/lxml-6.1.1-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:73bc2086f141224ebddb7fc5c6a36ca58b31b94b561e1dfe8e073e3270fad1e7", size = 5350730, upload-time = "2026-05-18T19:19:46.307Z" }, + { url = "https://files.pythonhosted.org/packages/90/42/43253f168388df4fae1f38c01df36ddb9bee39e2048167b54cdcbae85ea3/lxml-6.1.1-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:3779def59032b81e44a5f70096ef6bf2082f8d901937dca354474ba09782e245", size = 4697530, upload-time = "2026-05-18T19:19:49.889Z" }, + { url = "https://files.pythonhosted.org/packages/eb/a8/c5a8504f81bbdfc8e7094c2c850cdb4ed6777fc4d5ddd9e5ab819f3b0d54/lxml-6.1.1-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:86c89b9d55ebf820ad7c90bc533410f0d098054f293351f10603c0c46ff598f5", size = 5250670, upload-time = "2026-05-18T19:19:53.199Z" }, + { url = "https://files.pythonhosted.org/packages/77/b7/c7e76ab18744d75e21f320ebf9ff9d1ceae2b54dd431ea5a64caf26c9672/lxml-6.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19607c6bbff2a44cf3fe8250abccd20942d3462473e0a721d01d379ed017e462", size = 5084485, upload-time = "2026-05-18T19:19:08.422Z" }, + { url = "https://files.pythonhosted.org/packages/31/31/b35c53f8ef7b7c31cacd23d3638652fff7bcd1deb6eedb709ab43b685908/lxml-6.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:c6ed5141a5c7507cf3ee76bd363b0d6f801e3321adc35b5d825a23115faa5465", size = 4737635, upload-time = "2026-05-18T19:19:12.321Z" }, + { url = "https://files.pythonhosted.org/packages/d9/06/31f23c813a7fe8e0cb1b175e915b08c9bf4e86d225b210feadbdbe519667/lxml-6.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:62aeb7e85b5d60320b9d77eef2e773994e2c0ce10121b277e0a19804e1654a5a", size = 5670681, upload-time = "2026-05-18T19:19:15.001Z" }, + { url = "https://files.pythonhosted.org/packages/1a/bc/ce619bccc89b1fd9ad8a8e1330ee3f3beff9f2ff95b712d7bbcdd6e22fc3/lxml-6.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b1b963fd8f5caa68e99dfae060d54de1fe9cba899b8718b44a00cdca53c3e590", size = 5238229, upload-time = "2026-05-18T19:19:18.131Z" }, + { url = "https://files.pythonhosted.org/packages/2f/5d/b329acbbedc0b619ebc2be6cf7ee9ed07e80892c88d4dfd612c33805789a/lxml-6.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:63876be28efefa04a1df615b46770e82042cce445cfdce55160522f57b231ccb", size = 5264191, upload-time = "2026-05-18T19:19:21.118Z" }, + { url = "https://files.pythonhosted.org/packages/d6/85/be36fb1425b30db3c3f9df75fe86343ebffb79e6320bd7f588e25bfeac39/lxml-6.1.1-cp314-cp314-win32.whl", hash = "sha256:7f7a92e8583f06b1fd49d01158143b8461cfcd135dcb10ec807270a3051bd603", size = 3657202, upload-time = "2026-05-18T19:17:39.509Z" }, + { url = "https://files.pythonhosted.org/packages/b8/ce/3cf9a827342269f54d405a6202397de63f07c69cbd6ce7d183a3f0cba1e9/lxml-6.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:b2d444f2e66624d68e9c6b211e28a76e22fff5fcabcfff4deac18b529b7d4137", size = 4064497, upload-time = "2026-05-18T19:18:14.662Z" }, + { url = "https://files.pythonhosted.org/packages/d9/3e/1a957bde8f0760039e627f94699f82caa782c9d838d86c3d28245ee67212/lxml-6.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:3fd9728a2735fda14f4e8235830c86b539e9661e849665bf926d3f867943b4bf", size = 3741991, upload-time = "2026-05-19T19:22:59.111Z" }, + { url = "https://files.pythonhosted.org/packages/78/b2/00ed55b3a2efa4658fb795c38d1090ec9b3e8a6c3683d4441fa517f09c3b/lxml-6.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:787b2496d0dbe8cd180984e8d29e3a6f76e7ea34db781cb3bd55e4ba1ef8b4ee", size = 8827545, upload-time = "2026-05-18T19:18:41.193Z" }, + { url = "https://files.pythonhosted.org/packages/c0/73/74573db19baa618d5f266f2407898b087ff6927115b00b71e5fc1b700847/lxml-6.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2c8daa471358dc2d6fcf02165e80ec68f77871a286df95bc5cc3816153b0fd2c", size = 4735736, upload-time = "2026-05-18T19:18:46.761Z" }, + { url = "https://files.pythonhosted.org/packages/16/02/6f7061f4f95f51e545d48e87647c54791d204a4e881be4156e7a26ba5338/lxml-6.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acd7d70b64c0aae0c7922cca83d288a16f5f6da523637697872253415269baef", size = 4970291, upload-time = "2026-05-18T19:19:56.215Z" }, + { url = "https://files.pythonhosted.org/packages/b0/02/55fc057d8283427dea7d6edb102e7a840239c77a64a983d92f62a304c0e9/lxml-6.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4f0dd2f01f9f8a89f565d000e03abcf0a13d692a346c8d22f628d49af098777a", size = 5102822, upload-time = "2026-05-18T19:19:59.223Z" }, + { url = "https://files.pythonhosted.org/packages/e4/48/8e1cf78d89d66850121d9255a2a24414c98f775da93b90cf976956c24b14/lxml-6.1.1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b7e8a14c8634bf6f7a568634cb395305a6d964aeb5b7ee32248094bed3a7e2c", size = 5027923, upload-time = "2026-05-18T19:20:01.549Z" }, + { url = "https://files.pythonhosted.org/packages/ed/00/0632a0647612c8af24d26997b3b961397daa9d5b2581444805933629a4cb/lxml-6.1.1-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86281fbdd6a8162756f8d603f37e3435bfa38043adb79c6dc6a2dfee065e7525", size = 5595843, upload-time = "2026-05-18T19:20:03.93Z" }, + { url = "https://files.pythonhosted.org/packages/bc/86/ab008a7dc360711b66858d61c80a5979a70a09f2aa2b05d9698df80b803d/lxml-6.1.1-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5d7152ec39ca7c402d8fb9bad86140a15b9503bd0c54484e3f1bbe3dd37ceca", size = 5224515, upload-time = "2026-05-18T19:20:06.381Z" }, + { url = "https://files.pythonhosted.org/packages/75/c6/2702ff375e728e34f56d9a45339a9cf7e4427e917f542225242d63a05afa/lxml-6.1.1-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:88d8cb75b9d82858497a5393e3c63cfbf03035225e4b35a49ed7ccb151e4dc0e", size = 5312511, upload-time = "2026-05-18T19:20:09.308Z" }, + { url = "https://files.pythonhosted.org/packages/b7/57/a5807c98f87a86f10ef9ffab35516df7c0f0c4b6d5d33e9f608ab9c04a31/lxml-6.1.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f64ec5397ea6a41fc1b4af0380d79b44a755b5531dcaccd9940fb260dca93038", size = 4639206, upload-time = "2026-05-18T19:20:11.704Z" }, + { url = "https://files.pythonhosted.org/packages/1f/e1/8a0a2c35734812395f4da4eaf33748a7e5705bfb2a58b128da764339d5ec/lxml-6.1.1-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d34bbf07dbc7ca5970671b1512e928991fb5e9d95365636c9b2d8b4f53af405e", size = 5232404, upload-time = "2026-05-18T19:20:14.064Z" }, + { url = "https://files.pythonhosted.org/packages/c2/e2/0e6a4dd5ad84d01d99aa7bae7cfefd4a760a0e0f8176818241de17d9b6c0/lxml-6.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:17e0e18d4ad8adbd0399291bc44845b69d9dd68439a3cdebdf35ff902ec05072", size = 5083769, upload-time = "2026-05-18T19:19:23.758Z" }, + { url = "https://files.pythonhosted.org/packages/a0/7e/161f33d463f6ffc1c7679104b65086dea120080d49dde4d238f015aaee2f/lxml-6.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:3ab541146f1f6968c462d6c2ac495148e8cdba2f8347700b2141b6ec5a75bf52", size = 4758936, upload-time = "2026-05-18T19:19:27.256Z" }, + { url = "https://files.pythonhosted.org/packages/f1/fb/2369825e3f6ca99305bf9f7b7085fda91c8b0922a89e54d900974aa3ef85/lxml-6.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2a0217714657e023ef4293500f65aa20fce6164c8fd6b08fa5bd4a859fb14b9b", size = 5620296, upload-time = "2026-05-18T19:19:29.993Z" }, + { url = "https://files.pythonhosted.org/packages/30/90/d61e383146f74c5ab683947ea14dc7b82778838ab9b95ea73a23b60d0191/lxml-6.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:05a82eb6e1530a64f26225b55cbd178113bd0b5af1c2b625f25e5296742c26d2", size = 5228598, upload-time = "2026-05-18T19:19:33.523Z" }, + { url = "https://files.pythonhosted.org/packages/76/2d/2dafd8149e94b05bb070690efd5bb2680720681e03ff03fc57d2b70a1105/lxml-6.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9e36f163528fc50cbef305f02a5fd66d404edf7049cdaff211dbc2cba5a7013e", size = 5247845, upload-time = "2026-05-18T19:19:36.649Z" }, + { url = "https://files.pythonhosted.org/packages/ce/68/b30e913340c380ddac9580c6e6230991fc37240ec4f64704833e4f3e2769/lxml-6.1.1-cp314-cp314t-win32.whl", hash = "sha256:649dda677cf3bd6ac9ae14007ba0c824ded8ce5808b53fc7431d9140399118c1", size = 3897345, upload-time = "2026-05-18T19:17:33.562Z" }, + { url = "https://files.pythonhosted.org/packages/3c/4e/9eb2af5335545f9fbcd7af57bcf87c6025d31eaa31b14ec184a6c8675328/lxml-6.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:793033d6c5cdf33a573f910d9bea14ef8f5771820411d118da8e1182edb53d5e", size = 4393350, upload-time = "2026-05-18T19:18:10.076Z" }, + { url = "https://files.pythonhosted.org/packages/7f/2c/0f1e93c636720e8a3eb59af2bfda99d98b55891e1c53bc30c2e0e865f01b/lxml-6.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:58bb955caba94e467d2a96da17660d2d704e0675894cba21ab8a775b8621fd1c", size = 3817223, upload-time = "2026-05-19T19:22:56.823Z" }, +] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -2157,6 +2260,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0a/bf/ccff9be562e24207716d04ef9dc931c76aff0c89a7265da43e2104d7fe06/openai-2.38.0-py3-none-any.whl", hash = "sha256:ec6661c57b2dcc47414a767e6e3335c7ed3d19c9696999283a3c82e95c756a3c", size = 1344910, upload-time = "2026-05-21T21:23:39.636Z" }, ] +[[package]] +name = "openpyxl" +version = "3.1.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "et-xmlfile" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3d/f9/88d94a75de065ea32619465d2f77b29a0469500e99012523b91cc4141cd1/openpyxl-3.1.5.tar.gz", hash = "sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050", size = 186464, upload-time = "2024-06-28T14:03:44.161Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" }, +] + [[package]] name = "opentelemetry-api" version = "1.42.1" @@ -2714,6 +2829,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b8/ef/50433d346c56657a70d27f156c7b349ac59a068b01de4eb796e747eecc43/protobuf-7.35.0-py3-none-any.whl", hash = "sha256:c13f325cf242bad135c350629eeb5d54b24228eb472fb3e2e9ebbd4c5dc20ca0", size = 171659, upload-time = "2026-05-19T23:02:27.842Z" }, ] +[[package]] +name = "puremagic" +version = "2.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/24/74/ce5987ab9b8aec4ced06e2723ebb604205c9eb58abdad91453da93166380/puremagic-2.2.0.tar.gz", hash = "sha256:eb4bddf07c177c4b434554b92165b67449f5a51e152b976202d6254498810eef", size = 1189752, upload-time = "2026-04-08T01:39:55.562Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/95/81/314320aeffd88dadeac553ff9eb10f54507ab41deccd0c69f6221d254a0a/puremagic-2.2.0-py3-none-any.whl", hash = "sha256:c4f7ed7307f056c787199acfda839555921be1df13abba61e8e6db0c787ae1d0", size = 71971, upload-time = "2026-04-08T01:39:54.169Z" }, +] + [[package]] name = "py-serializable" version = "2.1.0" @@ -2935,6 +3059,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl", hash = "sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d", size = 122781, upload-time = "2026-01-21T03:57:55.912Z" }, ] +[[package]] +name = "pypdf" +version = "6.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/54/4d/39e48f5294a5a6bb78313839aae820666c2d30daeadb652ed50bd46cafac/pypdf-6.12.1.tar.gz", hash = "sha256:3953a097b9f26d4e0ead5ff95943d9971377557662a91d8872186053cd71d30a", size = 6467595, upload-time = "2026-05-22T10:07:59.979Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/aa/4d17fe9ff165d1341878c570b14f5b7291106d63411adf640942a7e638aa/pypdf-6.12.1-py3-none-any.whl", hash = "sha256:8fa2a2321cf16247ed848bd7c97f193a60c08670d04abed5b0138327e51c43b0", size = 343787, upload-time = "2026-05-22T10:07:57.801Z" }, +] + [[package]] name = "pytest" version = "9.0.3" @@ -3003,6 +3136,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/6f/a05a317a66fee0aad270011461f1a63a453ed12471249f172f7d2e2bc7b4/python_discovery-1.3.1-py3-none-any.whl", hash = "sha256:ed188687ebb3b82c01a17cd5ac62fc94d9f6487a7f1a0f9dfe89753fec91039c", size = 33185, upload-time = "2026-05-12T20:53:34.969Z" }, ] +[[package]] +name = "python-docx" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/f7/eddfe33871520adab45aaa1a71f0402a2252050c14c7e3009446c8f4701c/python_docx-1.2.0.tar.gz", hash = "sha256:7bc9d7b7d8a69c9c02ca09216118c86552704edc23bac179283f2e38f86220ce", size = 5723256, upload-time = "2025-06-16T20:46:27.921Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d0/00/1e03a4989fa5795da308cd774f05b704ace555a70f9bf9d3be057b680bcf/python_docx-1.2.0-py3-none-any.whl", hash = "sha256:3fd478f3250fbbbfd3b94fe1e985955737c145627498896a8a6bf81f4baf66c7", size = 252987, upload-time = "2025-06-16T20:46:22.506Z" }, +] + [[package]] name = "python-dotenv" version = "1.2.2" @@ -3012,6 +3158,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" }, ] +[[package]] +name = "python-pptx" +version = "1.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, + { name = "pillow" }, + { name = "typing-extensions" }, + { name = "xlsxwriter" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/52/a9/0c0db8d37b2b8a645666f7fd8accea4c6224e013c42b1d5c17c93590cd06/python_pptx-1.0.2.tar.gz", hash = "sha256:479a8af0eaf0f0d76b6f00b0887732874ad2e3188230315290cd1f9dd9cc7095", size = 10109297, upload-time = "2024-08-07T17:33:37.772Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d9/4f/00be2196329ebbff56ce564aa94efb0fbc828d00de250b1980de1a34ab49/python_pptx-1.0.2-py3-none-any.whl", hash = "sha256:160838e0b8565a8b1f67947675886e9fea18aa5e795db7ae531606d68e785cba", size = 472788, upload-time = "2024-08-07T17:33:28.192Z" }, +] + [[package]] name = "pywin32" version = "311" @@ -3178,7 +3339,7 @@ provides-extras = ["eval"] [[package]] name = "rag-core" -version = "0.7.0" +version = "0.8.0" source = { editable = "packages/core" } dependencies = [ { name = "opentelemetry-api" }, @@ -3231,6 +3392,46 @@ requires-dist = [ { name = "rag-core", editable = "packages/core" }, ] +[[package]] +name = "rag-parsers" +version = "0.1.0" +source = { editable = "packages/parsers" } +dependencies = [ + { name = "beautifulsoup4" }, + { name = "markdown-it-py" }, + { name = "openpyxl" }, + { name = "puremagic" }, + { name = "pypdf" }, + { name = "python-docx" }, + { name = "python-pptx" }, + { name = "pyyaml" }, + { name = "rag-core" }, +] + +[package.optional-dependencies] +dev = [ + { name = "mypy" }, + { name = "pytest" }, + { name = "pytest-asyncio" }, +] + +[package.metadata] +requires-dist = [ + { name = "beautifulsoup4", specifier = ">=4.12" }, + { name = "markdown-it-py", specifier = ">=3.0" }, + { name = "mypy", marker = "extra == 'dev'", specifier = ">=2.1" }, + { name = "openpyxl", specifier = ">=3.1" }, + { name = "puremagic", specifier = ">=1.27" }, + { name = "pypdf", specifier = ">=5.0" }, + { name = "pytest", marker = "extra == 'dev'", specifier = ">=9.0" }, + { name = "pytest-asyncio", marker = "extra == 'dev'", specifier = ">=1.3" }, + { name = "python-docx", specifier = ">=1.1" }, + { name = "python-pptx", specifier = ">=1.0" }, + { name = "pyyaml", specifier = ">=6.0" }, + { name = "rag-core", editable = "packages/core" }, +] +provides-extras = ["dev"] + [[package]] name = "rag-policy" version = "0.1.0" @@ -3256,6 +3457,7 @@ dependencies = [ { name = "rag-config" }, { name = "rag-core" }, { name = "rag-observability" }, + { name = "rag-parsers" }, { name = "typer" }, ] @@ -3264,6 +3466,7 @@ requires-dist = [ { name = "rag-config", editable = "packages/config" }, { name = "rag-core", editable = "packages/core" }, { name = "rag-observability", editable = "packages/observability" }, + { name = "rag-parsers", editable = "packages/parsers" }, { name = "typer", specifier = ">=0.12" }, ] @@ -3602,6 +3805,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" }, ] +[[package]] +name = "soupsieve" +version = "2.8.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/47/2c/0a5f6f8ee0d5589e48c7640213ed5175d52cf540a06725b628cc1a45d6ce/soupsieve-2.8.4.tar.gz", hash = "sha256:e121fd02e975c695e4e9e8774a5ee35d74714b59307868dcc5319ad2d9e3328e", size = 121110, upload-time = "2026-05-24T13:55:57.154Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/f5/0c41cb68dcae6b7de4fac4188a3a9589e21fb31df21ea3a2e888db95e6c9/soupsieve-2.8.4-py3-none-any.whl", hash = "sha256:e7e6b0769c8f51ed59acab6e994b00621096cfb1c640a7509295987388fbaf65", size = 37304, upload-time = "2026-05-24T13:55:55.406Z" }, +] + [[package]] name = "sqlalchemy" version = "2.0.49" @@ -4234,6 +4446,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1f/f6/a933bd70f98e9cf3e08167fc5cd7aaaca49147e48411c0bd5ae701bb2194/wrapt-1.17.3-py3-none-any.whl", hash = "sha256:7171ae35d2c33d326ac19dd8facb1e82e5fd04ef8c6c0e394d7af55a55051c22", size = 23591, upload-time = "2025-08-12T05:53:20.674Z" }, ] +[[package]] +name = "xlsxwriter" +version = "3.2.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e55b802a8b231770d286a9758b3d936bd1e04ce5ba88/xlsxwriter-3.2.9.tar.gz", hash = "sha256:254b1c37a368c444eac6e2f867405cc9e461b0ed97a3233b2ac1e574efb4140c", size = 215940, upload-time = "2025-09-16T00:16:21.63Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" }, +] + [[package]] name = "xxhash" version = "3.7.0"