feat: hierarchical chunk, file, and module retrieval

[Lucas-Bur]

Parent

#83

What to build

Add hierarchical retrieval levels for chunk, file, and module context. Each level has an explicit entry level and stable identity. Queries search all enabled levels as separate retrieval channels and fuse their ranked results without confusing a module match with a concrete source chunk.

Index v2 should reserve entry-level, parent identity, content hash, and source membership metadata needed by this slice.

Acceptance criteria

• Index entries distinguish chunk, file, and module levels
• File entries summarize the searchable content of one file
• Module entries summarize a documented module boundary
• Each level is ranked as a separate channel and fused through RRF
• Output clearly identifies result level and source membership
• Level-specific embeddings participate in cache invalidation
• Configuration can disable file or module levels
• Tests cover exact chunk queries and broader architectural queries

Blocked by

• feat: incremental indexing via mtime cache #84
• feat: embedding cache for incremental reindex #95
• feat: lazy text loading at query time #96

[Lucas-Bur]

Hierarchical levels need their own token and memory policy; see #164.

A file or module entry can be much larger than the active model window. It must not be passed as one raw embedding request and must not rely on Transformers.js truncation. For every level, define whether the representation is recursively chunked/aggregated, summarized, or deliberately omitted, and measure it with the active model tokenizer before inference.

The hard/runtime limits currently differ by model (MiniLM/BGE: 512 tokens; Jina code: 8192), while operational caps may need to be lower for RAM safety. Level-specific embeddings and cache keys should record the effective model/limit contract. Add tests proving that file/module embeddings never silently lose suffix content and that long batches remain memory-bounded.