Skip to content

architecture.md

Latest commit

 

History

History
147 lines (117 loc) · 6.26 KB

architecture.md

File metadata and controls

147 lines (117 loc) · 6.26 KB
Raw

Architecture

Product boundary

TreeSift selects entries below one explicit filesystem root and either serializes the selection as ZIP/TAR/TAR.GZ or prepares it as an AI context handoff. Selection is independent from every consumer so archive writers, context planning, listing, and verification share the same ordered filesystem decisions.

The project intentionally avoids a plugin registry or provider framework. Two concrete backends are sufficient; another abstraction is justified only by a new, demonstrated variation point.

Data flow

filesystem root
    ↓
HierarchicalGitIgnore + explicit policies
    ↓
select_tree → SelectionPlan
    ├── list / explain / JSON
    ├── ZIP writer
    ├── TAR writer
    ├── portability audit
    └── Git export-ignore filter
            ├── archive inspection → manifest / verification
            └── AI context planning
                    ↓
               ContextPlan
                    ├── analysis / token tree
                    ├── Markdown / JSON
                    ├── split parts
                    └── snapshot manifest / delta

Responsibilities

  • treesift.ignore: scoped ignore discovery, matching, precedence, and RuleDecision explanations.
  • treesift.selection: traversal, classification, observed metadata, SelectionPlan, symlink policy, and portability audit.
  • treesift.archive: ZIP serialization and the stable create_zip API.
  • treesift.tar_archive: TAR/TAR.GZ serialization and create_tar.
  • treesift.manifest: archive inspection, hashes, schema objects, and atomic sidecar writes.
  • treesift.verification: structural and optional manifest comparison.
  • treesift.config: explicit [tool.treesift] TOML loading.
  • treesift.git_attributes: optional Git export-ignore filtering.
  • treesift.cli: argument parsing, configuration precedence, rendering, and stable exit codes.
  • treesift._operation: shared low-level atomicity, metadata normalization, and strict-consistency helpers.
  • treesift.context: AI-safe filtering, UTF-8/binary classification, measurements, Git provenance, deltas, token estimates, and ContextPlan.
  • treesift.context_render: deterministic Markdown/JSON rendering, splitting, context output hashing, and atomic context-manifest writes.
  • treesift.sensitivity: intentionally small high-confidence sensitive-content detector set.
  • treesift.targets: static, dated provider upload-limit profiles.

Selection model

SelectedEntry carries the source path, archive path, entry kind, size, mode, mtime, device, inode, ownership metadata, and optional symlink target observed during traversal. Writers do not rediscover ignore rules and use the same member ordering.

SelectionPlan is an observed view, not an immutable filesystem snapshot. Strict mode compares identity and metadata before and after serialization to detect common races.

Ignore evaluation

For each candidate, rules are evaluated from <source>/.gitignore through each ancestor directory. Every pattern receives a path relative to its containing directory. The last matching decision wins. Explicit CLI/config exclusion rules run last.

RuleDecision preserves the winning source, line, pattern, and complete match history for --explain.

Automatically discovered symlinked .gitignore files are treated as data and are not followed.

Trust boundaries

The selected source is the discovery boundary. Parent ignores, global Git configuration, and .git/info/exclude are not loaded. An explicit config, custom root ignore, or exclusion file is read only when requested.

Symlink targets are untrusted and never traversed. safe policy rejects absolute and source-escaping targets; reject disallows all links.

Special files are rejected before opening. Archive verification rejects unsafe member names, duplicates, count-limit violations, and uncompressed-size limits without extracting data.

Atomicity

Each archive and manifest is written to a temporary sibling and committed with os.replace. Existing destinations require overwrite authorization. Temporary files and empty output directories created by failed operations are cleaned up.

An archive and its sidecar manifest are two individually atomic outputs. The pair is not a filesystem transaction; consumers should publish both only after both operations succeed.

Reproducibility

Normal mode preserves observed timestamps and metadata. Reproducible mode uses an explicit timestamp, SOURCE_DATE_EPOCH, or a fixed fallback; uses UTC ZIP time; normalizes file modes; and normalizes TAR ownership fields.

The guarantee is byte identity for the same selected bytes, configuration, TreeSift version, and compression implementation. Cross-version compressor identity is not promised without evidence.

Extension rules

  • Keep rule evaluation independent from writers.
  • Add explicit sibling APIs rather than dynamic backend discovery.
  • Preserve ordering, trust boundaries, and atomic failure behavior.
  • Add dependencies only when the standard library cannot satisfy an actual requirement.
  • Optimize only against the benchmark harness and preserve behavior tests.

AI context model

Context planning is a downstream consumer of SelectionPlan; it never reimplements ignore precedence. Context mode composes a non-overridable AI-safety decision layer with the existing matcher so dependency, cache, VCS, environment, and generated-output paths can be pruned during traversal. Optional Git export-ignore filtering follows selection, and positive --include patterns narrow the remaining regular files. Text classification, sensitive-content checks, measurements, and SHA-256 are then recorded in a full snapshot. changed-only and delta modes decide which snapshot entries are rendered without discarding the full snapshot state used by future deltas.

Context rendering re-reads and hashes each file immediately before serialization. If the digest differs from the planned snapshot, generation aborts instead of producing a manifest that describes different bytes. Split outputs never split an individual source file.

Provider profiles are static data, not adapters or plugins. The project still avoids a dynamic provider framework because TreeSift does not authenticate with or upload to AI providers.