TreeSift

TreeSift prepares filesystem trees for AI chats and coding agents, and can also package the same deterministic selection as ZIP, TAR, or TAR.GZ. It combines scoped ignore rules, explainable selection, AI-safe context policies, Git provenance, integrity manifests, reproducible output, and explicit safety boundaries.

The primary AI workflow is treesift context: turn a project into a structured, auditable Markdown or JSON handoff without silently deciding which code is "important". Archive workflows remain fully supported.

TreeSift guarantees deterministic selection and member ordering. Byte-for-byte archive reproducibility requires --reproducible and the same archive/compression implementation. AI context rendering is deterministic for the same selected bytes, options, TreeSift version, and Git state.

AI context quick start

# Create project-context.md
treesift context ./project

# Inspect size and provider-neutral token estimates without writing output
treesift context ./project --analyze
treesift context ./project --token-tree

# Create an auditable snapshot manifest for later deltas
treesift context ./project ./project-context.md --manifest

# Send only files changed since the previous snapshot
treesift context ./project ./update.md \
  --delta-from ./project-context.context.json

# Split a large handoff into upload-friendly parts
treesift context ./project ./context-parts \
  --split-tokens 100000 \
  --target chatgpt

AI context mode defaults to conservative exclusions for .git, dependency trees, virtual environments, caches, environment files, key/credential filenames, and root build output. Binary files are reported and skipped. Supported high-confidence secret-like patterns abort generation by default; --secrets warn reports and continues, while --secrets off disables those detectors. These checks are not an exhaustive secret scanner.

Context bundles include a project tree, per-file hashes and measurements, Git commit/branch/dirty provenance when available, and an explicit trust-boundary notice separating repository data from project instruction files such as AGENTS.md, CLAUDE.md, and GEMINI.md.

See docs/ai-context.md.

Core behavior

• Every regular .gitignore inside the selected source is evaluated in its own directory scope.
• Parent, global, and .git/info/exclude rules are intentionally not loaded.
• Deeper rules override matching parent rules inside their subtree.
• --gitignore FILE replaces only the source-root rules file.
• --exclude and --exclude-from add final root-scoped rules.
• Symlink targets are not traversed.
• FIFOs, sockets, devices, and unknown special entries are rejected.
• Archive replacement is atomic through a temporary file and os.replace.

The selected source directory is the automatic discovery and trust boundary.

Features

• ZIP, TAR, and TAR.GZ output.
• Reproducible timestamps and normalized permissions.
• SOURCE_DATE_EPOCH support.
• --list, --explain, and versioned JSON output.
• Sidecar manifests with SHA-256 hashes per entry and for the archive.
• Structural and manifest verification without extraction.
• Best-effort or strict source-consistency checking.
• store, safe, and reject symlink policies.
• Cross-platform path portability audit.
• Optional Git export-ignore integration.
• Explicit TOML configuration in [tool.treesift].
• Deterministic standard-library benchmark harness.

Requirements

• Python 3.10 or newer.
• pathspec 0.12 or newer, below 2.0.
• Git only when --respect-export-ignore is enabled.

Installation

TreeSift is a command-line application. Install it in an isolated environment rather than modifying a Python managed by Homebrew or the operating system.

macOS with Homebrew

brew install pipx
pipx ensurepath
pipx install .

Restart the terminal after pipx ensurepath if the command is not immediately available. A downloaded wheel can be installed with:

pipx install ./treesift-3.4.0-py3-none-any.whl

Do not use brew install .; this repository is not a Homebrew formula. Avoid --break-system-packages, which bypasses the protection of externally managed Python installations.

Development

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

On Windows PowerShell:

py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Quick start

# Create dist/project.zip
treesift ./project ./dist

# Create a reproducible TAR.GZ and adjacent integrity manifest
SOURCE_DATE_EPOCH=1700000000 treesift ./project ./dist \
  --format tar.gz \
  --reproducible \
  --manifest

# Verify the archive and its manifest without extracting it
treesift verify ./dist/project.tar.gz \
  --manifest ./dist/project.tar.gz.manifest.json

Package-module execution is also supported:

python -m treesift ./project ./dist

Selection inspection

# Stable human-readable listing; no archive is written
treesift ./project --list

# Machine-readable listing
treesift ./project --list --json

# Show the winning rule and all matching rules for one path
treesift ./project --explain build/output.log

treesift ./project --explain build/output.log --json

Archive formats and compression

treesift ./project ./dist --format zip --compression deflate
treesift ./project ./dist --format zip --compression store
treesift ./project ./dist --format zip --compression bzip2
treesift ./project ./dist --format zip --compression lzma
treesift ./project ./dist --format tar
treesift ./project ./dist --format tar.gz

ZIP uses DEFLATE at level 6 by default. TAR.GZ uses gzip at level 6. TAR is uncompressed.

Reproducible output

SOURCE_DATE_EPOCH=1700000000 treesift ./project ./dist --reproducible

treesift ./project ./dist --reproducible --timestamp 1700000000

The explicit timestamp overrides SOURCE_DATE_EPOCH. When neither is present, TreeSift uses a documented fixed epoch. Reproducible mode normalizes timestamps, ownership metadata where applicable, and permission modes while preserving executable-file semantics.

Safety policies

# Current compatibility behavior: store links without following them
treesift ./project ./dist --symlinks store

# Reject absolute or source-escaping link targets
treesift ./project ./dist --symlinks safe

# Reject every symlink
treesift ./project ./dist --symlinks reject

# Fail on case, Unicode, Windows-reserved, or trailing-character conflicts
treesift ./project ./dist --portable

# Detect selected files changed between traversal and serialization
treesift ./project ./dist --consistency strict

Strict consistency detects common source races but does not provide a transactional filesystem snapshot.

Git and additional rules

# Repeated final rules
treesift ./project ./dist --exclude "*.local" --exclude "build/"

# Rules loaded in file order
treesift ./project ./dist --exclude-from ./release.ignore

# Apply tracked export-ignore attributes after TreeSift selection
treesift ./project ./dist --respect-export-ignore

export-ignore requires the source to be a Git worktree and Git to be available. TreeSift does not implicitly exclude .git; use rules appropriate for the intended package.

Configuration

A source-local pyproject.toml can contain:

[tool.treesift]
format = "tar.gz"
reproducible = true
timestamp = 1700000000
manifest = true
symlinks = "safe"
portable = true
exclude = ["*.local", "build/"]

TreeSift checks only <source>/pyproject.toml; it does not search parent directories. Use --config FILE for another explicit file and --show-config to inspect the effective configuration.

See docs/configuration.md.

Python API

from pathlib import Path

from treesift import create_manifest, create_tar, create_zip, select_tree
from treesift.ignore import HierarchicalGitIgnore

source = Path("project")
plan = select_tree(source, HierarchicalGitIgnore(source))

zip_stats = create_zip(
    source,
    Path("dist/project.zip"),
    reproducible=True,
    timestamp=1_700_000_000,
    consistency="strict",
    symlink_policy="safe",
)

tar_stats = create_tar(
    source,
    Path("dist/project.tar.gz"),
    compression="gz",
)

manifest = create_manifest(Path("dist/project.zip"))

The canonical public API is declared through treesift.__all__.

Validation

python -m pytest
python -m pytest --cov=treesift --cov-branch --cov-report=term-missing
python -m compileall -q src tests benchmarks
ruff check .
python -m build

CI covers Python 3.10 through 3.14 on Linux and boundary versions on Windows and macOS. See docs/testing.md.

Documentation

• docs/ai-context.md: AI handoff model, policies, budgets, targets, and deltas.
• docs/architecture.md: component and trust boundaries.
• docs/configuration.md: configuration schema and precedence.
• docs/manifest-schema.md: manifest contract.
• docs/security.md: threat model and failure behavior.
• docs/performance.md: benchmark method and optimization policy.
• docs/testing.md: test strategy and CI matrix.

Known limitations

• Token counts in AI context mode are deterministic estimates, not provider-exact tokenization.
• Sensitive-content detection is conservative and non-exhaustive.
• AI provider upload profiles are dated snapshots and may become stale as provider products change.
• Split context parts are individually atomic, not one cross-file transaction.
• Strict consistency detects changes; it does not create a transactional snapshot.
• Reproducible output is bounded by the same TreeSift and compression implementation.
• Archive and sidecar manifest replacement are individually atomic, not one atomic pair.
• Symlink and POSIX metadata restoration depends on the extraction tool.
• Parent/global Git ignores and .git/info/exclude remain intentionally outside scope.
• Verification does not extract archives and does not prove that an external extractor is safe.
• No license is granted unless the repository owner adds one.