Our life is frittered away by detail… Simplicity, simplicity, simplicity! I say, let our affairs be as two or three, and not a hundred or a thousand… Simplify, simplify!
— Henry David Thoreau, Walden (1854)
Thoreau went to the woods to find out what was essential by removing everything that was not. That is the whole programme here, applied to a document: find the detail that has been added to your prose without your asking — the zero-width character carrying a payload, the sentence that runs to thirty words, the "delve" that arrived with a model — and take it back out.
The tool is two instruments sharing one workbench:
- a detector, which reports what is provably in a file and refuses to guess beyond it, and
- an editor in the Hemingway tradition, which measures prose and says plainly where it is hard to read.
They stay separate on purpose. Hemingway will tell you a sentence is dense; it will not tell you who wrote it. A CRC will tell you a frame is present; it will not tell you the prose is bad. Conflating the two produces a tool that sounds confident about everything and is trustworthy about nothing, so a style observation here never becomes a verdict and a verdict never becomes writing advice.
Everything is deterministic and local. The same input always gives the same
output, no detector calls an LLM or a remote classifier, and network access
happens only in the explicit scan-url command. Text and provenance tooling
uses the standard library; image watermarking adds NumPy and Pillow.
The economy Thoreau meant is also the engineering constraint. Two or three affairs, not a hundred: a fixed table, a closed-form formula, a checked-in corpus you can re-run. Nothing here is fitted at runtime, and every number it prints can be recomputed by hand from something in this repository.
Documentation lives in docs/. Methodology, measurements, and
limitations are in REPORT.md.
uv run thoreau generate --out corpus
uv run thoreau scan corpus
uv run thoreau scan ~/Documents/neo --git-visible --json
uv run thoreau scan-url https://example.com --max-pages 100 --json
# Or provide a nonstandard, same-origin sitemap with --sitemap.
uv run thoreau image-embed input.png marked.png --key 'owner-secret'
uv run thoreau image-detect marked.png --key 'owner-secret'
uv run thoreau image-evaluate corpus/images/manifest.jsonl
uv run thoreau evaluate corpus/manifest.jsonl
uv run thoreau stress corpus/manifest.jsonl
uv run python -m unittest discover -s tests -vStatistical detectors are deliberately off by default: a test is meaningful only when its key, tokenization, and scheme are known. Enable profiles explicitly:
uv run thoreau scan document.txt --profile kgw_context
uv run thoreau scan document.txt --profile keyed_token_bias \
--profile-key keyed_token_bias=my-key
# --key my-key is a compatibility shorthand that also enables keyed_token_bias.Multiple enabled statistical tests use a Bonferroni family-wise adjustment.
--alpha is the bound for the entire selected family, not for each test.
The result model avoids numerical “confidence” values that are not calibrated probabilities.
| Verdict | Meaning |
|---|---|
detected |
a self-checking scheme frame, a configured hypothesis test, or a recognized provenance container matched |
suspicious |
a contextual anomaly or explicit metadata marker deserves review but is not proof |
informational |
legitimate or generic structure is reported only with --include-informational |
Evidence types identify the basis: valid_frame, scheme_match,
statistical_test, provenance_structure, metadata_marker, and
unicode_anomaly. A valid CRC detects this toolkit's frame; it does not
authenticate an author. A C2PA container hit does not claim that its signature
was validated.
| Detector | Carrier | Signal |
|---|---|---|
homoglyph |
prose | Latin/Cyrillic lookalikes encode a CRC-framed payload |
zero_width |
text | U+200B/U+200C encode framed bits |
variation_selector |
text | supplementary selectors encode framed nibbles |
trailing_whitespace |
line-oriented text | final space/tab encodes framed bits |
markdown_markers |
Markdown | equivalent -/* list markers encode framed bits |
python_quote_style |
Python | equivalent quote styles encode framed bits |
acrostic |
multiline prose | initial characters contain a Base32 envelope |
Exact bit-channel detection searches all offsets and accepts only a tag, length, UTF-8 payload, and CRC-valid frame. This is why layered marks can still be decoded.
| Profile | Model-free demonstration |
|---|---|
keyed_token_bias |
contextual candidate words partitioned with SHA-256 |
kgw_context |
previous-token keyed green list with unique-bigram scoring |
unigram_watermark |
key-fixed vocabulary partition and unique-token scoring |
synthid_toy |
keyed n-gram g-values and a weighted-mean test |
These profiles are deterministic teaching and evaluation adapters. They are
not compatible with vendor production keys or tokenizers, and
synthid_toy is not Google's SynthID detector.
Contextual Unicode analysis reports default-ignorables, bidi controls, invalid joiner use, and mixed-script confusable tokens with exact locations. It preserves recognized emoji ZWJ/variation sequences, well-formed emoji tag flags, and joiners between characters of an eligible joining script. Whole Cyrillic words are not flagged merely for being Cyrillic.
Structured provenance routing supports PNG, JPEG, WebP, SVG, PDF, DOCX, ODT,
XLSX, PPTX, HTML, and Markdown. It parses defined metadata regions and
container structures instead of searching arbitrary file bytes. Examples: PNG
caBX requires a valid PNG chunk CRC; JPEG APP11 requires the C2PA JUMBF UUID;
WebP reads the RIFF chunk table rather than the VP8 pixel payload; XLSX and
PPTX read only docProps/ and customXml/, never cell or slide text; a
generic frontmatter model: or CMS generator does not count as AI metadata.
AVIF, HEIC, and base64 data: URIs embedded in HTML, Markdown, or SVG are not
yet parsed. See REPORT.md for the current gap list.
Directory scans skip hidden workspaces and common dependency/build directories,
cap each file before reading, distinguish non-UTF-8 binaries, hash scanned
bytes, and return an aggregate summary. --git-visible instead selects exactly
tracked plus non-ignored untracked files. Website scans are same-origin,
sitemap-bounded, size-bounded, and deterministic in URL order.
annular_angular_fourier_v1 is a known-key presence mark for arbitrary raster
images. It operates on luminance Fourier coefficients rather than compositing a
visible RGB layer. A SHA-256-derived balanced angular code is distributed over
12 log-spaced low/mid-frequency annuli. A smooth activity mask spends more of
the distortion budget in texture and edges.
The detector averages log magnitude into a log-polar grid and removes each
annulus's angular mean. For an approximately isotropic filter with transfer
function H(r), log|H(r)| is constant around the annulus and therefore
cancels. A bounded alignment search covers angular and log-radius shifts. The
decision uses the claimed key's maximum normalized correlation against 255
deterministically derived decoy keys:
p = (1 + number of decoy scores >= claimed-key score) / 256
Embedding is closed-loop rather than cover-blind: 0.12 controls the dose
increment, while the embedder measures its own claimed-key correlation after
each dose. It stops at the default 0.40 target, a 36 dB luminance PSNR floor, or
12 iterations, whichever comes first. The report says whether the target was
actually reached. Raising the target or lowering the PSNR floor can improve
survival at the cost of visible distortion. This first version intentionally
uses the key itself as the identity and carries no payload, spending its
available signal energy on robustness. Images must be at least 272 pixels on
each axis. Partly transparent images require an explicit rendering background,
for example --background '#fff'; the flattened output is what is marked.
Detection is blind—it does not need the original image—but the key must have
been selected independently of the image for the decoy-key test to have its
stated interpretation.
thoreau style is a deterministic writing audit in the Hemingway mould, and
thoreau clean removes the text marks thoreau can detect.
uv run thoreau style draft.md
uv run thoreau style draft.md --json
uv run thoreau style draft.md --fix -o edited.md # mechanical edits only
uv run thoreau clean draft.md --in-place # strip detectable marks
uv run thoreau clean draft.md --style # marks plus mechanical edits
uv run thoreau style-generate --out corpus/style
uv run thoreau style-evaluateNeither command writes over anything by default. Output goes to stdout unless
-o names a new path or --in-place is explicit, and -o refuses an existing
file.
| Measure | Basis |
|---|---|
| grade level | Flesch-Kincaid, per document and per sentence |
| hard / very hard sentences | grade 12 and grade 14, Hemingway's thresholds |
| passive voice, adverbs, hedges | closed word lists and a be-verb/participle rule |
| wordy phrases | a fixed substitution table |
| LLM register | the style-annotated excess vocabulary of Kobak et al. (2025) |
| marker over-representation | document marker rate ÷ pre-LLM reference rate |
| cross-entropy | unigram surprisal in bits against the same reference |
Frontmatter, table rows, block HTML, fenced code, inline code, and link targets are excluded from every rule. List items and paragraphs are their own units, and per-sentence grades are only reported for sentences of at least 8 words, because Flesch-Kincaid puts a four-word bullet of long nouns above grade 14.
uv run thoreau style draft.md --fix --target-grade 9
uv run thoreau style draft.md --fix --target-rarity 12--target-grade lowers grade by splitting sentences at coordinating
conjunctions and swapping in shorter words, and raises it by joining short
sentences. Raising grade never substitutes vocabulary. --target-rarity lowers
mean surprisal by substituting commoner words.
Raising rarity is deliberately not offered. It would need the substitution table run backwards, which is not meaning-preserving — "facilitate" → "help" is always safe, but "help" → "facilitate" breaks "help begin the work" — and the words it would inject are the same LLM register markers the audit flags. The command says so rather than doing it badly.
Both dials report the achieved value and whether the target was reached; an unreachable target is reported, never faked. Writing to hit a readability number is Goodhart's law with a word processor: these are for adjusting a draft you will then read, not for producing final copy.
A style observation is not provenance evidence, so the audit deliberately does
not use the Finding/Verdict model and never emits a verdict. Over-use of
"delve" is a fact about register; it establishes nothing about origin.
The over-representation ratio is relative to academic biomedical English (PubMed abstracts, 2010–2021, the pre-ChatGPT years of the corpus behind the excess-vocabulary study). A ratio of 1.0 means "matches that reference", not "matches human writing". thoreau's own README scores 0.25 against it. Supply a different baseline if your domain differs.
--fix applies only mechanical, meaning-preserving edits: phrase and word
substitutions from a fixed table, and, with --marks, mark removal. Anything
needing judgement—a hard sentence, a passive construction, an over-budget
adverb, a structural tic—is reported and left alone. The tool does not rewrite
your sentences.
clean removes zero-width characters, stray variation selectors, bidi
controls, tag characters, mixed-script confusables, and trailing whitespace,
routing through the same classifier as detection so it removes exactly what
thoreau reports. Legitimate emoji ZWJ sequences, flag tags, and same-script
joiners survive.
It does not touch Markdown list markers, Python quote style, or acrostics. Those channels are indistinguishable from an author's own formatting, so rewriting them would edit meaningful choices to chase a mark; the corpus records that limit rather than hiding it. Nothing here removes a statistical token-choice watermark, which lives in which words were chosen and survives every edit above.
The checked-in text/code corpus contains 12 marked samples and 9 clean controls
across plain text, Markdown, CSV, HTML, JSON, and Python. corpus/modern/
contains model-free adapters informed by public KGW, Unigram, and SynthID
research. corpus/manifest.jsonl is its ground truth.
corpus/images/ adds 3 clean controls and 6 marked images: the original mark,
JPEG quality 70 and 40, Gaussian blur, aggressive unsharp mask, and a 75%
resize round trip. Its manifest is corpus/images/manifest.jsonl.
corpus/style/ adds 13 samples: 4 written in a heavy LLM register, 4 plain, and
5 plain samples carrying a real text mark. thoreau style-evaluate checks two
things — that the over-representation ratio separates the registers, and that
cleaning a marked sample actually defeats detection afterwards. The second is
the claim worth testing: a remover that leaves a decodable frame behind is
worse than none, because it invites the belief that the text is clean.
The current fixture results are TP=12/TN=9 for text/code, TP=6/TN=3 for images, and TP=4/TN=9 with 0 residual marks for style, with no fixture errors. This validates implementation behavior on known fixtures; it is not a population accuracy estimate. Cross-entropy in particular does not separate the style fixtures and is reported as an observation, not a discriminator. See REPORT.md for the methodology, compression sweep, comparison review, repository audit, and limitations.