Primitive proposal: Sensitive Data Redaction

[jcpoyser]

Primitive Name

Sensitive Data Redaction

What concept does this teach?

A deterministic pre-ingest redaction pass that masks or skips sensitive strings — API keys, bearer tokens, database connection strings with embedded credentials, JWTs, SSNs, private key blocks, and similar high-risk identifiers — before external text is embedded or stored in Open Brain. The primitive teaches contributors how to keep the useful semantic meaning of imported content while ensuring exact secrets are never stored raw in the knowledge graph.

Which extensions would use this?

This is where I want to be transparent about where things stand, so maintainers can rule on fit before I open a PR.

Already wired in (both are recipes, not extensions):

• recipes/email-history-import — the Gmail importer already runs this redaction pass before embedding and database insert. The README and pull-gmail.ts reference the primitive and read patterns.json from it at runtime. The --no-redact flag is the opt-out. (Note: requires_primitives is documented in the README and implemented in code but not yet declared in metadata.json for this recipe — that would be part of the PR.)
• recipes/obsidian-vault-import — the Obsidian importer has the same integration: pre-ingest redaction pass, --no-redact opt-out, redaction counts recorded in thought metadata. requires_primitives: ["sensitive-data-redaction"] is already declared in its metadata.json.

The honest curation question: CONTRIBUTING.md states a primitive must be referenced by at least 2 extensions. Both contributions above are recipes, not extensions. I'm not claiming they meet that bar — I'm flagging it and asking. If the two-recipe integration doesn't satisfy the requirement as written, I'd like the maintainers' guidance on the right path: primitive as-is, rolled into a recipe, deferred until an extension takes a dependency on it, or something else. I don't want to submit a PR that the curation policy doesn't support.

Forward-looking: Any ingestion recipe or extension that imports raw third-party text (chat exports, document imports, social capture, etc.) would be a natural candidate to use this primitive. The pattern is broad, but today the only wired integrations are the two recipes above.

Content outline

The draft at primitives/sensitive-data-redaction/ already covers:

• What it is and why it matters — baseline safety layer for solo-operator ingestion stacks; policy rationale (keep semantic context, remove exact secrets)
• Scope — what must require the primitive (importers of raw external text) and what is explicitly out of scope (dashboards, schema-only contributions, metadata backfills)
• How it works — patterns.json with deterministic regex rules and two actions: redact (replace with a placeholder) and skip (reject the content entirely)
• Pattern coverage — OpenAI/OpenRouter keys, Stripe secret keys, Google API keys, JWTs, GitHub tokens, Slack tokens, AWS access keys, Supabase secret keys, Bearer tokens, database connection strings with credentials, generic secret assignments, URL token parameters, US SSNs, private key blocks (skip action)
• Step-by-step guide — how a consuming recipe declares the dependency, runs the pass, handles skip results, and logs redaction counts
• Troubleshooting — false positives, missing policy file, overly aggressive removal
• Extensions That Use This — listing the two wired integrations
• Limitations — exercised only against @jcpoyser's own deployment; not tested against a broader set of user vaults or email corpora

Do you have a draft?

Yes — I can submit a PR with the full guide

---

Additional context from @jcpoyser:

The implementation lives on the local branch contrib/jpoyser/sensitive-data-redaction (commit edb7fd0) and includes:

• primitives/sensitive-data-redaction/README.md — the concept guide
• primitives/sensitive-data-redaction/metadata.json — structured metadata
• primitives/sensitive-data-redaction/patterns.json — the canonical regex rule set (14 rules covering secrets, credentials, and PII)
• Wiring in recipes/email-history-import/ (pull-gmail.ts, README) applying the redaction pass with --no-redact opt-out
• Wiring in recipes/obsidian-vault-import/ (import-obsidian.py, README, metadata.json) with the same pattern

I have not attached or pasted the full implementation here. If maintainers want to see it, I'm happy to open a PR.

Limitation: This has only been exercised against my own deployment. I haven't run it against a wider sample of user vaults or email archives, so edge cases in the regex rules are possible.

Question for maintainers: Does the two-recipe integration satisfy the curation bar, or should this live in a recipe, wait for an extension dependency, or take another form? Happy to follow whatever direction you give before submitting any code.