DLP Insight

A self-hosted data loss prevention (DLP) analysis platform. Upload a file, paste text, or point it at a URL — DLP Insight extracts the content, scans it for sensitive information (national IDs, phone numbers, payment cards, credentials, health data and more), profiles the upload source via GeoIP, and generates a compliance-oriented risk report for the selected jurisdiction.

简体中文文档

Features

• Broad format coverage — plain text, CSV/JSON/XML, PDF, DOC/DOCX, XLS/XLSX, PPTX, images (OCR), Jupyter notebooks, SQLite databases, subtitles, media metadata, source code, key stores and archives, each handled by a dedicated parser with crash/OOM sandboxing for risky formats.
• Multi-jurisdiction detection rules — national IDs (CN resident ID with checksum, HKID, US SSN/ITIN/EIN, UK NINO, SG NRIC, and more), phone numbers, email, bank cards (Luhn-validated, BIN-aware), IBAN/SWIFT, VIN and license plates, GPS coordinates, device identifiers (IMEI/MAC), and crypto wallets.
• Credential & secret scanning — AWS/GCP keys, GitHub/Slack/SendGrid/Anthropic tokens, generic API keys and passwords, backed by detect-secrets.
• Combo detection with suppression — related hits (e.g. name + ID + phone) escalate into higher-risk combos; tokenized/masked/encrypted values suppress false positives.
• Risk scoring & compliance summary — findings are scored and mapped to the selected region's regulations (PIPL, GDPR, HIPAA/GLBA, PDPO, PDPA, UU PDP, …).
• Chinese name recognition — lightweight jieba-based NER for person names in CJK text.
• Fast matching — optional Hyperscan prefilter with automatic fallback to pure Python regex when Hyperscan is unavailable.
• GeoIP source profiling — offline GeoLite2-based lookup of the uploader's network origin.
• Synthetic sample library — built-in, downloadable test files for 11 regions plus rule-domain scenarios; all data is machine-generated, containing no real personal information.
• Zero-config storage — SQLite out of the box, PostgreSQL via a single DATABASE_URL environment variable.

Architecture

src/web   Next.js 16 + React 19 frontend (pure CSS, no UI framework)
src/api   FastAPI backend (Python 3.10+)
deploy    Docker Compose, Dockerfiles, example Nginx config
data      Synthetic sample library served by the app
scripts   Sample generator, card-BIN importer
docs      Architecture and engineering notes

Analysis is synchronous per upload: the API stores the file, extracts metadata and text through the parser registry, runs the rule engine (rules.py → Hyperscan prefilter → regex verification → combo/scoring passes), builds the compliance summary, and persists everything; the frontend then renders the report from the task detail endpoint. See docs/architecture.md for the full picture.

Quick start

Backend (FastAPI)

cd src/api
python -m venv .venv
. .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

Without DATABASE_URL the backend falls back to a local SQLite database in tmp/app.db — nothing else to set up.

Hyperscan note: hyperscan ships Linux wheels only. On Windows/macOS, install the requirements without it (pip install -r requirements.txt minus that line, or let the install fail through) — the engine automatically falls back to pure regex.

OCR (optional): install Tesseract for image text extraction, e.g. apt-get install tesseract-ocr tesseract-ocr-chi-sim tesseract-ocr-eng. Without it, images are analyzed for metadata only.

Frontend (Next.js)

cd src/web
npm install
NEXT_PUBLIC_API_BASE_URL=http://127.0.0.1:8000 npm run dev -- --hostname 127.0.0.1 --port 3000

Open http://127.0.0.1:3000.

Docker Compose

cd deploy
cp .env.example .env          # set POSTGRES_PASSWORD, ALLOWED_ORIGINS, etc.
docker compose up -d

This starts the web frontend, the API, and PostgreSQL on an internal network. An example Nginx reverse-proxy server block lives in deploy/nginx/dlp-insight.conf.

Configuration

Variable	Where	Default	Purpose
DATABASE_URL	API	SQLite tmp/app.db	PostgreSQL DSN, e.g. postgresql+psycopg://user:pass@host:5432/db
ALLOWED_ORIGINS	API	http://127.0.0.1:3000,http://localhost:3000	Comma-separated CORS origins
UPLOAD_DIR	API	tmp/uploads	Where uploaded files are stored
PROJECT_ROOT	API	auto-detected	Root used to locate the data/ sample library (set in Docker)
NEXT_PUBLIC_API_BASE_URL	Web	(empty)	Base URL of the API as seen from the browser
NEXT_PUBLIC_SITE_URL	Web	http://localhost:3000	Public site URL for sitemap/robots/canonical metadata

Sample data

Everything under data/ is synthetic — generated by scripts/generate_demo_samples.py and validated by the test suite. It exists so you can exercise the detection engine without touching real personal data. Do not upload real sensitive documents to a publicly reachable deployment.

Testing

cd src/api
. .venv/bin/activate
pytest                        # full suite
pytest tests/test_api.py -k test_name   # single test

Contributing

Contributions are welcome — see CONTRIBUTING.md. Security issues should be reported privately per SECURITY.md.

License

Apache License 2.0 © DLP Insight contributors