Skip to content

architecture.md

Latest commit

 

History

History
169 lines (153 loc) · 11.2 KB

architecture.md

File metadata and controls

169 lines (153 loc) · 11.2 KB
Raw

Scan2Target Architecture & Design

📚 Learning Project: This project was created with AI/Copilot assistance as a learning exercise.

Overview

Scan2Target is a web-based scan server for Linux systems, Raspberry Pi, and virtual machines. It centralizes scanner control for USB and network devices, offering a modern web interface and REST API to initiate scans and automatically route documents to configured destinations (SMB shares, email, cloud storage, webhooks, etc.).

Goals and Constraints

  • Platform: Linux servers, Raspberry Pi, VMs; efficient resource usage
  • Backends: eSCL/AirScan and SANE for scanning
  • Targets: SMB/CIFS, SFTP, Email (SMTP), Paperless-ngx, Webhooks, Cloud (Google Drive, Dropbox, OneDrive, Nextcloud/WebDAV)
  • Security: Optional JWT authentication, encrypted credential storage (Fernet AES-128), reverse-proxy friendly HTTPS
  • Extensibility: Pluggable targets, scan profiles, real-time WebSocket updates
  • Analytics: Hourly scan distribution, scanner/target statistics, timeline tracking with proper timezone conversion

High-Level Architecture

+--------------------------------------------------------------+
|                          Web UI (Svelte)                    |
+--------------------------------------------------------------+
         | REST/HTTPS (FastAPI)            | WebSocket events
+----------------------+        +---------------------------------+
|     API Routers      |        |          Auth layer             |
+----------------------+        +---------------------------------+
         |                            |
         |                            v
         |                 +-------------------+
         |                 | Config & Secrets  |
         |                 |  (SQLite + YAML)  |
         |                 +-------------------+
         v                            |
+--------------------------------------------------------------+
|                      Application Core                        |
|  Scanning  | Printing | Targets | Jobs/History | Services     |
|  (SANE/    | (CUPS)   | (SMB,   | (SQLite)     | Schedulers   |
|  eSCL)     |          | SFTP,   |              | Event bus)   |
|            |          | Email,  |              |              |
|            |          | Paperless, Webhook)                |
+--------------------------------------------------------------+
         |
         v
   Device/Backend layer (SANE/eSCL/AirScan, CUPS/IPP)

Module Responsibilities

  • API: Versioned REST endpoints, authentication middleware, request validation, event streaming for job updates.
  • Core/Scanning: Discover scanners (mDNS/Avahi and SANE backends), normalize capabilities, start scans, manage temporary files, and hand off to Targets.
  • Core/Printing: Enumerate printers via CUPS, submit jobs, query queues, print test pages.
  • Core/Targets: Upload/route scan outputs to configured destinations; provide connectivity tests.
  • Core/Config: Manage settings, credentials, profiles; encryption-at-rest for secrets; pluggable storage backends (SQLite primary, YAML for bootstrapping and offline edits).
  • Core/Jobs: Track scan and print jobs, persist status, and expose history.
  • Services: Background schedulers (e.g., retry failed targets), watcher for Paperless consume folder handoff, IP allowlist enforcement helper.
  • Web UI: Dashboard, scan/print pages, targets management, history, and configuration.

Data & Control Flows

Scan Workflow

  1. User Action: User selects scanner, profile (DPI, color, size, format), and target; optional filename prefix.
  2. API: POST /api/v1/scan/start validates payload and enqueues a scan job.
  3. Scanner Selection: Core/Scanning resolves the device via eSCL/AirScan or SANE backend based on discovered inventory.
  4. Execution: Start scan with requested parameters; stream to temporary file.
  5. Post-processing: Convert to PDF/JPEG as needed; apply filename template {prefix}{profile}_{date}_{time}.{ext}.
  6. Target Delivery: Core/Targets writes or uploads the file to the selected destination; optional webhook notification.
  7. Job Tracking: Core/Jobs records status transitions (queued → running → completed/failed/cancelled) and stores metadata (scanner, profile, target, file path/URL). All timestamps stored as UTC. Jobs can be cancelled via POST /api/v1/history/{id}/cancel which stops the background task and updates status.
  8. UI Feedback: Clients poll GET /api/v1/scan/jobs/{id} or receive WebSocket events for progress. Frontend converts UTC timestamps to browser local time. Active scans show cancel button for immediate termination.

Guided Multi-Page Workflow

  1. POST /api/v1/scan/sessions creates a persistent SQLite session and a private directory below /data/scan-sessions.
  2. POST /api/v1/scan/sessions/{id}/capture stores one page in interactive mode or a complete ADF stack in automatic mode. Only authenticated thumbnail blobs are sent to the browser.
  3. Page rotation, deletion and ordering are persisted immediately. An active scanimage process is registered under the session ID and can be terminated by deleting the session.
  4. GET /api/v1/scan/sessions restores unfinished sessions after a browser or service restart. Sessions expire after SCAN2TARGET_SCAN_SESSION_TTL_HOURS.
  5. POST /api/v1/scan/sessions/{id}/finalize applies optional contrast/margin optimization and blank-page removal, creates one PDF, and optionally runs OCRmyPDF for deskewed searchable PDF/A-2 output.
  6. The resulting file enters the persistent delivery retry queue; temporary session pages are removed only after the delivery job has been created.

Print Workflow

  1. User Action: Upload PDF/JPEG/PNG, choose printer and options.
  2. API: POST /api/v1/print stores the file (temp) and submits to CUPS via Core/Printing.
  3. CUPS Handling: Job ID returned; status polled via GET /api/v1/printers/{id}/jobs.
  4. Job Tracking: Core/Jobs records print job metadata and status.

Target Handling

  • Local folder: Ensure directory path exists (/data/scans/YYYY/MM/DD), write file, set permissions.
  • SMB/CIFS: Mount-on-demand using smbclient or pysmb; unmount/cleanup after upload.
  • SFTP (optional): Use paramiko for uploads.
  • Email: Send via SMTP with TLS; attach file and include metadata.
  • Paperless-ngx: Write to consume folder OR POST to HTTP API with token.
  • Webhook: POST JSON metadata plus signed URL/location of the file.

API Overview (v1)

  • GET /api/v1/scan/devices — list discovered scanners and capabilities.
  • GET /api/v1/scan/profiles — list predefined scan profiles.
  • POST /api/v1/scan/start — start a scan with device/profile/target selection.
  • GET /api/v1/scan/sessions — list resumable multi-page sessions.
  • POST /api/v1/scan/sessions — create a persistent multi-page session.
  • POST /api/v1/scan/sessions/{id}/capture — capture one page or one ADF stack.
  • PUT /api/v1/scan/sessions/{id}/pages — persist page ordering.
  • DELETE /api/v1/scan/sessions/{id} — cancel the session and active scanner process.
  • POST /api/v1/scan/sessions/{id}/finalize — optimize, OCR, create and deliver the PDF.
  • GET /api/v1/scan/jobs — list scan jobs.
  • GET /api/v1/scan/jobs/{id} — job status & result link.
  • GET /api/v1/printers — list printers & status.
  • GET /api/v1/printers/{id}/jobs — list jobs for a printer.
  • POST /api/v1/print — upload + submit print job.
  • POST /api/v1/printers/{id}/test — print a test page.
  • GET /api/v1/targets — list targets.
  • POST /api/v1/targets — create target (SMB/SFTP/Email/Paperless/Webhook/Local folder).
  • PUT /api/v1/targets/{id} — update target settings.
  • DELETE /api/v1/targets/{id} — delete target.
  • POST /api/v1/targets/{id}/test — connectivity test.
  • GET /api/v1/history — unified scan/print history.
  • DELETE /api/v1/history — clear completed jobs.
  • DELETE /api/v1/history/{id} — delete single job.
  • POST /api/v1/history/{id}/cancel — cancel running or queued job.
  • POST /api/v1/history/{id}/retry-upload — retry failed upload.
  • GET /api/v1/stats/overview — total scans, success rate, averages.
  • GET /api/v1/stats/timeline — daily scan counts (last 30 days).
  • GET /api/v1/stats/scanners — per-scanner usage statistics.
  • GET /api/v1/stats/targets — per-target delivery statistics.
  • DELETE /api/v1/stats/targets/{name} — delete all jobs for a target.
  • POST /api/v1/auth/login — obtain session/token.
  • POST /api/v1/auth/logout — revoke session.
  • POST /api/v1/homeassistant/scan — Home Assistant scan trigger (supports favorites).
  • GET /api/v1/homeassistant/status — Home Assistant status sensor.

Example payloads:

POST /api/v1/scan/start
{
  "device_id": "escl:HP_Envy_6400",
  "profile_id": "scan_a4_color_300",
  "target_id": "smb:nas_scans",
  "filename_prefix": "invoices_"
}
POST /api/v1/print
{
  "printer_id": "ipp://printer.local/ipp/print",
  "options": {"sides": "two-sided-long-edge"},
  "file_id": "upload-temp-uuid"
}

Configuration Model

  • Storage Choice: SQLite for structured configuration, users, jobs, and targets; YAML for bootstrap defaults and easy manual edits. SQLite chosen for atomic updates, concurrent access, and simple backups on Pi. Secrets stored encrypted (e.g., Fernet key in /etc/scan2target/secret.key or ~/.scan2target/encryption.key).
  • Entities:
    • User: username, password hash, roles, allowed IP subnets.
    • Scanner: id, type (eSCL/SANE), capabilities, last_seen.
    • Printer: id, uri, name, status, defaults.
    • ScanProfile: id, dpi, color_mode, paper_size, format.
    • Target: id, type (local, smb, sftp, email, paperless_folder, paperless_api, webhook), config blob (encrypted fields for credentials), enabled flag.
    • Job: id, type (scan/print), device_id, target_id (scan), printer_id (print), file_path/url, status (queued/running/completed/failed/cancelled), timestamps, logs.
    • ScanSession: id, scanner/profile/source, interactive or automatic capture mode, status, ordered server-side pages and timestamps.

Security Model

  • Auth: Session or JWT tokens; password hashing via argon2 or bcrypt; HTTPS recommended behind Caddy/nginx; optional IP allowlist enforced per request.
  • Secrets: Encrypted credential fields; filesystem isolation (/etc/scan2target/ for secrets, /data/scan2target for runtime data).
  • Network Exposure: Run FastAPI on port 80 (or behind reverse proxy for TLS); disable anonymous access if needed; CSRF protection for web UI.

Deployment

  • Native: Systemd units to start FastAPI (uvicorn) and optional workers; Avahi for mDNS/AirPrint; CUPS installed with IPP Everywhere.
  • Container: Docker Compose defining API, frontend, CUPS, Avahi reflector, and a data volume for /data/scan2target.

Non-Functional Considerations

  • Performance: Use asynchronous I/O for network transfers; scanning/printing operations offloaded to worker threads to avoid blocking event loop.
  • Logging: Structured logging (JSON) with per-module loggers; include job_id correlation; log target upload failures clearly.
  • Observability: Health endpoint, metrics hook (e.g., Prometheus exporter) optional.

Future Extensions

  • Cloud storage targets; hardware button support via GPIO; multi-tenant user roles.