Saccade is a high-performance desktop application designed for real-time video stream manipulation and face swapping. Built with a systems-first approach, it decouples the native GUI and processing pipeline from the heavy machine-learning inference engine.
By leveraging Rust for frame orchestration, concurrency, and zero-copy memory management, alongside ONNX Runtime for hardware-accelerated model execution, Saccade achieves ultra-low latency suitable for live streaming and local video processing without Python runtime overhead.
- Core & Pipeline: Rust (Async Tokio channels, thread pooling, zero-copy buffer management)
- Desktop Shell: Tauri v2 (Native OS window management, IPC bridge)
- Frontend UI: Preact + Vite (Lightweight, reactive control panel)
- Inference Engine: ONNX Runtime (
ortcrate) with hardware acceleration support (CUDA / DirectML / CoreML) - Computer Vision: OpenCV bindings / custom image processing shaders for face cropping, alignment, and blending
Saccade/
├── src-tauri/ # Rust backend core
│ ├── src/
│ │ ├── models/ # Directory for ONNX weight files
│ │ ├── pipeline/ # Frame capture, queue management, and threading
│ │ ├── inference/ # ONNX Runtime wrappers (Detector & Inswapper)
│ │ ├── processing/ # Face alignment, color correction, and blending
│ │ └── main.rs # Tauri application entry point
│ └── Cargo.toml
├── src/ # Frontend UI (Preact)
└── README.md
Phase 1: Core Architecture & Pipeline Foundation
- Initialize Tauri v2 workspace and Rust backend structure.
- Implement async frame capture pipeline for local webcams and video files.
- Set up zero-copy frame buffer pooling to minimize memory allocations.
- Integrate basic Preact UI dashboard for device selection and previews.
Phase 2: Inference Integration
- Configure ONNX Runtime (
ort) with CUDA and CPU execution providers. - Optimize tensor conversion pipelines between image buffers and model inputs.
- Implement post-processing core utilities (NMS algorithm, IoU math, and detection data types).
- Implement SCRFD / RetinaFace model loader & anchor decoder for high-speed face detection and landmark extraction.
- Integrate InsightFace (
inswapper_128.onnx) embedding extraction and face-swapping inference loop.
Phase 3: Post-Processing & Blending Optimization
- Implement facial alignment and affine transformation matrices.
- Add seamless blending algorithms (Poisson blending / Mask feathering) to eliminate harsh edges.
- Implement temporal smoothing filters to reduce jitter and flickering across consecutive video frames.
Phase 4: Advanced Vision & Virtual Camera Pipeline
- Integrate native Virtual Camera driver support (OBS VirtualCam / v4l2loopback IPC).
- Implement multi-face tracking and target selection (ID-based face locking).
- Add real-time facial expression transfer / reenactment module (e.g., LivePortrait / FOMM integration).
- Support custom target mask fine-tuning and Occlusion Aware Blending (hair, hands, glasses handling).
Phase 5: Audio, Latency Optimization & Security Controls
- Add real-time Voice Conversion (RVC) pipeline synchronized with video output streams.
- Implement hardware-accelerated NVENC / VAAPI / AMF video encoding for low-bitrate streaming.
- Integrate local model weight encryption and secure memory handling for proprietary weights.
- Add automated bench-harness for FPS, latency jitter, and VRAM consumption profiling.
Phase 6: Performance Tuning & Release
- Benchmark end-to-end latency and throughput across various GPU architectures.
- Implement multi-threaded frame dropping and queue backpressure handling for real-time streaming.
- Package production binaries with bundled ONNX runtimes and asset management.
- Rust toolchain (stable)
- Node.js & pnpm / npm
- CUDA Toolkit (optional, recommended for GPU acceleration)
git clone https://github.com/itsventie/Saccade.git
cd Saccade
pnpm install
pnpm tauri dev