JARVIS is an architectural framework and system design for a private, single-user personal AI assistant. Designed for high intelligence, context awareness, local-first privacy, fluid voice interactions, and cinematic user experience.
JARVIS is built from the ground up to be a true personal assistant, not a generic multi-tenant AI chatbot or SaaS wrapper. It combines hierarchical semantic vector memory, dynamic model routing (evaluating task difficulty vs. risk), modular tool execution with explicit approval flows, replaceable AI/Voice providers, and a minimal cinematic dark user interface across desktop and mobile.
The complete system architecture, design specifications, and operational models are documented in detail below:
- AGENTS.md — Core developer principles, AI agent guidelines, coding standards, and Definition of Done.
- PRODUCT.md — Product vision, single-user personal AI identity, core UX principles, and capability spectrum.
- ARCHITECTURE.md — System architecture, subsystem boundaries, modular replaceability, and flow diagrams.
- DESIGN.md — Cinematic dark visual design system, Core visualizer states, typography, and UX patterns.
- MEMORY.md — Hierarchical vector memory system (ephemeral, contextual, project, preference, core), multi-factor retrieval scoring, and memory suppression rules ("remember more than exposed").
- AI.md — Multi-provider LLM abstraction layer (cloud providers, local models via Ollama/vLLM, fallback strategies).
- MODEL_ROUTING.md — Task Difficulty vs. Risk evaluation matrix, deterministic route bypasses, and dynamic model selection.
- APPROVALS.md — Risk engine, multi-tier permission boundaries, biometric step-up, interactive confirmation flows, and audit logs.
- TOOLS.md — Tool registry, dynamic schema definition, secure sandboxed execution, and lifecycle management.
-
VOICE.md — Modular voice pipeline (Mic
$\rightarrow$ VAD$\rightarrow$ STT$\rightarrow$ Core$\rightarrow$ TTS), replaceable providers, and latency optimization. - CALLING.md — In-app VoIP/calling architecture (WebRTC / LiveKit abstraction) for private real-time audio calls.
- MOBILE.md — React Native + Expo iOS/Android mobile client design, background services, and native bridges.
- DESKTOP.md — Tauri desktop application architecture, hotkeys, system tray, and OS-level integrations.
- SECURITY.md — Zero-secret policy, local credential keychains, sandbox boundaries, and threat model.
- PRIVACY.md — Local-first principles, private single-user data isolation, telemetry policy, and cloud minimization.
- DEVELOPMENT.md — Project structure, tooling setup, markdown linting, and development workflow.
- ROADMAP.md — 20-milestone implementation roadmap from Core Runtime through Autonomous Mode, including dependencies and exit criteria.
JARVIS now follows a dependency-driven M1–M20 milestone plan:
- M1–M4 establish the core runtime, vector memory, provider abstraction, and intelligent model routing.
- M5–M8 add multi-agent orchestration, budgets, recovery, and parallel execution.
- M9–M14 deliver controlled tools plus desktop, mobile, voice, calling, and cinematic interface layers.
- M15–M20 introduce long-term personal intelligence, autonomous workspaces, external/distributed agents, security hardening, and autonomous mode.
See ROADMAP.md for the full milestone table, dependency graph, and exit criteria.
┌─────────────────────────┐
│ User Clients (UI) │
│ Desktop (Tauri) / │
│ Mobile (React Native) │
└────────────┬────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ JARVIS Orchestrator │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Voice Subsystem│ │ Dynamic Router │ │ Approval Engine │ │
│ │ (STT/TTS/Call) │ │ Difficulty/Risk │ │ (Risk & Auth) │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬─────────┘ │
│ │ │ │ │
│ └─────────────────────────┼─────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────┐ │
│ │ Hierarchical Memory Engine │ │
│ │ (Vector Retrieval & Contextual) │ │
│ └────────────────┬────────────────┘ │
└─────────────────────────────────────┼──────────────────────────────────────┘
│
┌───────────────────────┴───────────────────────┐
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────┐
│ AI Provider Abstraction │ │ Execution Sandboxes │
│ (Cloud / Local Ollama) │ │ (Local Tools / System) │
└──────────────────────────┘ └──────────────────────────┘
- Private Single-User: Built exclusively for one user. No multi-tenant complexity or SaaS friction.
- Semantic Vector Memory: Memory uses embeddings and semantic retrieval, with contextual activation and strict suppression thresholds so old or irrelevant facts are not improperly exposed.
- Difficulty vs. Risk Routing: Simple tasks bypass LLMs or use cheap local models; risky tasks require explicit user authorization regardless of model size.
- Clean Replaceability: All key dependencies (LLM, STT, TTS, Vector Store, Desktop Shell) are hidden behind abstract interfaces.
Private / Confidential — Single-User Personal AI System Architecture.