Realtime Browser Object Detection

100% client-side real-time object detection. YOLOv8n runs entirely in the browser via ONNX Runtime Web: webcam in, bounding boxes out, no backend and no server-side inference.

Stack: React 18, TypeScript, Vite, Tailwind CSS, ONNX Runtime Web, Vitest. Model export pipeline in Python (Ultralytics, ONNX, onnxruntime quantization).

Architecture

webcam (getUserMedia) → letterbox preprocess (640x640, HWC→CHW) → ONNX Runtime Web
  (WebGL, falls back to WASM) → decode YOLOv8 output (8400 anchors) → NMS → canvas overlay

• src/inference/: preprocessing, ONNX session management, postprocessing (decode + NMS), in pure TypeScript with no framework dependency.
• src/hooks/: useWebcam (camera access + front/back toggle), useDetection (requestAnimationFrame inference loop with configurable frame skip), useBenchmark (rolling FPS/latency stats), useBenchmarkRunner (FP32 vs INT8 static-image comparison).
• src/components/: video + canvas overlay, live stats panel, controls, benchmark modal.
• scripts/export_model.py: exports YOLOv8n to ONNX, quantizes it to INT8 and benchmarks both.

Detection runs against whichever frame is current when each inference call starts; frame skip controls how many requestAnimationFrame ticks are spent between inference calls, trading detection latency for CPU headroom.

Technical highlights

• Zero backend: npm run build produces a static bundle; the model files are the only network requests the app makes, served as static assets.
• WebGL acceleration with WASM fallback: ONNXDetector (src/inference/detector.ts) tries the WebGL execution provider first and falls back to WASM if unavailable, logging which provider was used.
• Letterbox preprocessing preserves aspect ratio when resizing to the model's fixed 640x640 input, then maps detected boxes back to the original frame's coordinates (src/inference/preprocess.ts, postprocess.ts).
• INT8 quantization trades a 4x smaller download for (on the CPU benchmark below) slower inference.
• COOP/COEP headers configured in both vite.config.ts (dev) and vercel.json (prod), required for ONNX Runtime Web's multi-threaded WASM backend (SharedArrayBuffer).

Getting started

Requirements: Node.js 20+ (CI runs on the current LTS) and, to generate the model files, Python 3.9+.

1. Generate the model files

The ONNX models are not committed (they are build artifacts, ignored by .gitignore). Generate them once into models/, which is served to the app through the public/models symlink:

python3 -m venv .venv
source .venv/bin/activate
pip install -r scripts/requirements.txt
python scripts/export_model.py

This downloads yolov8n.pt, exports models/yolov8n.onnx, quantizes it to models/yolov8n_int8.onnx, benchmarks both on CPU (100 runs each), and writes models/export_benchmark.json.

2. Run the app

npm ci
npm run dev      # http://localhost:5173

Allow camera access when the browser prompts for it.

Scripts

Script	What it does
npm run dev	Vite dev server with COOP/COEP headers
npm run build	Type-check (tsc -b) and build the static bundle into dist/
npm run preview	Serve the production build locally
npm run typecheck / npm run lint	tsc --noEmit
npm test	Vitest unit tests (YOLOv8 output decoding, NMS, latency stats)
npm run deploy	vercel --prod (requires the Vercel CLI and a linked project)

vercel.json applies the same COOP/COEP headers in production.

Benchmark results (CPU, ONNX Runtime Python)

Recorded from one local run of scripts/export_model.py; numbers will differ by machine.

Model	Size	Mean	P50	P99
YOLOv8n FP32	12.26 MB	25.8 ms	25.8 ms	29.1 ms
YOLOv8n INT8 (dynamic)	3.34 MB	61.7 ms	60.4 ms	106.8 ms

INT8 was 2.4x slower than FP32 in this run. Dynamic post-training quantization (onnxruntime.quantization.quantize_dynamic) mainly accelerates MatMul/Gemm-heavy ops; YOLOv8n is convolution-heavy, and the onnxruntime CPU build used had no optimized INT8 conv kernel path, so quantized convs fall back to slower generic kernels. INT8 does win on size (3.34 MB vs 12.26 MB, 4x smaller, so faster to download). The in-app benchmark (Run Benchmark in the UI) measures the ONNX Runtime Web comparison live in your browser, which uses a different kernel set (WASM/WebGL) and may behave differently. Static quantization with calibration would likely close this gap but is not implemented here.

Testing

npm test

Unit tests cover the pure inference logic: decoding the raw [1, 84, 8400] YOLOv8 tensor, confidence filtering, class-aware non-max suppression, mapping letterboxed coordinates back to the source frame, and latency percentile/FPS statistics. Webcam capture and canvas rendering are not covered by automated tests and need a real browser to verify.

Known limitations

• The models must be generated locally before the app can run detections (see above).
• INT8 dynamic quantization underperforms FP32 on the CPU/runtime combination benchmarked above; the model selector lets you compare both live.

License

MIT © 2026 Parakh Sharma