Appearance Anomaly Detection for Autonomous Driving

This project develops an additional Safety Layer integrated directly into the Perception Stack of advanced driver-assistance systems (ADAS) and high-level autonomous vehicles. It addresses the challenge of detecting appearance and geometric anomalies (Out-of-Distribution - OOD obstacles) on road surfaces that traditional 2D object detectors (such as YOLO) often fail to classify.

🎬 Live Demos

---

🚀 Overview & Project Motivation

Traditional 2D object detection systems are bounded by their training dataset distribution (In-Distribution), making them prone to missing or misclassifying unexpected, rare obstacles (Out-of-Distribution - OOD) on the road, such as:

• Deep water puddles, slippery oil spills.
• Newly formed potholes, missing/damaged manhole covers.
• Scattered rocks, debris from accidents.
• Fallen traffic cones or irregularly shaped objects.

The Appearance Anomaly Detection system identifies any abnormal entity/geometric variation on the vehicle's trajectory by combining 2D appearance feature analysis with 3D physical height estimation, offering early warnings for steering or braking actions.

---

🛠️ Design Constraints & Embedded Hardware Deployment

• Zero Heavy Dependencies: The use or import of heavy deep learning/scientific libraries (such as PyTorch, Torchvision, Transformers, SciPy, Scikit-learn) is strictly prohibited in all production and testing code.
• Core Libraries: Only numpy, opencv-python (cv2), and onnxruntime (ort) are used to ensure the lightest footprint for deployment on vehicles' embedded hardware.
• Flexible Hardware Acceleration (Execution Providers):
  • ONNX Runtime Execution Providers are not hardcoded.
  • Automatically detected via the ONNX_PROVIDERS environment variable or configured via PipelineConfig.
  • On macOS M-series: CNN models (YOLOv8n, YOLOPv2) utilize Apple Neural Engine acceleration (CoreMLExecutionProvider), while Transformer models (Depth Anything v2) are executed on the CPU (CPUExecutionProvider) to avoid graph partitioning overhead.

---

📐 System Architecture (4-Layer Pipeline)

The system employs parallel processing and asymmetric fusion divided into 4 main layers:

[Camera Input]
      │
      ├──► Layer 1: Road Segmentation (YOLOPv2 ONNX) ──┐
      │                                                ├──► [ROI & Background Mask]
      ├──► Layer 2: Object Detection (YOLOv8n ONNX) ───┘
      │
      ├──► Layer 3: CV-based Appearance Anomaly Detection
      │     ├── Multi-seed Region Growing (LAB Color Space)
      │     ├── Voters (Edge Voter & Texture Voter)
      │     ├── SurfaceConflictGate (Semantic-Appearance Conflict)
      │     └── Sub-Anomaly Scanner (Small Obstacles in Shadows)
      │
      ├──► Layer 4: Monocular Depth & Geometric Verification
      │     ├── Depth Anything v2 (ONNX) ──► RANSAC Ground-Plane
      │     └── Road Slope Removal ──► Elevation Residual (Height) Calculation
      │
      ▼
[Fusion Classifier] ──► Class A/B/C Classification ──► 2x4 Visualization Grid (demo.py)

Layer Details:

1. Layer 1: Road Segmentation (YOLOPv2): Identifies the drivable area to define the Region of Interest (ROI), eliminating sidewalks and irrelevant peripheral elements.
2. Layer 2: Object Detection (YOLOv8n): Detects vehicles, pedestrians, etc., to create a masking layer, preventing known objects from triggering anomaly tracking.
3. Layer 3: CV-based Appearance Anomaly Detection:
   • Multi-seed Region Growing: Performs color-based segmentation (in LAB color space) using seed boxes located at the bottom of the image to extract candidate road anomalies.
   • Voters (Edge/Texture): Evaluates contour edges (Edge) and surface texture consistency (Texture) to calculate a 2D confidence score.
   • SurfaceConflictGate: Detects semantic-appearance conflicts (e.g., YOLOPv2 segments the area as drivable, but the appearance deviates significantly from the ego lane, indicating paint markings or oil spills).
   • Sub-Anomaly Scanner: Scans for small 3D obstacles hidden within larger anomaly regions (e.g., small rocks within shadow regions).
4. Layer 4: Monocular Depth & Geometric Verification:
   • Employs Depth Anything v2 (ONNX) to estimate relative depth/disparity.
   • Fits a ground plane model using RANSAC (subsampling up to 5000 road pixels to minimize computational overhead).
   • Computes elevation residual (height profile): Distinguishes flat anomalies (elevation $\approx 0$ like oil spills, water puddles) from 3D obstacles (elevation $> 0$ like rocks, cones).
5. Ego-Motion Compensation:
   • Uses Lucas-Kanade Optical Flow tracking on road features.
   • Estimates a local Homography $H$ ($3 \times 3$) representing road plane motion.
   • Performs local area scale compensation at the object's centroid $(x,y)$: $s_{\text{area}} = |\det H| / |w|^3$ (where $w = h_{20}x + h_{21}y + h_{22}$).
   • Measures Centroid Residual via $H$. If the residual is $< 2.0$ px, the movement is explained by camera ego-motion $\rightarrow$ Re-classified as MOTION_STATIC, eliminating False Positives of static objects during forward vehicle movement.

---

🚦 Anomaly Classification Philosophy (Class A/B/C)

To balance False Positives and False Negatives, anomalies are classified into 3 classes:

• Class A (Safe - Ignore): Flat appearance variations that do not affect tire dynamics (shadows, road paint). The vehicle proceeds at default speed.
• Class B (Caution - Slow Down): Flat anomalies with potential friction hazards (oil spills, loose gravel, deep water puddles). The vehicle gently decelerates and prepares for anti-slip control, avoiding emergency braking or abrupt steering.
• Class C (Danger - Emergency Avoidance): 3D obstacles with distinct physical elevation or dynamic hazards. The vehicle triggers emergency braking or evasive steering immediately.
• Current Policy: Temporarily classify all flat anomalies as Class B (Caution) to guarantee safety until auxiliary sensors are integrated.

---

📂 Project Directory Structure

Appearance-Anomaly-Detection/
├── anomaly_detection/          # Core pipeline package
│   ├── core/                   # Region growing logic
│   ├── fusion/                 # Fusion classifier (Class A/B/C)
│   ├── layers/                 # YOLO, YOLOPv2, Depth, Async worker
│   ├── postprocessing/         # Morphological processing
│   ├── tracking/               # Centroid tracker & Ego-motion Homography
│   ├── voters/                 # Edge and Texture voters
│   ├── visualization/          # Render utilities
│   ├── config.py               # Centralized configuration hyperparameters
│   ├── pipeline.py             # Main pipeline coordinator
│   └── types.py                # Common data structures and types
├── tests/                      # Python unittest suite
│   ├── acceptance_test.py      # Integration and smoke tests
│   ├── test_depth_estimation.py# Ground plane fitting and depth estimation tests
│   ├── test_ego_motion.py      # Homography compensation and fallback tests
│   ├── test_performance_opt.py # Real-time optimization validation tests
│   ├── test_sub_anomaly.py     # Sub-anomaly scanner validation tests
│   └── test_surface_conflict.py# SurfaceConflictGate validation tests
├── models/                     # Directory for ONNX model weights
├── demo.py                     # Entry point for GUI and CLI demos
└── requirements.txt            # Minimal dependencies list

---

⚙️ Installation & Usage

1. Setup Environment

Create a Python virtual environment and install dependencies:

# Create venv
python3 -m venv venv

# Activate venv
source venv/bin/activate

# Install requirements
pip install -r requirements.txt

2. Download ONNX Models

Place the following ONNX models in the models/ directory:

• models/yolopv2.onnx
• models/yolov8n.onnx
• models/deep_anything_v2.onnx

3. Run Demo

The project provides two run modes via demo.py:

Interactive GUI Mode (OpenCV):

Allows real-time tuning of seed box sizes, color tolerances, horizon lines, etc., via trackbars on a 2x4 visualization grid.

# Run on a video file
python demo.py --mode gui --input input_demo/demo4_shorter_video.mp4

# Run on synthetic road simulation
python demo.py --mode gui --input synthetic

Offline CLI Mode:

Processes the entire video offline and writes the output directly to a video file.

python demo.py --mode cli --input input_demo/demo4_shorter_video.mp4 --output output_demo/result.mp4

---

🧪 Testing Suite

Run all 61 tests using Python's standard unittest:

python -m unittest discover -s tests

(Tests automatically mock model inference and skip heavy ONNX checks if local model files are missing to ensure high execution speed).