MOSIS — Multimodal OCR-Based Semantic Image Retrieval System

MOSIS is an AI-powered OCR and semantic understanding platform that processes handwritten and printed text from images, intelligently classifies content type, and generates context-aware outputs. It combines Vision LLMs, transformer-based OCR, traditional OCR engines, and image generation models into an end-to-end multimodal document understanding pipeline.

---

Features

• Multi-engine OCR pipeline — Ollama Vision, TrOCR, EasyOCR, and Tesseract working in concert
• Intelligent content classification across 5 categories:
  • Single word
  • Short phrase
  • Paragraph
  • Prescription
  • Mixed math/text
• Semantic image retrieval with AI-generated contextual visuals
• Prescription extraction — structured JSON output for medicine data
• Math OCR — LaTeX rendering support for mathematical expressions
• Real-time streaming via Server-Sent Events (SSE)
• Metrics dashboard — accuracy, confusion matrix, precision, recall, and F1-score tracking

---

Tech Stack

Layer	Technology
Backend	Python, Flask, Flask-CORS
Frontend	React
OCR Engines	Ollama Vision, TrOCR, EasyOCR, Tesseract
Image Generation	Stable Diffusion, Unsplash API
Image Processing	OpenCV, PIL
Metrics Storage	JSON-based analytics system

---

Project Structure

aaa/
├── backend/
│   ├── app.py                # Main Flask application
│   ├── pipeline.py           # Core OCR pipeline
│   ├── rx_pipeline.py        # Prescription OCR pipeline
│   ├── seed_metrics.py       # Metrics seeding script
│   ├── test_pipeline.py      # Pipeline tests
│   ├── requirements.txt      # Python dependencies
│   ├── metrics_log.json      # Metrics log
│   └── metrics_store.json    # Metrics store
├── frontend/
│   ├── public/
│   │   └── index.html
│   └── src/
│       ├── App.jsx
│       ├── MetricsDashboard.jsx
│       └── index.js
├── test_images/
│   └── generate_test_images.py
├── start.ps1                 # One-click startup script (Windows)
└── .gitignore

---

Getting Started

Prerequisites

• Python 3.8+
• Node.js 16+
• Ollama installed and running
• Tesseract OCR installed (Windows installer)

1. Clone the Repository

git clone https://github.com/ggbadbi/Multimodal-OCR-Based-Semantic-Image-Retrieval-System.git
cd Multimodal-OCR-Based-Semantic-Image-Retrieval-System

2. Backend Setup

cd backend

# Create and activate virtual environment
python -m venv venv

# Windows
.\venv\Scripts\Activate.ps1

# macOS/Linux
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Seed initial metrics (optional)
python seed_metrics.py --reset

3. Start Ollama

# Windows (PowerShell)
taskkill /F /IM ollama.exe 2>$null; Start-Sleep 2; ollama serve

4. Run the Backend

# From /backend with venv activated
python app.py

5. Frontend Setup

cd ../frontend
npm install
npm start

The app will be available at http://localhost:3000.

Quick Start (Windows)

Alternatively, use the included PowerShell script from the project root:

.\start.ps1

---

Use Cases

• Intelligent OCR systems for scanned documents
• Medical prescription understanding and extraction
• Handwritten text recognition
• Semantic document analysis
• Educational math OCR with LaTeX output
• AI-assisted image retrieval systems

---

Project Goal

MOSIS moves beyond traditional OCR by understanding what the text represents and responding contextually — through semantic analysis, structured extraction, and intelligent visual generation.

---

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you'd like to change.

---

License

This project is licensed under the MIT License.