ISL Real-Time Translator 🇮🇳

Indian Sign Language (ISL) real-time translator — webcam se hand gestures detect karke text + voice (TTS) mein convert karta hai.

Built with MediaPipe + 1D-CNN/BiLSTM. Runs live on laptop CPU — no GPU required.

Live Demo

📹 Demo GIF yahan aayega — demo.gif file ko repo mein daal ke niche replace karo:

![Live Demo](demo.gif)

Demo kaise banao (5 min):

1. python realtime_translator.py run karo
2. Windows Game Bar kholo (Win + G) → Record dabao
3. 3-4 gestures karke dikhao (namaste, thank_you, yes...)
4. Record khatam karo, phir video ko ezgif.com pe upload karke GIF banao
5. GIF ko repo mein demo.gif naam se daal do, upar wali line uncomment karo

Features

• 9 gestures: background, help, hey, namaste, please, sorry, thank_you, water, yes
• Real-time detection (~1.2–1.5 sec) — confidence threshold (0.80) + history smoothing, fake detection filter
• Text-to-Speech (pyttsx3, threaded — app hang nahi hota)
• ~96% test accuracy (10-class benchmark: 9 gestures + background)
• Feature engineering: wrist-origin normalization + velocity features (168 features/frame)
• Data augmentation: jitter, scale, shift, frame-drop, time-warp
• Smart buffer: 30-frame sliding window + no-hand (background) handling

Tech Stack

Component	Library
Hand tracking	MediaPipe 0.10.21 (21 landmarks × 2 hands)
Model	TensorFlow 2.19.1 — Conv1D + BatchNorm + BiLSTM
Vision	OpenCV 4.11.0
TTS	pyttsx3
Metrics	scikit-learn

Project Structure

isl_translator/
├── models/                  # Trained artifacts
│   ├── isl_model.keras      # Trained model (~1.5MB)
│   ├── label_map.json       # Gesture → index mapping
│   └── index_to_label.json  # Model output → label
├── dataset/                 # (NOT committed — ~4GB, self-recorded)
│   └── <gesture_name>/      #   har gesture ka folder: sample_000.npy ...
├── record_data.py           # Basic data recorder (30 samples/gesture)
├── record_data_v2.py        # Advanced recorder (variation phases, 50 samples)
├── check_data.py            # Recorded samples verify karo
├── extract_landmarks.py     # Raw frames → normalized landmarks (.npy)
├── train_model.py           # Model train karo (augmentation + BiLSTM)
├── realtime_translator.py   # LIVE app — webcam + text + voice
├── requirements.txt         # Exact tested dependencies
└── README.md

Setup (Windows)

# Python 3.11 recommended
python -m venv isl_env
isl_env\Scripts\activate

pip install -r requirements.txt

How to Use

Step 1 — Record data (har gesture ke liye alag folder, dataset/ ke andar):

python record_data_v2.py
# Gesture name poochega (e.g. namaste), then 50 samples record karo

Step 2 — Extract landmarks:

python extract_landmarks.py

Step 3 — Train:

python train_model.py

Step 4 — Run live:

python realtime_translator.py
# Q dabao quit karne ke liye

Note: dataset/ folder intentionally not committed (bahut bada). Har machine pe apna data record karo. Naya gesture add karna ho toh bas dataset/ mein naya folder banao — pipeline automatically pick up karega.

Dataset (Self-Recorded)

Dataset khud record kiya hai — 270 samples, 9 gestures (30 samples per gesture, 60 frames each, 320×240 webcam).

Kyun Kaggle dataset use nahi kiya:

• Public ISL datasets (jaise ISLHTD) mostly isolated word images hain — continuous video sequence ke liye nahi
• ISL regional variations hain — kisi aur region ka dataset hamare use-case se match nahi karta
• Self-recorded dataset = apna data engineering skill ka proof — interview mein "dataset kahan se aaya?" ka answer: "Maine khud record kiya — model exactly mere target environment (laptop webcam) pe trained hai"

Ye intentionally ek feature hai, limitation nahi. Naya gesture add karna = bas naya folder banana.

Results

Version	Gestures	Accuracy
v1 (2 gestures, basic CNN)	2	83.33%
v2 (8 gestures, BiLSTM + augmentation)	8	91.67%
v3 (normalization + velocity features)	8	95.83%
v4 (with hey)	9	~96%

Per-gesture detail (v4): background/please/sorry/thank_you/yes 100%, water 91%, namaste 91%, help 86%, hey f1 0.91.

Roadmap

• TFLite conversion → Android app
• .exe build (PyInstaller)
• Reverse mode: speech → ISL (Vosk)
• GUI dashboard
• 50+ samples per gesture → 95%+ live accuracy

License

MIT