vector-doc-engine

Production-resilient TypeScript engine for PDF text chunking, Google Gemini vector embeddings, and local ChromaDB semantic vector search.

Overview

vector-doc-engine is a lightweight library and CLI tool designed for document processing pipelines. It splits documents cleanly at natural text boundaries, generates embeddings with Google Gemini, and indexes them into ChromaDB for fast similarity retrieval.

Features

• Recursive Text Chunking: Splits paragraphs, lines, and sentences cleanly before applying character limits.
• Resilient API Layer: Uses exponential backoff with delay retries to handle rate limits and temporary network drops.
• Streaming Batch Processing: Processes chunks in steady batches to keep memory usage minimal on large documents.
• Similarity Threshold Filtering: Filters search results against a similarity floor to exclude off-topic matches.

Installation

Install as a project dependency:

npm install git+https://github.com/DileepWick/vector-doc-engine.git

Or via GitHub Packages registry:

npm install @dileepwick/vector-doc-engine

Quick Start

1. Environment Configuration

Create a .env file in your root directory:

CHROMA_URL=http://localhost:8000
GEMINI_API_KEY=your_gemini_api_key
GEMINI_EMBED_MODEL=gemini-embedding-2-preview

2. Infrastructure Setup

Start a local ChromaDB instance:

docker run -p 8000:8000 chromadb/chroma

3. Programmatic Usage

View Code Examples

Ingest PDF Document

import { ingestPdfDocument } from "@dileepwick/vector-doc-engine";

const result = await ingestPdfDocument({
  filePath: "./data/documents/sem-reg.pdf",
});

console.log(`Ingested ${result.totalChunks} chunks.`);

Perform Vector Search Query

import { queryVectorSearch } from "@dileepwick/vector-doc-engine";

const matches = await queryVectorSearch({
  query: "What are the main key takeaways?",
  topK: 3,
  minSimilarity: 0.35,
});

matches.forEach((match, idx) => {
  console.log(`[${idx + 1}] Score: ${match.score.toFixed(4)} | Excerpt: "${match.doc}"`);
});

CLI Execution

View CLI Commands

Build Package

npm run build

Ingest Document via CLI

npx ts-node src/ingest.ts sem-reg.pdf

Query Vector Search via CLI

npx ts-node src/ask.ts "What are the key takeaways?"

Project Structure

vector-doc-engine/
├── src/
│   ├── index.ts               # Public API exports
│   ├── ingest.ts              # Document ingestion API
│   ├── ask.ts                 # Vector search query API
│   ├── chunker.ts             # Text chunking utility
│   └── embedder.ts            # Gemini API embedder with retries
├── data/documents/            # Default document storage
├── dist/                      # Built JavaScript binaries & declarations
├── docs/                      # Technical documentation & guides
└── tests/                     # Unit test suites

Documentation

• Architecture & Foundations
• System Hardening & Audit
• GitHub Packages Publishing