LoomIndex

A memory-efficient, high-performance concurrent web crawler in modern C++20.
Async I/O (libcurl) · custom thread pool · Bloom-filter URL dedup · strict RAII & thread safety.

---

LoomIndex is a lightweight, high-performance, concurrent web crawler developed in modern C++ (C++20). Engineered for speed and scalability, it serves as a robust foundation for high-throughput web scraping and data indexing projects.

Key Features

• Asynchronous I/O: Leverages libcurl (curl_multi) for scalable, non-blocking HTTP requests, capable of handling dozens of concurrent connections efficiently.
• Custom Thread Pool: Native C++20 thread-pool implementation that safely dispatches parser and processor workloads.
• Memory-efficient Bloom Filter: Built-in Bloom filter for rapid URL deduplication, drastically reducing the RAM footprint compared to traditional hash sets.
• Strict RAII & Thread Safety: Deleted copy semantics, move-aware resources, and graceful shutdown across every component.
• Docker Support: Fully containerized environment for instant, reproducible builds and zero-config execution.

Architecture

A reliable multi-threaded producer–consumer model with a clear separation between network I/O and data processing:

graph TD;
    subgraph Core Engine
        CE[CrawlerEngine] -->|Spawns| TP[ThreadPool / Workers]
        CE -->|Pumps I/O| AF[AsyncFetcher]
    end
    subgraph Memory & Queue
        TP -->|Pops URLs| UF[URLFrontier]
        UF -->|Filters duplicates| BF[BloomFilter]
        AF -->|Callback on parse| TP
        TP -->|Pushes new links| UF
    end
    style CE fill:#f9f,stroke:#333,stroke-width:2px;
    style BF fill:#bbf,stroke:#333,stroke-width:2px;

Loading

Quality & Stability

Reliability is a core pillar of LoomIndex. Every component is covered by a GoogleTest suite, and the CI pipeline builds the project in both Release and Debug on every push and runs the full test suite via ctest.

Component	Coverage
Core logic (Bloom Filter, ThreadPool, URLFrontier)	Unit-tested
Network layer (AsyncFetcher init/cleanup)	Unit-tested
System lifecycle (startup → graceful shutdown)	Integration-tested

The suite validates the Bloom filter's false-positive rate and ensures thread safety across the CrawlerEngine.

Getting started

Using Docker (recommended)

The easiest way to build and run the crawler with zero local setup:

# Build the image
docker build -t loomindex .

# Run the demo (defaults to https://example.com)
docker run --rm loomindex

# Run with custom seed URLs
docker run --rm loomindex https://github.com https://wikipedia.org

Building with CMake (Linux / macOS / WSL)

Requires a C++20 compiler and libcurl (libcurl4-openssl-dev on Debian/Ubuntu):

git clone https://github.com/LTolo/LoomIndex.git
cd LoomIndex

# Configure & build
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

# Run the unit tests
cd build && ctest --output-on-failure && cd ..

# Run the crawler
./build/LoomIndex https://example.com

Project structure

LoomIndex/
├── include/LoomIndex/   # Public headers (CrawlerEngine, ThreadPool, BloomFilter, ...)
├── src/                 # C++ implementations + main.cpp entrypoint
├── tests/               # GoogleTest unit tests (concurrency + data structures)
├── docs/                # Project plan & images
├── CMakeLists.txt       # Top-level build configuration
├── run_project.sh       # Compile → test → run helper
├── Dockerfile           # Container definition
└── .github/workflows/   # CI: build (Release + Debug) and run tests

Tech stack

C++20 · libcurl (curl_multi) · CMake · GoogleTest · Docker · GitHub Actions

License

Released under the MIT License.