GEO Observatory

A local-first toolkit for monitoring, analyzing, and improving how AI crawlers and search engines discover and cite your websites.

Live instance: geo.bedvibe.studio

Measured 2026-08-27: 7,063 AI-crawler events across 19 crawler families, 11,502 validated events in collection health, 1,709 bot events in the preceding 24 hours.

What it does

Parses nginx and Cloudflare edge logs, classifies crawlers, and verifies the ones that can be verified — Googlebot and Bingbot by reverse DNS with forward confirmation, the method both publishers document. Every observation is normalized into an immutable, append-only ledger, and the raw provider payload is preserved exactly as received, so any past day can be reconstructed rather than remembered.

What it does not do, stated plainly

IP-range verification for GPTBot, ClaudeBot and PerplexityBot is not implemented. Those are matched on user-agent and recorded as unverified rather than assumed genuine.

That distinction is not pedantry. In a separate analysis over Cloudflare edge logs on 2026-08-25, every client IP claiming to be Googlebot was checked against Google's own published ranges — 585 prefixes from googlebot.json and special-crawlers.json. On one host, 138 of 146 Googlebot-labelled requests were spoofed, from seven addresses. Counting crawler visits by user-agent string is not measurement.

No causal claims. This data supports temporal association between site changes and crawler behaviour. It does not support "this change caused that indexing outcome", and nothing here will say that it does. NOT MEASURED is a third state and is never reported as zero.

Structure

geo_observatory/
├── crawler/          # Parse nginx logs, classify bots, verify authenticity, ingest to DB
├── dashboard/        # Read-only Flask dashboard at http://127.0.0.1:8700
├── verifier/         # (Day 2) Verify structured data, entity consistency, schema coverage
├── inventory/        # (Day 2) Website inventory scanner — titles, schema, metadata
├── retrieval/        # (Day 3) Ask AI systems if they cite your content
├── reports/          # Auto-generated markdown reports
├── data/             # SQLite database (append-only)
├── sample_logs/      # Drop nginx combined-format logs here
└── ROADMAP.md        # Development roadmap

Quick Start

# 1. Drop an nginx access log into sample_logs/
cp /var/log/nginx/access.log sample_logs/

# 2. Parse and ingest
python crawler/ingest.py sample_logs/access.log

# 3. Launch dashboard
python dashboard/app.py
# → http://127.0.0.1:8700

Requirements

• Python 3.10+
• Flask (pip install flask)
• No other external dependencies

Data Model

SQLite at data/crawls.sqlite, append-only. See crawler/db.py for schema.