GitHub Tech Pulse

An end-to-end DataOps platform designed to extract, model, test, and visualize open-source repository trends using the GitHub REST API.

---

Table of Contents

• Overview
• Architecture
• Tech Stack
• Data Modeling & Quality
• Project Structure
• Orchestration & Lineage
• Getting Started
  • Prerequisites
  • Environment Setup
  • Running with Docker Compose
  • Running Locally
• Interactive Features & CLI
• Observability & Alerting
• CI/CD & Testing
• Roadmap

---

Overview

GitHub Tech Pulse automates the ingestion of repository metadata across target technology domains (e.g., data engineering, DevOps, platform engineering, machine learning operations). Ingested data lands in a raw storage layer, transforms into a Star Schema data warehouse via dbt incremental models, undergoes automated quality tests, and is orchestrated with Dagster.

Key capabilities:

• Multi-field search extraction (in:name,description) capturing untagged and keyword-rich repositories.
• Rate-limit aware extraction with exponential backoff retries via Tenacity.
• Asset-based pipeline orchestration and automated daily scheduling via Dagster.
• Incremental data modeling and delta loading via dbt.
• Automated data quality checks on primary keys, nullability, and uniqueness.
• Live pipeline observability and alerting via Telegram Bot.
• Dynamic parameterization supporting custom topic and keyword ingestion via CLI and web UI.
• Containerized deployment with Docker and automated testing via GitHub Actions.

---

Architecture

                       +-------------------+
                       |  GitHub REST API  |
                       +---------+---------+
                                 |
                    (Ingestion Worker - Python)
                                 |
                                 v
                +----------------------------------+
                |      PostgreSQL (Raw Layer)      |
                |     raw.github_repositories      |
                +----------------+-----------------+
                                 |
                          (dbt incremental)
                                 |
                                 v
                +----------------------------------+
                |  PostgreSQL (Analytics Layer)    |
                |  - analytics_staging             |
                |  - analytics_marts (Star Schema) |
                +----------------+-----------------+
                                 |
                 +---------------+---------------+
                 |                               |
                 v                               v
  +------------------------------+ +---------------------------+
  |     Streamlit Dashboard      | |    Dagster Orchestrator   |
  |     (Port 8501 / Web UI)     | |    (Port 3000 / Web UI)   |
  +------------------------------+ +---------------------------+

---

Tech Stack

• Language & Package Management: Python 3.11, Astral uv
• Data Ingestion: requests, tenacity, psycopg2-binary
• Data Transformation & Modeling: dbt-postgres (v1.8)
• Pipeline Orchestration: Dagster, dagster-webserver, dagster-dbt
• Database / Warehouse: PostgreSQL 16
• Visualization: Streamlit, Pandas
• Observability & Alerts: Telegram Bot API
• Containerization & Orchestration: Docker, Docker Compose
• Quality Assurance & CI/CD: GitHub Actions, pytest, ruff, sqlfluff

---

Data Modeling & Quality

The analytics layer implements dimensional modeling (Star Schema) with incremental processing:

1. Staging Layer (analytics_staging):

   • stg_github_repositories: Parses raw JSONB fields into typed relational attributes. Declared as a view dependent on source('raw', 'github_repositories').

2. Marts Layer (analytics_marts):

   • dim_repositories: Deduplicated entity table containing repository metadata. Configured as incremental with unique_key='repo_id'.
   • fct_daily_repo_snapshots: Daily metric snapshot capturing star count, fork count, open issues, and daily star velocity computed using window functions (LAG()). Configured as incremental with composite key ['repo_id', 'snapshot_date'].

3. Data Quality Tests:

   • Primary key uniqueness on dim_repositories.repo_id.
   • Null checks on critical identifiers and metrics (repo_id, topic, stars_count, snapshot_date).

---

Project Structure

github-tech-pulse/
├── .github/
│   └── workflows/
│       └── ci.yml                # CI pipeline (lint, unit tests, dbt build, workflow_dispatch)
├── config/
│   ├── settings.py               # Environment configuration loader
│   └── topics.yaml               # Target topic definitions
├── dbt_transforms/
│   ├── models/
│   │   ├── staging/              # Staging views, sources, and schema tests
│   │   └── marts/                # Incremental fact and dimension tables
│   ├── dbt_project.yml
│   └── profiles.yml
├── src/
│   └── github_tech_pulse/
│       ├── bot/                  # Telegram notification helper
│       ├── config/               # Settings and topic configs
│       ├── dashboard/            # Streamlit dashboard application
│       ├── ingestion/            # API client and database loader
│       └── orchestration/        # Dagster asset definitions and schedules
├── tests/                        # Pytest suite with mocked responses
├── scripts/
│   └── init_db.sql               # Database schema initialization script
├── Dockerfile                    # Container definition optimized with uv
├── docker-compose.yml            # Multi-service composition
├── Makefile                      # Command shortcuts for common workflows
├── pyproject.toml                # Project metadata and dependencies
└── uv.lock                       # Deterministic dependency lockfile

---

Orchestration & Lineage

The platform uses Dagster for asset-based orchestration. Dagster manages pipeline execution order, data dependencies, and scheduling.

Global Asset Graph

raw/github_repositories (Python Ingestion Asset)
       │
       ▼
staging/stg_github_repositories (dbt View)
       │
       ├──► marts/dim_repositories (dbt Incremental Table)
       │
       └──► marts/fct_daily_repo_snapshots (dbt Incremental Table)

• Daily Schedule: daily_github_pulse_job runs daily at 08:00 AM UTC (0 8 * * *).
• Materialization: Trigger individual assets or the full graph from the Dagster UI.

To launch the Dagster UI:

make dagster

Navigate to http://localhost:3000 to inspect the global lineage graph and execution logs.

---

Getting Started

Prerequisites

• Docker and Docker Compose (v2.0+)
• Python 3.11+ and uv (for local development)

Environment Setup

Create an environment configuration file from the template:

cp .env.example .env

Configure environment variables in .env:

GITHUB_TOKEN=your_token_here
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=github_pulse
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
TELEGRAM_BOT_TOKEN=your_telegram_bot_token
TELEGRAM_CHAT_ID=your_telegram_chat_id

Running with Docker Compose

To build images and launch PostgreSQL and the Streamlit Dashboard:

docker compose up --build -d

Access the dashboard at http://localhost:8501.

To stop services:

docker compose down

Running Locally

1. Install dependencies:

uv sync

2. Start PostgreSQL:

docker compose up -d postgres

3. Run data ingestion:

uv run python -m github_tech_pulse.ingestion.pipeline

4. Execute dbt transformations and data tests:

cd dbt_transforms
uv run dbt run --profiles-dir .
uv run dbt test --profiles-dir .
cd ..

5. Launch the Dagster webserver:

make dagster

6. Launch the Streamlit dashboard:

make dashboard

---

Interactive Features & CLI

CLI Ingestion Options

The ingestion pipeline uses a multi-field search strategy (in:name,description) to discover relevant repositories even if authors omitted GitHub topic tags. It supports custom keyword parameters and execution flags:

# Ingest specific topics or multi-word phrases with custom record count
uv run python -m github_tech_pulse.ingestion.pipeline --topics "data platform" "lakehouse" rust --per-page 20

# Ingest without triggering dbt transformation
uv run python -m github_tech_pulse.ingestion.pipeline --topics ai-agents --no-dbt

Dashboard On-Demand Ingestion

The Streamlit dashboard includes an Explore & Ingest widget in the sidebar, allowing users to enter custom keywords directly in the UI (e.g., data platform, lakehouse, llm). When submitted, the application executes the multi-field ingestion worker, refreshes dbt models, and updates analytics charts in real time.

---

Observability & Alerting

The platform integrates a lightweight Telegram notification system (src/github_tech_pulse/bot/notifier.py):

• Job Success: Sends an execution summary with the total repository count ingested.
• API Errors: Dispatches alerts on GitHub API rate limits or HTTP failures.
• Transformation Failures: Dispatches error messages if dbt model execution or tests fail.
• Fault-Tolerant: Skips alerting silently if credentials are not configured, preventing pipeline crashes.

---

CI/CD & Testing

The continuous integration workflow (.github/workflows/ci.yml) runs automatically on push and pull requests, and supports manual execution with custom inputs via workflow_dispatch:

• Code Quality: Static analysis and formatting checks using ruff.
• Unit Testing: Ingestion client, Telegram notifier, and error handling tests via pytest.
• Integration Testing: Ephemeral PostgreSQL service initialization, pipeline execution, and dbt build verification.
• Manual Triggering: Run custom topic test suites directly from the GitHub Actions UI.

Run checks locally:

# Run unit tests
uv run pytest

# Run linter
uv run ruff check .

# Run dbt tests
cd dbt_transforms && uv run dbt test --profiles-dir .

---

Roadmap

Planned platform enhancements:

• Observability: Prometheus metrics exporter for API rate limits and execution duration, visualized via Grafana dashboards.
• Data Lake Storage: Object storage integration with S3/MinIO using Apache Iceberg or Parquet table formats.
• Change Data Capture (CDC): Debezium connector for real-time Postgres table replication.