Mosaic AI Sales Data Project

Overview

This project implements an end-to-end data pipeline for sales data using the Medallion Architecture (Bronze → Silver → Gold), followed by logging, embedding generation, and AI-driven processing using Databricks Mosaic AI.

Architecture

Raw Sales Data
      │
      ▼
┌─────────────┐
│   Bronze    │  Raw ingestion (unprocessed sales data)
└─────────────┘
      │
      ▼
┌─────────────┐
│   Silver    │  Cleaned, validated, and transformed data
└─────────────┘
      │
      ▼
┌─────────────┐
│    Gold     │  Aggregated, business-ready data
└─────────────┘
      │
      ▼
┌─────────────────────────────┐
│  Mosaic AI Processing        │
│  - Logging                   │
│  - Embedding Generation       │
│  - AI Model Processing        │
└─────────────────────────────┘

Project Structure

MOSAIC_AI/
├── pipeline/              # Medallion architecture pipeline notebooks
│   ├── bronze_ingestion.py
│   ├── silver_transformation.py
│   └── gold_aggregation.py
├── notebooks/             # Mosaic AI processing notebooks
│   ├── logging.py
│   ├── embedding_generation.py
│   └── ai_processing.py
└── README.md

⚙️ Pipeline Details

1. Bronze Layer

Raw sales data is ingested as-is from the source system with minimal transformation, preserving the original data for traceability.

2. Silver Layer

Data is cleaned, deduplicated, validated, and transformed into a structured, queryable format.

3. Gold Layer

Business-level aggregations and metrics are created, ready for reporting and downstream consumption.

Mosaic AI Processing

After the Gold layer is ready, the data flows into the Mosaic AI processing stage:

• Logging: Tracks pipeline runs, data quality checks, and processing metadata.
• Embedding Generation: Converts relevant sales data into vector embeddings for semantic search / similarity use cases.
• AI Processing: Uses Databricks Mosaic AI to run inference/analysis on the processed data.

Tech Stack

• Databricks (Delta Lake, Workflows)
• Mosaic AI (Databricks' AI/ML platform)
• PySpark / Python
• Delta Live Tables

How to Run

1. Clone this repository into Databricks Repos.
2. Run the pipeline notebooks in order: bronze_ingestion.py → silver_transformation.py → gold_aggregation.py.
3. Run the Mosaic AI notebooks: logging.py → embedding_generation.py → ai_processing.py.

Future Improvements

• Automate pipeline scheduling via Databricks Workflows/Jobs.
• Add data quality monitoring dashboards.
• Expand embedding use cases (semantic search, recommendations).

---

This is a practice/portfolio project built using dummy sales data to demonstrate a Medallion Architecture pipeline integrated with Mosaic AI capabilities.