MLOps Batch Job — Rolling Mean Signal Pipeline

A minimal MLOps-style batch job demonstrating reproducibility, observability, and deployment readiness.

What it does

1. Loads config from config.yaml
2. Reads data.csv (10,000 rows OHLCV)
3. Computes a rolling mean on the close column
4. Generates a binary signal: 1 if close > rolling_mean, else 0
5. Writes structured metrics.json and detailed run.log
6. Runs locally and inside Docker

---

Project Structure

mlops_task/
├── run.py            # Main batch job
├── config.yaml       # Job configuration
├── data.csv          # OHLCV input dataset (10,000 rows)
├── requirements.txt  # Python dependencies
├── Dockerfile        # Docker container definition
├── README.md         # This file
├── metrics.json      # Sample success output
└── run.log           # Sample log output

---

Local Setup & Run

Requirements

• Python 3.9+

Install dependencies

pip install -r requirements.txt

Run the job

python run.py --input data.csv --config config.yaml --output metrics.json --log-file run.log

Output

• metrics.json — structured metrics (written in both success and error cases)
• run.log — detailed execution log

---

Docker Build & Run

Build

docker build -t mlops-task .

Run

docker run --rm mlops-task

The container includes data.csv and config.yaml, runs the full pipeline, prints metrics.json to stdout, and exits with code 0 on success.

---

Config

seed: 42      # Random seed for reproducibility
window: 5     # Rolling mean window size
version: "v1" # Pipeline version tag

---

Example metrics.json (Success)

{
  "version": "v1",
  "rows_processed": 9996,
  "metric": "signal_rate",
  "value": 0.4991,
  "latency_ms": 31,
  "seed": 42,
  "status": "success"
}

Note: rows_processed is 9996 because the first window-1 (4) rows are excluded from signal computation due to insufficient history for rolling mean.

Example metrics.json (Error)

{
  "version": "v1",
  "status": "error",
  "error_message": "Required column 'close' not found. Available columns: ['date', 'open']"
}

---

Error Handling

The job handles and logs these cases cleanly, writing an error metrics.json for each:

Error	Behaviour
Config file not found	Logs error, writes error metrics, exits 1
Missing config fields	Logs error, writes error metrics, exits 1
Input CSV not found	Logs error, writes error metrics, exits 1
Empty CSV	Logs error, writes error metrics, exits 1
Invalid CSV format	Logs error, writes error metrics, exits 1
Missing close column	Logs error, writes error metrics, exits 1

---

Reproducibility

All runs with the same data.csv and config.yaml produce identical signal_rate and rows_processed values. latency_ms varies by machine but all other fields are deterministic.