feat: add code coverage, Python linting, and Spark AQE optimization

- Code coverage: Codecov integration with XPlat collector + badge in README
- Python linting: flake8 step in CI before tests, .flake8 config at repo root
- Spark AQE: adaptive.enabled, coalescePartitions, skewJoin with inline comments
  explaining behavior at 100M+ record scale

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: ritunjaym

.flake8

@@ -0,0 +1,8 @@
+[flake8]
+max-line-length = 120
+exclude =
+    *_pb2.py
+    *_pb2_grpc.py
+    venv
+    .git
+ignore = E203, W503

.github/workflows/ci.yml

@@ -44,6 +44,15 @@ jobs:
       - name: Run unit tests
         run: dotnet test tests/VectorCatalog.Api.Tests/VectorCatalog.Api.Tests.csproj --configuration Release --no-build --verbosity normal --logger "trx;LogFileName=test-results.trx"
 
+      - name: Test with coverage
+        run: dotnet test --configuration Release --collect:"XPlat Code Coverage" --results-directory ./coverage
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v4
+        with:
+          directory: ./coverage
+          fail_ci_if_error: false
+
       - name: Publish test results
         uses: dorny/test-reporter@v1
         if: always()
@@ -70,6 +79,11 @@ jobs:
           python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip'
 
+      - name: Lint with flake8
+        run: |
+          pip install flake8
+          flake8 sidecar/ spark/ scripts/ --max-line-length=120 --exclude=*_pb2*.py,*_pb2_grpc.py
+
       - name: Install sidecar dependencies
         working-directory: ./sidecar
         run: |

README.md

@@ -1,6 +1,7 @@
 # Vector Catalog Service
 
 [![CI Pipeline](https://github.com/ritunjaym/vector-catalog-service/actions/workflows/ci.yml/badge.svg)](https://github.com/ritunjaym/vector-catalog-service/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/ritunjaym/vector-catalog-service/branch/main/graph/badge.svg)](https://codecov.io/gh/ritunjaym/vector-catalog-service)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 > **Production-ready microservice for vector search over 100M+ NYC Taxi records**

spark/jobs/ingest_and_embed.py

@@ -79,7 +79,7 @@ def generate_embedding(text):
 
 
 def create_spark_session():
-    """Create Spark session with Delta Lake support"""
+    """Create Spark session with Delta Lake support and AQE optimizations."""
     return (SparkSession.builder
             .appName("NYCTaxi-Ingestion-Embedding")
             .config("spark.jars.packages", "io.delta:delta-spark_2.12:3.1.0")
@@ -88,6 +88,16 @@ def create_spark_session():
             .config("spark.sql.shuffle.partitions", "8")
             .config("spark.executor.memory", "4g")
             .config("spark.driver.memory", "2g")
+            # Adaptive Query Execution (AQE) — Spark 3.x:
+            # Dynamically coalesces shuffle partitions to reduce small-file overhead
+            # at 100M+ record scale where static partition counts cause imbalance.
+            .config("spark.sql.adaptive.enabled", "true")
+            # Merges small post-shuffle partitions into target size (~64MB default)
+            # preventing thousands of tiny tasks on the taxi embedding pipeline.
+            .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
+            # Splits skewed partitions (e.g., popular pickup zones with 10x more
+            # records than average) to prevent stragglers from blocking the stage.
+            .config("spark.sql.adaptive.skewJoin.enabled", "true")
             .getOrCreate())