PySpark Playground 🔥

A simple, browser-based environment for learning and practicing Apache Spark with Python.

Features

• User-friendly Web UI - Clean, modern interface accessible via any browser
• Data Ingestion - Upload .txt and .csv files directly through the UI
• Code Editor - Write and execute PySpark code with syntax highlighting
• Pre-loaded Examples - Get started quickly with built-in code samples
• Pre-installed Spark - Apache Spark 3.5.3 ready to use out of the box

Quick Start

Prerequisites

• Docker installed on your machine
• Docker Compose (included with Docker Desktop)

Installation

1. Clone or download this folder to your machine

2. Build and start the container:

   cd pyspark-playground
   docker-compose up --build

3. Open your browser and go to:

   http://localhost:8080
   

4. Start coding! Use the examples in the dropdown or write your own PySpark code.

Stopping the Service

docker-compose down

Usage

Running Code

1. Write PySpark code in the editor
2. Click ▶ Run Code or use the examples dropdown
3. View output in the Output panel

Available Objects

The following are pre-loaded and ready to use:

• spark - SparkSession instance
• sc - SparkContext instance
• F - pyspark.sql.functions module
• pd - pandas library

Uploading Data

1. Drag & drop files onto the upload area, or click to browse
2. Supported formats: .txt, .csv
3. Files are saved to /app/uploads/
4. Click the 📋 button to copy the file path

Reading Your Data

# Read a CSV file
df = spark.read.csv("/app/uploads/your_file.csv", header=True, inferSchema=True)
df.show()

# Read a text file
text_rdd = sc.textFile("/app/uploads/your_file.txt")
print(text_rdd.take(5))

Code Examples

The UI includes several built-in examples:

Example	Description
Hello Spark	Basic connectivity test
RDD Basics	Core RDD operations
DataFrame Operations	Working with DataFrames
Read CSV File	Loading external data
Spark SQL	SQL queries on DataFrames
Word Count	Classic MapReduce example

Configuration

Memory Settings

Edit docker-compose.yml to adjust Spark memory:

environment:
  - SPARK_DRIVER_MEMORY=4g
  - SPARK_EXECUTOR_MEMORY=4g

Port

Default port is 5000. Change in docker-compose.yml:

ports:
  - "8080:5000"  # Access via localhost:8080

File Structure

pyspark-playground/
├── Dockerfile           # Container setup with Spark
├── docker-compose.yml   # Container orchestration
├── requirements.txt     # Python dependencies
├── app.py              # Flask backend
├── templates/
│   └── index.html      # Main UI template
├── static/
│   ├── css/
│   │   └── style.css   # Styling
│   └── js/
│       └── main.js     # Frontend logic
├── uploads/            # Uploaded files (created at runtime)
└── README.md           # This file

Troubleshooting

Docker build fails with apt-get error (exit code 100)

This is a common networking issue. Try these solutions:

Option 1: Restart Docker and rebuild

# Restart Docker Desktop, then:
docker-compose build --no-cache
docker-compose up

Option 2: Use the alternative Dockerfile (recommended)

# Rename Dockerfiles
mv Dockerfile Dockerfile.original
mv Dockerfile.alternative Dockerfile

# Rebuild
docker-compose build --no-cache
docker-compose up

Option 3: Clear Docker cache completely

docker system prune -a
docker-compose up --build

Container won't start

• Ensure Docker is running
• Check if port 5000 is available
• Try docker-compose down then docker-compose up --build

Out of memory errors

• Increase memory in docker-compose.yml
• Ensure Docker has enough allocated memory (Docker Desktop settings)

Code execution is slow

• First run initializes Spark (takes ~10-15 seconds)
• Subsequent runs are faster

Tech Stack

• Apache Spark 3.5.3 - Data processing engine
• Python 3.11 - Runtime
• Flask - Web framework
• PySpark - Python API for Spark