A from-scratch, step-by-step implementation of GPT (a decoder-only Transformer), built in PyTorch while following Andrej Karpathy's Let's build GPT lecture — and then extended with a small experiment comparing each architectural addition head-to-head.
Rather than jumping straight to the full model, this repo builds it up in five stages, training each one on the same data with the same hyperparameters, to see exactly what each piece (embeddings, attention, multi-head attention, full transformer blocks) actually buys you.
tokenizer.py # CharTokenizer: char-level encode/decode
data.py # loads input.txt, builds train/val splits, get_batch()
bigram.py # Model 1 — bigram baseline
gpt_step1_embeddings.py # Model 2 — + token/positional embeddings
gpt_step2_attention.py # Model 3 — + single-head self-attention
gpt_step3_multihead.py # Model 4 — + multi-head attention
gpt_step4_transformer.py # Model 5 — full transformer (blocks + FFN + LayerNorm + residuals)
demo.py # standalone demo of the weight-init scheme
notebooks/ # exploratory notebooks (tokenizer, batching, attention mask, results)
original/ # Karpathy's original nanoGPT-lecture code, kept for reference/diffing
input.txt # tinyshakespeare corpus (character-level training data)
| # | Model | New mechanism | Script |
|---|---|---|---|
| 1 | Bigram | Single [V×V] embedding lookup, no context |
bigram.py |
| 2 | Embeddings only | Token + positional embeddings, no interaction between tokens | gpt_step1_embeddings.py |
| 3 | Single-head attention | One causal self-attention head — tokens can finally attend to the past | gpt_step2_attention.py |
| 4 | Multi-head attention | 4 parallel attention heads, concatenated + projected | gpt_step3_multihead.py |
| 5 | Full transformer | 4 stacked blocks (attention + feed-forward + LayerNorm + residuals) | gpt_step4_transformer.py |
Each script is self-contained and runnable on its own — data.py and
tokenizer.py are the only shared pieces.
All five models trained on the same character-level Shakespeare corpus with identical hyperparameters (batch size 32, block size 64, embedding dim 128, 4 attention heads, dropout 0.1, 10,000 steps, AdamW lr 3e-3):
| Model | Params | Val loss @ 10k steps |
|---|---|---|
| 1 — Bigram | ~0.01M | 2.473 |
| 2 — Embeddings only | ~0.05M | 2.490 |
| 3 — Single-head attention | ~0.15M | 2.404 |
| 4 — Multi-head attention | ~0.20M | 2.115 |
| 5 — Full transformer | ~0.80M | 1.593 |
Takeaways:
- Positional embeddings alone don't help (model 2 is marginally worse than the bigram) — position info is only useful once there's a mechanism (attention) that can act on it.
- The biggest single jump is multi-head vs. single-head attention.
- Depth (stacking transformer blocks with residuals + LayerNorm) beats width — model 5 hadn't even converged yet at 10k steps and still had headroom.
See notebooks/gpt_complexity_comparison.ipynb for the full training curves.
Requires Python 3.10+.
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
# train any of the models — each prints loss curves and a text sample at the end
python bigram.py
python gpt_step4_transformer.pyEach script trains for a few thousand steps on CPU in well under a minute and
prints Shakespeare-flavored generated text at the end. Swap in a CUDA-enabled
torch build if you have a GPU — the scripts pick it up automatically.
To explore interactively, launch the notebooks:
jupyter lab notebooks/input.txt is the classic tinyshakespeare
corpus (~1MB) used throughout Karpathy's lecture — the complete works of
Shakespeare, concatenated into one plain-text file.
Built while following Andrej Karpathy's Neural Networks: Zero to Hero series, specifically the "Let's build GPT" lecture. The original/ folder preserves his original lecture code as a reference point / diffing base.
MIT — see LICENSE.