A hands-on tutorial introducing DNA foundation models, with a particular focus on the Nucleotide Transformer family and SegmentNT.
The notebooks move from fundamental DNA sequence representations to contextual embeddings, biological feature exploration, supervised fine-tuning, and nucleotide-resolution genome annotation.
Google Colab is the easiest way to run the tutorials because the notebooks install their required dependencies and can use a hosted GPU runtime.
Foundation models learn general-purpose representations from large collections of unlabeled data. In genomics, these models are trained on DNA sequences and can learn contextual patterns associated with motifs, regulatory elements, genes, sequence composition, and other biological properties.
However, a DNA foundation model is shaped by several design decisions:
- How the nucleotide sequence is tokenized
- Which genomes and species are included during pretraining
- The model architecture and context length
- The self-supervised training objective
- Which model layer is used to extract representations
- How sequence-level or nucleotide-level predictions are constructed
This repository explores these ideas through two complementary tutorials.
| Tutorial | Main focus | Model |
|---|---|---|
| Part 1 | DNA encoding, tokenization, contextual embeddings, biological interpretation, probing, and supervised fine-tuning | Nucleotide Transformer |
| Part 2 | Genome annotation at single-nucleotide resolution | SegmentNT with a Nucleotide Transformer backbone |
After completing the tutorials, you should be able to:
- Explain why biological sequences require appropriate numerical representations.
- Compare one-hot encoding with learned token embeddings.
- Understand how k-mer tokenization is applied to DNA sequences.
- Load a pretrained Nucleotide Transformer from Hugging Face.
- Extract contextual representations from different transformer layers.
- Explore whether embedding components respond to sequence motifs or genomic features.
- Evaluate pretrained embeddings using visualization and supervised probes.
- Fine-tune a DNA foundation model for binary sequence classification.
- Evaluate genomic classifiers using F1 score and Matthews correlation coefficient.
- Use SegmentNT to generate nucleotide-resolution genome annotation tracks.
- Recognize important computational, reproducibility, and data-leakage considerations when working with genomic foundation models.
flowchart LR
A[DNA sequence] --> B[DNA tokenizer]
B --> C[6-mer token representation]
C --> D[Nucleotide Transformer]
D --> E[Contextual embeddings]
E --> F[Motif and feature exploration]
E --> G[Embedding visualization]
E --> H[Linear classification probe]
D --> I[Supervised fine-tuning]
I --> J[Promoter or enhancer classification]
E --> K[SegmentNT convolutional head]
K --> L[14 nucleotide-resolution annotation tracks]
DNA-FMs/
├── README.md
├── T1_workshop_FM_NT.ipynb
└── T2_workshop_sNT_infer.ipynb
Introduces DNA foundation-model concepts and demonstrates how to:
- Represent DNA sequences as strings and numerical tensors
- Calculate basic sequence properties
- Construct one-hot encodings
- Understand k-mer tokenization
- Load and inspect a Nucleotide Transformer
- Extract hidden-state representations
- Investigate motif-associated embedding components
- Annotate biological features such as TATA boxes and translated regions
- Evaluate sequence embeddings with PCA and logistic regression
- Fine-tune a Nucleotide Transformer for promoter and enhancer classification
Introduces SegmentNT and demonstrates how to:
- Load the pretrained SegmentNT architecture
- Combine Nucleotide Transformer embeddings with a segmentation head
- Retrieve a genomic region from Ensembl
- Divide long genomic regions into model-compatible windows
- Produce nucleotide-level predictions across 14 annotation tracks
- Visualize predicted genome annotations as heatmaps
- Run the model on a user-selected genomic interval
Reduce the batch size or sequence length and restart the runtime to release cached GPU memory.
For fine-tuning, also consider gradient accumulation, mixed precision, partial freezing, or a smaller model.
Confirm that:
- The runtime has internet access
- Hugging Face is reachable
- Sufficient disk space is available
- The selected model repository is public
- The local Hugging Face cache is writable
The SegmentNT head is downloaded using gdown. Network restrictions or changes to the hosted file may interrupt the download. Confirm that the checkpoint file exists and was downloaded completely before initializing the model.
Ensembl REST requests can fail because of:
- Temporary service unavailability
- Invalid genomic coordinates
- An unsupported species name
- Missing request headers
- Excessively large requests
Test a smaller interval and verify the coordinates in the Ensembl genome browser.
SegmentNT windows are padded to the model’s expected size. Ensure that padded positions are removed after prediction before interpreting or plotting the reconstructed genomic interval.
Create a clean environment and install compatible versions of PyTorch, Transformers, Accelerate, and Datasets. Pinning known working versions is recommended for workshops and long-term reproducibility.
-
Dalla-Torre et al. The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics.
-
Benegas et al. SegmentNT: Annotating the genome at single-nucleotide resolution with DNA foundation models. https://www.biorxiv.org/content/10.1101/2023.05.24.542096v1
-
Nucleotide Transformer models on Hugging Face https://huggingface.co/InstaDeepAI
-
Nucleotide Transformer repository https://github.com/instadeepai/nucleotide-transformer
-
Genomic Benchmarks https://github.com/ML-Bioinfo-CEITEC/genomic_benchmarks
-
Ensembl REST API https://rest.ensembl.org/
-
Ensembl Genome Browser https://www.ensembl.org/
-
JASPAR transcription-factor binding profiles https://jaspar.elixir.no/
-
Eukaryotic Promoter Database https://epd.expasy.org/epd/
This tutorial was prepared for the Kircherlab Workshop.
The notebooks build on models, datasets, code examples, and educational resources developed by the Nucleotide Transformer, SegmentNT, InstaDeep, Hugging Face, Ensembl, JASPAR, Genomic Benchmarks, and broader computational-genomics communities.
The SegmentNT inference notebook is adapted from the official Nucleotide Transformer example:
instadeepai/nucleotide-transformer/notebooks/inference_segment_nt.ipynb
Please cite the original model, dataset, and software publications when using or adapting material from this repository.
Suggestions, corrections, and educational improvements are welcome.
You can contribute by:
- Opening an issue describing the proposed change
- Forking the repository
- Creating a focused branch
- Updating the notebook or documentation
- Opening a pull request with a clear explanation of the change
Potential contributions include additional exercises, tested environments, alternative models, new genomic datasets, improved visualizations, and reproducibility fixes.
No license file is currently included in this repository.
Unless a license is added, standard copyright restrictions apply. Contact the repository owner before redistributing or substantially reusing the tutorial materials.
Pyaree Mohan Dash
- GitHub: @pyareedash
- Portfolio: pyareedash.github.io
DNA foundation models provide powerful representations, but their outputs become biologically useful only when paired with careful evaluation, appropriate datasets, reproducible workflows, and domain knowledge.