Russian Language Adaptation of BLOOM

A research prototype for adapting BLOOM to Russian without full-model fine-tuning. The project combines a Russian tokenizer, newly trained token embeddings, and parameter-efficient tuning on the Russian OSCAR corpus.

Reported result: the BLOOM-560M adaptation improved DaNetQA performance by 12% compared with the unadapted baseline.

Approach

The pretrained transformer is kept frozen while a small subset of parameters is trained:

1. Load a pretrained BLOOM checkpoint.
2. Replace the original tokenizer with the RuGPT-3 tokenizer.
3. Train new token embeddings and a tied language-modeling head on Russian OSCAR text.
4. For larger models, add prompt tuning and distribute the frozen backbone through Petals.

This keeps the experiment substantially cheaper than updating every model parameter.

Experiments

Model	Trainable components	Runtime	Entry point
BLOOM-560M	Token embeddings and tied LM head	Local PyTorch / CUDA	train_hf.py
BLOOM-7.1B	Token embeddings, LM head, and prompt tuning	Petals	train_petals.py
BLOOM-176B	Token embeddings, LM head, and prompt tuning	Petals	train_petals.py with --big

Training data is streamed from the Russian split of OSCAR, so the full corpus does not need to be downloaded in advance.

Running the experiments

The code expects Python, PyTorch with CUDA, Transformers, Datasets, and tqdm. Petals experiments additionally require petals and termcolor.

# BLOOM-560M
python train_hf.py

# Resume BLOOM-560M from a local checkpoint
python train_hf.py --resume

# BLOOM-7.1B through Petals
python train_petals.py --exp_name bloom-7b1

# BLOOM-176B through Petals
python train_petals.py --big --exp_name bloom-176b

Repository map

• train_hf.py — BLOOM-560M adaptation loop with streaming OSCAR data.
• train_petals.py — distributed 7.1B/176B experiments with prompt tuning.
• train.py — earlier local training prototype.
• inference.py — checkpoint loading and exploratory evaluation code.
• *.ipynb — experiment notebooks and training-loss exploration.

Scope and limitations

This repository contains the adaptation and training experiments, not a production package. Paths and CUDA settings are configured in the scripts, dependencies are not locked, and model checkpoints are not committed. The DaNetQA evaluation was run separately; its benchmark harness and prediction artifacts are not included in this public repository.

References

• BLOOM-560M
• OSCAR
• Russian SuperGLUE / DaNetQA
• Adapting BigScience Multilingual Model to Unseen Languages