Mon is spoken by roughly one million people across Myanmar and Thailand and is classified by UNESCO as vulnerable. Usable NLP corpora for it remain scarce.
This is a Mon-language text corpus for NLP research, language-model pretraining, and OCR training data. It is the training data source for MonOCR.
| Source | Shards | Lines | Characters | Mon/Myanmar | Other |
|---|---|---|---|---|---|
| Mon Wikipedia | 5 | 910,080 | 25,590,114 | 21,725,259 | 3,864,855 |
| Mon News Agency | 3 | 121,020 | 12,066,365 | 10,993,107 | 1,073,258 |
| Custom Collections | 1 | 119,739 | 6,831,401 | 3,681,874 | 3,149,527 |
| MonDictDB | 1 | 94,658 | 2,484,241 | 2,357,122 | 127,119 |
| Telegram / Facebook | 2 | 4,479 | 95,098 | 81,479 | 13,619 |
| OCR Extracted | 1 | 733 | 37,624 | 36,824 | 800 |
| Machine-generated | 1 | 962 | 23,142 | 21,567 | 1,575 |
| Total | 14 | 1,251,671 | 47,127,985 | 38,897,232 (82.5%) | 8,230,753 (17.5%) |
Raw file size: ~124 MB (uncompressed UTF-8)
The Other column is not noise to be filtered out. Mon and Burmese are mixed in ordinary written use, and any system reading real pages will meet them together — so mixed-script material is kept as it appears rather than separated. The MonDictDB shard is the clearest case: Mon headwords with Burmese definitions, kept whole.
Unicode NFC normalization — All text is strictly normalized to NFC, ensuring consistent grapheme cluster representation regardless of input method or source platform.
Preservation pipeline — The pipeline preserves all Myanmar script blocks (U+1000–U+109F, Extended-A/B) and intentional spacing essential to Mon script readability. Only non-linguistic noise is stripped (BOM, ZWJ, ZWNJ, control codes).
Deduplication — New content is deduplicated against the existing corpus at the sentence level (Mon-script skeleton) before it is added, so re-scraped material is not appended twice. A small number of short segments still recur across shards, so treat shard boundaries as packaging rather than guaranteed dedup splits.
MonCorpusCollection/
├── shards/ # Distribution shards (~20MB each)
│ ├── monnews_shard_*.txt # Mon News Agency (IMNA) articles
│ ├── wikipedia_shard_*.txt # Mon Wikipedia articles
│ ├── telegram_*_shard_*.txt # Telegram channel messages
│ ├── facebook_shard_*.txt # Facebook page posts
│ ├── mondictdb_shard_*.txt # MonDictDB dictionary entries and examples
│ ├── gemini_generated_shard_*.txt # LLM-authored, unverified — see below
│ ├── ocr_extracted_shard_*.txt # OCR-extracted text
│ └── custom_shard_*.txt # Specialized and legacy collections
├── results/latest/ # Character/bigram/trigram frequency over the shards
├── scripts/ # build_shards.py, shard_stats.py, frequency counters
└── docs/CORPUS.md # Cleaning and normalization spec
Iterate through shards/ for model training. Each file is standard UTF-8 text.
# Per-source stats (shards, lines, characters, Mon/Myanmar share)
python scripts/shard_stats.py
# Character / bigram / trigram frequency over all shards
python scripts/corpus_counter_normalized.py shards --output-dir results/latest --all-chars
# Add newly scraped .txt files as deduplicated shards (dry-run first)
python scripts/build_shards.py --source monnews --input path/to/monnews --dry-runMIT. If you use this data, please attribute Mon Corpus Collection and the underlying sources below — every shard traces to one of them.
| Source | Shards | Origin |
|---|---|---|
| Mon Wikipedia | wikipedia_shard_* |
mnw.wikipedia.org, CC BY-SA |
| Mon News Agency (IMNA) | monnews_shard_* |
Independent Mon News Agency |
| MonDictDB | mondictdb_shard_* |
MonDictDB by Barnista, MIT |
| Custom Collections | custom_shard_* |
Specialized and legacy collections |
| Telegram / Facebook | telegram_*, facebook_shard_* |
Public channel and page posts |
| OCR Extracted | ocr_extracted_shard_* |
Text recovered from scanned material |
| Machine-generated | gemini_generated_shard_* |
Authored by Google Gemini, not transcribed from any source |
On the OCR-extracted and MonDictDB shards.
ocr_extracted_shard_*is the output of an OCR system, so it can carry that system's recognition errors. MonDictDB records some definitions produced by machine translation; those rows are excluded at import, but the exclusion is a property of the importer rather than of this file. Treat both as lower-confidence than the Wikipedia and news shards if your use is sensitive to transcription accuracy.
Warning
gemini_generated_shard_* is not human-authored Mon and is not verified.
A "Mon history" dataset produced by Google Gemini from its own knowledge —
neither the Mon prose nor the historical claims (dates, kingdoms, place names)
have been checked by a fluent reader or against a source. Exclude this shard
for language-model pretraining, and do not cite its history.
What is measured about it, so the label is not the only thing you have to go
on. It is novel text, not recycled: of 2,050 sentences long enough to
deduplicate, exactly 1 matched the rest of the corpus, and that one is the
connective ပ္ဍဲ ခေတ် ပစ္စုပ္ပန်၊. It is also heavily repetitive at source —
466 distinct sentences expanded to 2,050 by internal repetition (77.3%),
one fragment appearing 178 times. The 962 lines here are what survives
deduplication. At 23,142 characters it is 0.05% of the corpus.
- Normalize all text to NFC before submission.
- Provide clear source attribution for new data, and add a row to the table above.
- Add shards with
scripts/build_shards.py, which deduplicates against the existing corpus. Run--dry-runfirst. - Re-run
scripts/shard_stats.pyand update the Dataset table in the same change.