Repository navigation
Conversation
Add Romanian (`ro`) as a self-contained language: - RomanianNumberNormalizer: spelled-out cardinals to digits (units, teens, tens, "și" compounds, hundreds, mie/milion/miliard with the "de" Romanian uses from 20 up), the colloquial contracted forms (cinșpe = 15, douăj = 20), ordinals after "al"/"a" and digit ordinals (al 3-lea, a 3-a). "un"/"o" count as 1 only before sută/mie/milion/miliard, so articles stay words. Words match with or without diacritics, and the legacy cedilla letters ş/ţ are read as ș/ț. - LanguageConfig: decimal comma with "virgula" (folded, since the decimal-word steps run after remove_diacritics), thousands dot, symbols and currencies, digit and number words. - Filler words only where their diacritic-folded form cannot be a Romanian word: "ăă" becomes "aa" and is removed, but "ă" (→ "a") and "îî" (→ "ii", the pronoun "îi") are kept. - Replacements: okay/oke/okei/okey → ok. - Unit tests for the number normalizer and operators, a gladia-3 e2e CSV, README row.
- Drop the "de" that links a number of at least 20 (or ending in 00) to its noun, so
"20°", "20 de grade" and "douăzeci de grade" all normalize to "20 grade" (and
"300 de lei" / "trei sute de lei" to "300 lei"). "de" after smaller numbers and in
"de la" ("from") is kept.
- Recognize the informal teens of everyday speech ("cincișpe"/"cincispe" = 15) next to
the standard and most reduced forms.
- Read "și" compounds written as one word ("douăzecișicinci" = 25).
- E2e cases rewritten as natural sentences (battery level, discounts, prices, temperature).
"1.200 €", "1.200€" and "€1.200" become "1200 euro", matching the spoken "o mie două sute de euro" (the shared pipeline drops a symbol written after the amount with a space). As in the Danish normalizer, the Romanian number normalizer rewrites the config's currency symbols itself.
- Spoken clock times after "la"/"ora"/"e" become H:MM before the time-colon protection, so they match written times: "la opt și jumătate" / "la opt jumate" / "la ora 8.30" → 8:30, "opt și(-un) sfert" → 8:15, "opt fără un sfert" and "șapte patruzeci și cinci" → 7:45, "opt fără zece" → 7:50 (time_normalizer.py, via normalize_numeric_time_formats). - A half after an amount is its decimal: "trei kile jumate", "trei kilograme și jumătate" and "3,5 kg" all become "3 virgula 5 kg"; "jumătate de ..." is left alone. - Units: kil/kile/kilogram(e) → kg, kilometru/kilometri → km. - Currency codes and acronyms stay words (expand_all_caps_letter_by_letter=False): "RON" is "ron", not "r o n", and stays distinct from "lei". - Informal numbers: douășpe/doușpe (12, feminine), fused tens-and-five (douășcinci, treișcinci, patrușcinci).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Romanian (
ro) normalization: spelled-out numbers to digits, Romanian fillers, symbols,currencies, clock times and decimal conventions, so Romanian STT output and references compare fairly.
Type of change
Checklist
New language
normalization/languages/romanian/withoperators.py,replacements.py,__init__.pyreplacements.py(not hardcoded inoperators.py)LanguageConfigis filled in with the language's data (separators, currency words, digit words, …)LanguageOperators— onlyexpand_written_numbersandget_word_replacementsare overridden@register_languageand imported innormalization/languages/__init__.pytests/unit/languages/tests/e2e/files/gladia-3/ro.csvNotes
text2numhas no Romanian, sonumber_normalizer.pyparses cardinals: units, teens, tens,"și" compounds (douăzeci și cinci = 25), hundreds, mie/milion/miliard with the "de" Romanian
uses from 20 up (douăzeci de mii = 20000), colloquial contracted forms (cinșpe = 15,
douăj = 20), and ordinals after "al"/"a" (al treilea = al 3-lea = al 3).
removed, but "ă" (→ "a") and "îî" (→ "ii", the pronoun "îi") are kept.
decimal_wordis the folded "virgula", because the decimal-word steps run afterremove_diacritics; this makes "3,5" and "trei virgulă cinci" agree."o sută de lei"); written forms ("20°", "20 grade") have none, so that "de" is dropped after
such numbers. It is kept after smaller numbers and in "de la" ("from").
forms ("cinșpe") are recognized, as are "și" compounds written as one word ("douăzecișicinci").
matching "o mie două sute de euro". The shared pipeline drops a symbol written after the
amount with a space, so, as in the Danish normalizer, the Romanian number normalizer
rewrites the config's currency symbols itself.
"opt fără un sfert", "șapte patruzeci și cinci", and "8.30" written with a dot) become H:MM.
They are converted in
normalize_numeric_time_formatsbecause it runs before the time-colonprotection; a dedicated hook for spoken times may be preferable in a future preset.
"3,5 kg"); kilogram/kilometre spellings map to kg/km.
expand_all_caps_letter_by_letter=False): "RON" is not "lei".How was this tested?