We currently use the LatinWhitespaceTokenizer in machine.py to tokenize the pretranslations in order to align and then reinsert the paragraph markers. For scripts that do not delimit words with spaces, this tokenizer outputs only one big token with the entire pretranslation meaning that we cannot intelligently place the paragraph marker.
A simple solution would be to default to a character tokenizer when there is no whitespace. Many of the languages that do not use spaces don't use spaces precisely because most words are monosyllabic and are thus represented by a single character (or maybe a couple in compound words).
We've had this as an issue in Serval for a while sillsdev/machine.py#280 and marked it blocked pending satisfactory resolution in silnlp, but I've just realized there was no issue in silnlp to track this.
We currently use the
LatinWhitespaceTokenizerin machine.py to tokenize the pretranslations in order to align and then reinsert the paragraph markers. For scripts that do not delimit words with spaces, this tokenizer outputs only one big token with the entire pretranslation meaning that we cannot intelligently place the paragraph marker.A simple solution would be to default to a character tokenizer when there is no whitespace. Many of the languages that do not use spaces don't use spaces precisely because most words are monosyllabic and are thus represented by a single character (or maybe a couple in compound words).
We've had this as an issue in Serval for a while sillsdev/machine.py#280 and marked it blocked pending satisfactory resolution in silnlp, but I've just realized there was no issue in silnlp to track this.