Tests don't handle non-UTF-8 locales well

[aitap]

This grew out of a response to a comment.

We have approximately 24 locale-dependent tests, and they do slightly different things:

• Some of them (e.g. 1590.12) are there to test behaviour in a UTF-8 environment.
• Others (like 1590.14) assume that a non-UTF-8 locale uses the Windows-1252 encoding; I started a branch to address that.
• 2253.* try to establish a locale in which the tests should pass.
  • Strictly speaking, it's not a UTF-8 code page that the Japanese tests depend on, it's identical(enc2native('\u4E00'), '\u4E00'); they should also pass on a pre-UCRT R version on Japanese Windows, or GNU/Linux with ja_JP.EUC-JP).
• 2266.* instead skips the tests, without trying to set a UTF-8 (or Latin-1/Latin-3/Latin-4/ISO-8859-15) locale.

Maybe we should be skipping tests more proactively?

• Want to test behaviour when the system encoding is UTF-8? Skip if !l10n_info()$`UTF-8`.
• Want to test printing of non-ASCII characters? Skip if !identical(enc2native('\uXXXX'), '\uXXXX').
  • Or spend extra effort to decide what should the code be doing with non-representable characters (truncate enc2native(toprint)?), then enc2native(output) before comparing it with capture.output().
• Want to test behaviour on Windows-1252 specifically? Skip if !identical(l10n_info()$codepage, 1252L).

[jangorecki]

Related PR: #7335
Skips tests that are now failing on our vanilla test job.

[MichaelChirico]

Related: tests 2298.* don't seem to be skipping as intended:

Test 2298.1 produced 1 errors but expected 0
Expected: 
Observed: Column 1 ['x'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
Test 2298.2 produced 1 errors but expected 0
Expected: 
Observed: Column 1 ['ä'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
Test 2298.3 ran without errors but failed check that x equals y:
> x = rbindlist(list(x, y), use.names = TRUE, fill = TRUE) 
    ä  ö  ü  x  y  z
1:  1  2  3 NA NA NA
2: NA NA NA  4  5  6
    [Key= Types=dou,dou,dou,dou,dou,dou Classes=num,num,num,num,num,num]
1:                                                                      
2:                                                                      
> y = data.table(ä = c(1, NA), ö = c(2, 4), ü = c(3, 5)) 
    ä ö ü  [Key= Types=dou,dou,dou Classes=num,num,num]
1:  1 2 3                                              
2: NA 4 5                                              
Different number of columns
Different column names
Test 2298.4 ran without errors but failed check that x equals y:
> x = rbindlist(list(y, x), use.names = TRUE, fill = TRUE) 
    x  y  z  ä  ö  ü
1:  4  5  6 NA NA NA
2: NA NA NA  1  2  3
    [Key= Types=dou,dou,dou,dou,dou,dou Classes=num,num,num,num,num,num]
1:                                                                      
2:                                                                      
> y = data.table(ö = c(4, 2), ü = c(5, 3), ä = c(NA, 1)) 
   ö ü  ä  [Key= Types=dou,dou,dou Classes=num,num,num]
1: 4 5 NA                                              
2: 2 3  1                                              
Different number of columns
Different column names

Mon Jul 27 22:27:25 2026  endian==little, sizeof(long double)==16, capabilities('long.double')==TRUE, longdouble.digits==64, sizeof(pointer)==8, TZ=='UTC', Sys.timezone()=='UTC', Sys.getlocale()=='LC_COLLATE=C;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252', l10n_info()=='MBCS=FALSE; UTF-8=FALSE; Latin-1=TRUE; codepage=1252; system.codepage=1252', getDTthreads()=='OpenMP version (_OPENMP)==201511; omp_get_num_procs()==4; R_DATATABLE_NUM_PROCS_PERCENT==unset (default 50); R_DATATABLE_NUM_THREADS==unset; R_DATATABLE_THROTTLE==unset (default 1024); omp_get_thread_limit()==2; omp_get_max_threads()==4; OMP_THREAD_LIMIT==2; OMP_NUM_THREADS==unset; RestoreAfterFork==true; data.table is using 2 threads with throttle==1024. See ?setDTthreads.', .libPaths()=='C:/Users/runneradmin/AppData/Local/Temp/RtmpqGmwcN/RLIBS_16507c6637cc','C:/R/library', zlibVersion()==1.2.11 ZLIB_VERSION==1.2.11
Error in test.data.table() : 
  4 errors out of 12775. Search tests/tests.Rraw for test numbers 2298.1, 2298.2, 2298.3, 2298.4. Duration: 00:01:03 elapsed (54.7s cpu).

https://github.com/Rdatatable/data.table/blob/5c91aadf7306fcb6d27fe97d4f6428e620b49a80/inst/tests/tests.Rraw#L20331

See https://github.com/Rdatatable/data.table/actions/runs/30308520863/job/90118454982?pr=7832.

It seems to be related to older versions of R: I only see the errors on Windows x {3.5,3.6,4.0,4.1}. It's somewhat surprising to see ä et al render correctly; on older R it doesn't (https://github.com/Rdatatable/data.table/actions/runs/30308520863/job/90118454967?pr=7832):

Test 2298.1 produced 1 errors but expected 0
  Expected: 
  Observed: Column 1 ['x'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
  Test 2298.2 produced 1 errors but expected 0
  Expected: 
  Observed: Column 1 ['�'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
  Test 2298.3 ran without errors but failed check that x equals y:
  > x = rbindlist(list(x, y), use.names = TRUE, fill = TRUE) 
      �  �  �  x  y  z
  1:  1  2  3 NA NA NA
  2: NA NA NA  4  5  6
      [Key= Types=dou,dou,dou,dou,dou,dou Classes=num,num,num,num,num,num]
  1:                                                                      
  2:                                                                      
  > y = data.table(� = c(1, NA), � = c(2, 4), � = c(3, 5)) 
      � � �  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:  1 2 3                                              
  2: NA 4 5                                              
  Different number of columns
  Different column names

[MichaelChirico]

Gemini diagnosis roughly makes sense from the evidence

• On Windows R ≤ 4.1 (MSVCRT / Windows-1252): R's parser on pre-UCRT Windows did not tag string literals in the 0x80–0xFF range as UTF-8; it tagged both "\u00e4" and enc2native("\u00e4") as Latin-1 byte 0xe4.
• Later iconv(..., from = "UTF-8", to = "latin1") sets Latin-1 column names on y, then rbindlist() attempts to hash and match UTF-8 column names against Latin-1 column names.
• translateCharUTF8 behavior was updated/improved in the UCRT effort for R 4.2.0

I'm trying to reproduce & then improve utf8_check() on 3.5.0 devcontainer with encoding ISO-8859-1 now.