idn: hand libidn2 the same flags on both sides

The client path called idn2_lookup_ul() without IDN2_NFC_INPUT while the
daemon path passed it to idn2_to_ascii_8z(). Both normalize either way --
idn2_lookup_ul() ors the flag in itself, and TR46 normalizes as it maps --
but there is no reason for the two calls to read differently, so pass one set
of flags from one place. The flag asks libidn2 to normalize the label rather
than promising that it already is: it gates the u32_normalize() call, and
without it a decomposed label comes back IDN2_NOT_NFC.

Adds composed/decomposed cases to testsuite/idn, which sees the exact host
name rsync hands out, and a decomposed hosts allow token to
testsuite/daemon-access-idn.

Author: cicku

rsync.1.md

@@ -416,9 +416,10 @@ If rsync was built with IDN support (look for "IDN" in `rsync --version`), the
 daemon host may contain non-ASCII characters: those labels are converted to
 their IDNA A-label ("Punycode") form before the name is looked up.  The name is
 read using your locale's character encoding, so be sure your locale is set
-correctly.  Only the non-ASCII labels change, so an address literal, a name you
-punycoded yourself, and a name that is not a valid IDN are all looked up just
-as you typed them.
+correctly.  A name typed with combining characters is normalized on the way, so
+it is looked up the same as its precomposed spelling.  Only the non-ASCII
+labels change, so an address literal, a name you punycoded yourself, and a name
+that is not a valid IDN are all looked up just as you typed them.
 
 This applies to the host of a daemon connection only.  The host of a plain
 remote-shell transfer (the single-colon syntax) is passed to your remote-shell

rsyncd.conf.5.md

@@ -857,11 +857,13 @@ in the values of parameters.  See that section for details.
     a hostname pattern may contain non-ASCII characters: this file is read as
     UTF-8, and each non-ASCII label is converted to its IDNA A-label
     ("Punycode") form before matching, since the name the daemon has for a
-    client always reaches it from DNS as ASCII.  Only those labels change, so
-    an address, a mask, an already-punycoded name, and the wildcard characters
-    are all matched exactly as written.  A pattern that cannot be converted
-    that way is matched as it stands, and thus matches nothing.  That includes
-    a pattern whose conversion would have introduced a character it was not
+    client always reaches it from DNS as ASCII.  A pattern written with
+    combining characters is normalized on the way, so it matches the same as
+    its precomposed spelling.  Only those labels change, so an address, a
+    mask, an already-punycoded name, and the wildcard characters are all
+    matched exactly as written.  A pattern that cannot be converted that way
+    is matched as it stands, and thus matches nothing.  That includes a
+    pattern whose conversion would have introduced a character it was not
     written with, such as the U+FF0A FULLWIDTH ASTERISK that the IDNA mapping
     turns into a "*".
 

testsuite/daemon-access-idn_test.py

@@ -114,6 +114,12 @@ def fullwidth(name):
     # stay denied, and neither may be mistaken for a wildcard.
     ('idn-other',     {'hosts allow': 'čičku.example'}),
     ('idn-puny',      {'hosts allow': 'xn--iku-eqab.example'}),
+    # idn-other's token respelled with combining carons: an equivalent name
+    # under Unicode, so it has to be treated the same way.  (A decomposed
+    # token that *matches* isn't constructible here -- the peer name comes
+    # from DNS and is ASCII -- so idn_test checks the two spellings convert
+    # alike, and this checks the daemon agrees they don't match.)
+    ('idn-nfd',       {'hosts allow': 'c\u030ci' 'c\u030cku.example'}),
     # Tokens whose IDNA mapping yields ASCII the author never wrote.  Left
     # unconverted they match nothing; converted blindly, the first two would
     # allow every host and the third would read as an address/mask.
@@ -154,6 +160,7 @@ def denied(mod, why):
 denied('idn-deny', "hosts deny sees the folded token too")
 denied('idn-other', "a different IDN must not match the peer")
 denied('idn-puny', "an A-label for a different host must not match the peer")
+denied('idn-nfd', "a decomposed spelling of that name must not match either")
 denied('wide-star', "U+FF0A must not become a '*' that allows every host")
 denied('wide-star-dom', "U+FF0A must not become a wildcard label")
 denied('wide-mask', "U+FF0F must not become an address/mask separator")

testsuite/idn_test.py

@@ -3,7 +3,8 @@
 # its IDNA A-label (Punycode) form, and that it leaves an ASCII host name
 # alone.  Only the labels that are not ASCII get rewritten, so an address
 # literal, an already-punycoded name, and a name that isn't a valid IDN all
-# reach the resolver as typed.
+# reach the resolver as typed.  A name typed with combining marks is normalized
+# on the way, so it converts the same as its precomposed spelling.
 #
 # Two daemon connection methods carry the host name out of rsync, so both are
 # checked:
@@ -55,6 +56,10 @@ def find_utf8_locale():
 
 idn_host = "\u010ci\u010dku.example"
 ascii_host = "xn--iku-eqab.example"
+# The same name with each caron letter spelled as a plain "c" plus a combining
+# caron (U+030C).  Unicode calls the two spellings equivalent, so both have to
+# come out as the same A-label; libidn2 is what normalizes them.
+nfd_host = "c\u030ci" "c\u030cku.example"
 
 env = os.environ.copy()
 env['LC_ALL'] = utf8_locale
@@ -103,6 +108,9 @@ def check_rsh(url_host, want, what):
 # A U-label becomes its A-label, case-folded by the IDNA mapping.  An ASCII
 # label is handed on byte for byte, case included, since DNS doesn't care.
 check_rsh(idn_host, ascii_host, "a Unicode host")
+check_rsh(nfd_host, ascii_host, "a decomposed Unicode host")
+check_rsh("C\u030cI" "C\u030cKU.Example", "xn--iku-eqab.Example",
+          "a decomposed mixed-case Unicode host")
 check_rsh("ČIČKU.Example", "xn--iku-eqab.Example",
           "a mixed-case Unicode host")
 check_rsh(ascii_host, ascii_host, "an already-punycoded host")

util1.c

@@ -986,14 +986,18 @@ int idn_to_ascii(const char *name, int from_locale, char *buf, size_t buflen)
 			memcpy(buf + len, lab, lablen);
 			len += lablen;
 		} else {
+			/* IDN2_NFC_INPUT has libidn2 normalize the label, so a name
+			 * typed with combining marks folds to the same A-label as
+			 * its composed spelling.  IDN2_NONTRANSITIONAL asks for the
+			 * TR46 processing that everything else does these days. */
+			int flags = IDN2_NFC_INPUT | IDN2_NONTRANSITIONAL;
 			int rc;
 			if (lablen >= sizeof label)
 				return 0;
 			memcpy(label, lab, lablen);
 			label[lablen] = '\0';
-			rc = from_locale
-			   ? idn2_lookup_ul(label, &idn, IDN2_NONTRANSITIONAL)
-			   : idn2_to_ascii_8z(label, &idn, IDN2_NFC_INPUT | IDN2_NONTRANSITIONAL);
+			rc = from_locale ? idn2_lookup_ul(label, &idn, flags)
+					 : idn2_to_ascii_8z(label, &idn, flags);
 			if (rc != IDN2_OK)
 				return 0;
 			alen = strlen(idn);