Snapshot add_ids' scope, retire dead registrations (#356)

Two findings on the shadow-mode id registry.

add_ids recorded one spec per attribute name over the *scope*, and the
replay re-walked that scope when something was signed — so the ids a
signature could resolve were the ones the tree carried by then, not the
ones xmlSecAddIDs had registered at the call. An element appended to the
scope afterwards became resolvable where the fast path had never
registered it; and the walk applied one name at a time across the whole
scope, where xmlSecAddIDs takes the elements in document order and the
names within each element. With <Y B="v"/><X A="v"/> and
add_ids(root, ['A', 'B']), the raw path signs Y and the shadow signed X:
both happily, over different content.

RecordIds now expands the scope where the registration happens: one spec
per element carrying one of the names, element by element in document
order, names in the caller's order. Specs are single-node throughout —
AddIdsBelow, the replay's subtree walk and the duplicate check's XPath
probe over a scope are gone — so the shadow registers exactly the
attributes the caller registered, never one that appeared later.

A registration also outlived its element. The registry holds the element
proxy (lxml refuses weak references) and released the slot only with the
whole document entry, so a document that registers and drops temporary
elements grew without bound, and kept claiming their id values:
registering a value a dropped element had carried raised "duplicated
id." where the fast path, whose id entry dies with the attribute,
accepts it. A slot is now retired when the registry holds the only
reference to the proxy, the tree it hangs in is not its document's, and
no other proxy remains anywhere in that tree — exactly when lxml frees
such a subtree and libxml2 drops the id entries of the attributes in it.
Vacated slots are reused, so neither the elements nor the list grow, and
the sweep runs before every registration and before every duplicate
check.

What this does not reproduce is lxml clearing its own id entry whenever
an element is *moved* — even within the one document, and for a whole
subtree when an ancestor moves (verified against lxml's id hash). The
registry cannot observe a move, so a #id the fast path stops resolving
keeps resolving under the shadow, to the element it was registered for
and never to another one. Documented with the other divergences.

Cost, worst case (2000 id-bearing elements): add_ids 0.000 -> 0.035 s,
the sign that follows 11 -> 14 ms, and 2000 register_id calls on that
document 0.54 -> 1.18 s, each call now sweeping 2000 registrations.

328 passed / 6 skipped on the mismatch build, also at
PYXMLSEC_TEST_ITERATIONS=50; 340 / 6 on the matched static wheel, plain
and with PYXMLSEC_FORCE_SHADOW=1. 10k sign+verify with an element
registered and dropped per iteration: RSS 26.2 -> 26.3 MiB, output
byte-identical throughout; 20k registration churn on one document
25.9 -> 25.9 MiB, where it was 27.2 -> 34.0 before. Three new tests fail
on the shadow path before the fix and pass on the raw path; two more
guard against retiring a registration that is still live.

Author: mxamin

developer.md

@@ -217,18 +217,35 @@ Under the shadow they record the id-attribute specs in a registry keyed by
 document identity (`RegisterId`, `RecordIds`), and every `BeginDoc` replays them onto its
 copy so that `#id` references resolve during sign/verify/decrypt. An entry
 keeps the registered elements themselves, so a spec is replayed at exactly
-the node it was registered for — that node alone for `register_id`, its
-subtree for `add_ids`, the scope `xmlSecAddIDs` walks. Registering every
-matching attribute of the copy instead would be unsafe, not merely generous:
-an unrelated element sharing the id value would claim it first and a `#id`
-reference could then resolve to content the caller never registered. lxml's
+the node it was registered for. Registering every matching attribute of the
+copy instead would be unsafe, not merely generous: an unrelated element
+sharing the id value would claim it first and a `#id` reference could then
+resolve to content the caller never registered.
+
+`add_ids` covers a whole scope, and that scope is walked **at the call**:
+`RecordIds` expands it into one spec per element carrying one of the names,
+element by element in document order and, within an element, in the order of
+the names — the registration `xmlSecAddIDs` makes, at the moment it makes it.
+Leaving the scope to be walked at the replay would register whatever the tree
+had become by the time something was signed: an element that grew the
+attribute, or joined the scope, after the call would resolve under the shadow
+where the fast path never registered it, and two elements claiming one value
+under different names would be ordered by name rather than by document
+order — a `#id` covering different content on the two paths. lxml's
 classes refuse weak references, so the entry keeps strong references (to the
 document and to those elements) instead: the key (the document's address) can
 then never go stale, and since every element proxy holds a reference to its
 document, a document whose reference count is exactly what the registry holds
 — and whose registered elements nothing else holds — is provably unreachable,
 so its entry, and the document with it, is dropped before the next
-registration. The registry therefore tracks the documents still in use and
+registration. Registrations are retired one by one on the same principle: a
+slot whose proxy the registry alone holds, hanging in a tree that is not its
+document's, cannot be reached again — lxml keeps an unlinked subtree only for
+as long as a proxy remains somewhere in it, and the fast path's id entry dies
+at exactly that moment too, when libxml2 frees the attribute. The slot is
+vacated (and reused by the next registration), so registering and dropping
+elements on a long-lived document neither grows the registry nor keeps their
+values claimed. The registry therefore tracks the documents still in use and
 never evicts a live one. The two bindings are the only places, together with
 encrypt_xml/decrypt's replacement bodies, that branch on `IsActive()`.
 
@@ -238,8 +255,8 @@ the same call: `xmlGetID(doc, value) != attr` — the test that raises
 under the shadow. What lxml's own parse declared (a DTD id attribute, an
 `xml:id`) is read back through XPath's `id()`, the one door into lxml's id
 hash that passes nothing but strings and elements; what earlier
-`register_id`/`add_ids` calls claimed is read from the registry, a subtree
-spec through one XPath over its scope.
+`register_id`/`add_ids` calls claimed is read from the registry, spec by
+spec.
 
 Both halves compare *attributes*, not elements: `<N xml:id="dup" ID="dup"/>`
 answers `N` to `id('dup')` whichever attribute is asked about, while the fast
@@ -292,6 +309,13 @@ All invisible to the documented API:
   list of ids), so such a value is compared against the registry alone; a
   registration whose element has since been adopted into another document
   claims nothing, as at replay;
+- a registration follows the element it was made for. lxml drops its own id
+  entry whenever an element is *moved* — even within the one document, and
+  for a whole subtree when an ancestor moves — which the registry cannot
+  observe, so a `#id` the fast path stops resolving after such a move keeps
+  resolving under the shadow. It resolves to the registered element, never to
+  another one: the shadow registers exactly the attributes the caller
+  registered;
 - `encrypt_xml` encrypts a *copy* of the template, so the caller's template
   proxy is not the returned element; a template attached in the target's own
   document is unlinked afterwards (keeping its tail text where libxml2 would