Move public changelog registry ownership to the scrubber Lambda

[cotti]

TL;DR

The scrubber Lambda becomes the sole producer of the public bundle/{product}/registry.json manifests, recomputing them from what is actually in the public bucket (registry = f(state), never f(event)). It also maintains two new shallow per-tree change maps for cache opt-out. The changelog pool manifests (changelog/{org}/{repo}/{branch}/registry.json) are not reconciled — they stay client-authored pass-through until #3760 retires them.

Phase 1 of elastic/docs-eng-team#688. In the stack: base of #3760 (retire the client-side refresh) → #3761 (docs) / #3794 (migrate-from-web); sibling #3801 is the first consumer of the shallow maps (elastic/docs-eng-team#737).

Why

Today the public registry is written client-side against the private bucket and pass-through copied by the Lambda — a log of upload operations, not a description of the public bucket. Every known gap follows from that: ad-hoc uploads leave it stale (the 2026-07-22 invisible-bundle incident), deletions orphan entries, failed scrubs leave phantom entries, and the recorded ETag never matches what CDN consumers see.

How it works

• Events are triggers; state decides. The handler ignores the event type: each distinct key in a batch gets one object-level reconcile against the private bucket (present → scrub and copy; absent → delete the public copy), with post-write HEAD validation to catch mid-flight changes.
• Group reconcile (bundle tree only). Each touched bundle/{product}/ group gets its manifest rebuilt from the public listing: entries with matching ETags are reused, the rest recomputed from the scrubbed YAMLs (amends always recomputed — their target depends on the parent). Empty group → conditional delete (absent ≠ empty for consumers). Optimistic concurrency throughout (If-Match/If-None-Match: *, bounded retries, batch-item failure on exhaustion). A producer field versions the algorithm; any mismatch forces a full recompute, which is also how the legacy pass-through manifests get healed at cutover. Recorded ETags are now the public object's — finally usable by CDN consumers.
• Shallow maps. bundle/registry.json and changelog/registry.json, shaped {"<folder>": "<token>"} — one small conditional write per touched tree per batch. Consumers treat the token as an opaque change signal and bail out early on unchanged folders. Nothing consumes them in this PR (Changelog CDN fetchers: skip unchanged folders via shallow registry maps #3801 does).
• Convergence, not atomicity: any successfully processed event repairs all accumulated drift in the group and tree it touches. Consumers' listed-but-missing tolerance stays.
• Code shape: handler logic extracted from Program.cs into testable classes in Elastic.Changelog (ScrubberProcessor, BundleRegistryReconciler, ShallowRegistryReconciler, ChangelogContentScrubber); per-invocation EMF metrics (source-generated contract); FakeS3 test fake harvested from Add changelog registry state discovery and reconciliation commands #3717.

Two deliberate deviations from the review — please confirm

The review asked to drop the per-product manifests. Full rationale in this comment; short version:

1. bundle/{product}/registry.json stays, reconciler-owned. It is the one place consumers genuinely enumerate: the {changelog} directive and external CDN consumers list a product's bundles over HTTP (no listing API), and date-based targets (serverless) are not derivable client-side. Dropping it without a consumer-protocol redesign would reintroduce the 2026-07-22 incident. The pool tree's per-group manifests — what the review's argument targeted — are indeed no longer reconciled.
2. The shallow-map token is a digest over the folder's sorted file/ETag listing, not the last-touched file's ETag: deleting an older file wouldn't change the newest object's ETag, so deletions would be invisible to caches.

Deploy & rollback

Requires the IAM grants from elastic/docs-infra#360 (s3:GetObject/s3:ListBucket on the public bucket) at deploy time. The CLI upload path is untouched in this phase (#3760 retires it); rollback = revert the Lambda deploy.

Testing

Elastic.Changelog.Tests: 895/895 passing — object/group/shallow reconcile suites cover heal-missing-entry, drop-orphan, ETag reuse, amend recomputation, producer cutover, pagination and /-delimiter isolation, corrupt-manifest rebuild, empty-group conditional delete, conditional-write races and exhaustion, out-of-order and overlapping events, batch coalescing, and partial-batch failure marking. Scrubber Lambda publishes Native AOT with zero trim warnings. No CLI surface change (docs/cli-schema.json untouched); docs follow in #3761 once this shape settles.

History: original shape vs. the 2026-08-06 rework

The first iteration reconciled per-product/per-pool manifests in both trees and included group heal machinery for the pool tree. The review rejected the pool-tree inventory ("would just accumulate forever for little consumer value") and asked for shallow folder→ETag maps instead. The rework (7f4181c) scoped group reconciliation to the bundle tree, returned pool manifests to client-authored pass-through (mirrored verbatim, so #3760's cleanup deletes still propagate), and introduced the two shallow maps. A follow-up (b7fe622) renamed RegistryReconciler → BundleRegistryReconciler and fixed stale Registry.cs doc comments to match.

Part of elastic/docs-eng-team#688 (Phase 1).

[Mpdreamz]

This is overly defensive IMO, we use AOT trimming and source generated json contracts heavily. We should not fallback to handcrafted json emission.

These are emitted as cloudwatch metrics, should we push (some of) them as OTEL metrics too?

[Mpdreamz]

Request changes: drop per-product changelogs/<product>/registry.json

Pushing back on the core assumption in this stack — that we need (and should reconcile) a per-product / per-group changelogs/<product>/registry.json as a description of every changelog object under that prefix.

Why we don't need it

Release-note discovery is not "walk the registry and see what's there." The automated release discover moments (see elastic/docs-eng-team#698) always start from a list of PRs:

• Prestage → query for list of PRs
• GitHub release → already have a list of PRs
• Serverless release → query for list of PRs

Given that list, we can hit CloudFront and decide whether a checked-in changelog exists by whether the object returns 404. If it 404s, changelog bundle inlines a changelog for that PR into the bundle anyway.

The changelogs/ tree on S3 is only a mirror of explicitly checked-in changelogs in repositories. Those objects are not mandatory to exist at bundle time. A growing changelogs/<product>/registry.json (or changelogs/products/registry.json) would just accumulate forever for little consumer value — please remove that model and the reconcile/heal machinery built around it.

A lot of the code in this PR (group listing → entry-level reconcile → per-group manifest write/delete, producer/schema cutover heal, etc.) exists to keep that per-product registry correct. With the assumption dropped, that surface can shrink substantially.

What we do want instead

Keep (or introduce) two shallow folder-level registries on S3 for cache opt-out only:

• changelogs/registry.json
• bundles/registry.json (or bundle/registry.json — match existing key layout)

Shape:

{
  "<folder>": "<etag>"
}

Where <etag> is the ETag of the last-touched file under that folder. Caching layers can then bail out eagerly before diving into a per-folder / per-product bundle registry.

Please rework this PR around that thinner contract: scrubber remains the public-bucket writer for changelog objects, but it should maintain the shallow folder→etag map — not a full per-product object inventory.

[cotti]

Agreed on the thinner contract. I'll rework this PR so the scrubber maintains the shallow folder→ETag maps (changelogs/registry.json, bundles/registry.json) for cache opt-out instead of per-product object manifests, and drop the group reconcile/heal machinery that existed to keep those manifests correct. Converting to draft in the meantime.

[cotti]

Pushed b7fe622 with two clarity fixes, and I want to close an ambiguity my own 2026-08-06 reply created, so the deviation from the review is a recorded decision rather than a surprise at re-review time.

What my reply promised vs. what the rework does. I wrote I'd have the scrubber maintain the shallow maps "instead of per-product object manifests, and drop the group reconcile/heal machinery" — without qualifying that this referred to the changelogs/<product> pool tree the review contested. The rework does exactly that for pools: changelog/{org}/{repo}/{branch}/registry.json is back to client-authored pass-through, with no reconcile and no heal. But it keeps bundle/{product}/registry.json reconciler-owned, reading the review's "bail out eagerly before diving into a per-folder / per-product bundle registry" as acknowledging that catalog survives. It has to, today: CdnChangelogFetcher has no other way over HTTP to enumerate a product's bundles (no listing), select by target — including serverless dates that aren't derivable client-side — locate amends, or get cache-valid ETags. Dropping it without redesigning the consumer protocol would reintroduce exactly the 2026-07-22 invisible-bundle incident that motivated docs-eng-team#688. If you did mean the bundle catalog should go too, I'd rather scope that consumer redesign as its own issue than fold it in here.

On the surface that remains: group listing → entry reconcile → manifest write/delete → producer cutover heal all still exist, but scoped to the bundle tree only — the minimum for registry = f(state) on the one tree that keeps a catalog. All pool-side reconcile machinery is gone.

Second deviation, asking for an explicit ok: the shallow-map value is a digest over the folder's sorted file/ETag listing, not the last-touched file's ETag. Deleting an older file doesn't change the newest object's ETag, so deletions would be invisible to caches; consumers treat the value as an opaque change token (Reconcile_TokenChangesWhenAnyFileChangesOrIsDeleted covers it).

The commit itself: renamed RegistryReconciler → BundleRegistryReconciler so the narrowed scope is visible in the code, and fixed the stale Registry.cs doc comments — public bundle indexes are no longer "mirrored verbatim", and their recorded ETag is now the public object's (cache-valid), while client-authored manifests keep recording the private pre-scrub ETag. Also added a re-scope note to docs-eng-team#688, which still described reconciling both trees.