Rewrite changelog registry docs for scrubber Lambda ownership - #3761
Draft
cotti wants to merge 5 commits into
Draft
Rewrite changelog registry docs for scrubber Lambda ownership#3761cotti wants to merge 5 commits into
cotti wants to merge 5 commits into
Conversation
Phase 1 of elastic/docs-eng-team#688. The public registry.json was a log of upload operations (client-written, pass-through copied); every known consistency gap followed from that. The scrubber Lambda now derives it from the public bucket's actual state: registry = f(state), never f(event). - Extract the Lambda's top-level handler logic into testable classes in Elastic.Changelog: ScrubberProcessor (batch coalescing by key and group, object-level reconcile with post-write source validation) and RegistryReconciler (delimited/paginated group listing, ETag reuse with amends always recomputed, semantic idempotence, conditional PUT/DELETE with bounded jittered retries on 412/409, newer-schema refusal). Program.cs is now a thin adapter. - Retire the registry pass-through in the same deploy: registry-key events only schedule a group reconcile, so client-authored JSON no longer reaches the public bucket uninspected. - Add a producer (algorithm version) field to the manifest; a mismatch — including legacy pass-through manifests — forces a full metadata recompute and a write even when entries are identical. - Emit per-invocation reconcile metrics as CloudWatch EMF (the Phase 0 observability item that could only land with the reconciler). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the Lambda-owned public registry: - `changelog registry reconcile` plans groups (one scope, or the union of both buckets so orphan public groups are covered) and sends one versioned, discriminated reconcile message per group to the scrubber queue — {kind, version, scope, group, correlation_id}, validated through ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the public bucket's single writer. --dry-run prints the plan; the non-dry-run path asks for confirmation (--yes for CI). Each run stamps one correlation id and prints a ledger line per group. - On a reconcile message the Lambda performs a full group heal: object-level reconcile over the union of both buckets' listings (copy what's live, delete what isn't), then the group reconcile — recovering lost/DLQ-expired scrub events. Requires the new optional PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to the DLQ where the Phase 0 alarm surfaces them. - `changelog registry verify` is the read-only sibling and cutover gate: compares each public manifest against what a reconcile would write (same listing spec and entry rules by construction) and reports divergence as missing/stale/corrupt/object-divergent, with unsupported schemas reported distinctly. - Fix the manifest ETag wire format: the snake_case policy serialized the producer-side field as "e_tag" while consumers and the documented format read "etag" — recorded ETags were invisible to every consumer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same suppression RegistryBuilderTests carries: xUnit owns the test class lifetime and TestDiagnosticsCollector needs no disposal in these tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scrubber Lambda is the sole producer of the public registry.json, reconciled from public bucket state on the S3 events every upload already emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects only; RegistryBuilder and the private-manifest write path are removed, and the amend end-to-end test exercises RegistryReconciler instead.
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
from
August 4, 2026 01:50
4cefbe5 to
cc65ecf
Compare
The registry docs still described the retired model: client-side refresh, registry pass-through, pre-scrub ETags, a 1 h CloudFront TTL (caching is disabled), and a refresh "skipped for --artifact-type changelog". Documents the reconciler as sole producer, the public-object ETag, convergence semantics, absent-vs-empty manifests, the reconcile message contract, and the registry reconcile/verify operator commands (docs-eng-team#688 Phase 4).
cotti
force-pushed
the
changelog-registry-docs
branch
from
August 4, 2026 01:50
7af75b0 to
9bebf38
Compare
Contributor
Author
|
Converting to draft: these docs describe the per-product registry model that's being dropped per the review on #3738. I'll rewrite them once the thin folder→ETag registry shape lands there. |
cotti
marked this pull request as draft
August 6, 2026 13:07
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
2 times, most recently
from
August 10, 2026 16:21
95b64ed to
5695678
Compare
Mpdreamz
approved these changes
Aug 10, 2026
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
2 times, most recently
from
August 11, 2026 12:54
ba57fc6 to
31d19e6
Compare
Base automatically changed from
changelog-retire-client-registry-refresh
to
main
August 11, 2026 14:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Phase 4 of elastic/docs-eng-team#688, stacked on #3760. After Phases 1–3 the registry docs described a retired system: client-side refresh, registry pass-through, an ETag "documented as useless", a CloudFront "1 h TTL" that hasn't existed since caching was disabled, and a claim that the refresh was "skipped for
--artifact-type changelog".What
docs/development/changelog-bundle-registry.mdnow documents the end state: the scrubber Lambda'sRegistryReconcileras the manifest's sole producer (registry = f(state)), theproducerfield,bundles[].etagas the public object's ETag (finally usable for CDN cache revalidation), consistency stated as convergence with the consumer's listed-but-missing tolerance kept, the deliberate absent ≠ empty manifest semantics, and theregistry reconcile/registry verifyoperator tooling with the reconcile message contract. The scrubber README's event-handling section andcmd-upload.md's registry paragraph are aligned with the same model.Part of elastic/docs-eng-team#688 (Phase 4).