Seed the catalogue from other people's directories, and keep measured apart from recorded - #8
Merged
Merged
Conversation
…r by tier MUI.Import reads existing MU* directories (spec §7.6) and splits what it finds by whether the source measured or merely recorded. - The two tiers, and the rule enforced by construction rather than by care. AssertedHistorySink takes no constructor parameters and holds no store, no writer and no clock, so "no history, no presence, no grace" is a fact about the type. It still counts what it refused, because a silent refusal reads identically to a source that offered nothing. - imported_measured history is written through IImportedAvailabilityWriter and never IAvailabilityStore.OpenAsync, whose origin column DEFAULTs to 'first_party' — getting that wrong would credit a third party's decade at full weight, silently, and only a real database can tell you. - import_provenance (migration 0100) carries §7.6's other requirement: the originating site and the import date on every value. It is NOT on the grace path; the half weight is availability_interval.origin, computed once in ArchiveSweeper. - Etiquette is configuration and refusals, not prose: bulk export ▸ documented API ▸ scrape, robots.txt read before the first content fetch, rate limit on an injected TimeProvider that adopts a longer Crawl-delay and never a shorter one, and an identifying User-Agent without which nothing fetches at all. A scrape-only source is refused outright until a human has written to its maintainer. The about page's attribution list is derived from the registry, so a source that is read and not credited is unreachable. - An import may not mint a game. Identity is one signal — an endpoint we already measured — so below it a listing becomes a CrawlTarget and stops, which is spec §7.2 arrived at by having no mechanism to break it. Two sources, behind IDirectorySource so a third is a class: - TinTin++ MSSP Mud Crawler, imported_measured, read as a bulk export. Its fixed-width box table is parsed with the markup's colours and links kept: the website URL exists only in an href, and the crawler's own bright-red "invalid data" verdict exists only in a colour — a value its author distrusted is not imported. Capabilities land as capability.x.declared, never .measured: TinTin measured that the server answered, the game declared what was in the answer. - MudStats, imported_measured, a scrape and therefore gated. Values are read by element id, never by their label — the page puts the two in separate elements, so the obvious regex captures the gap between two closing tags. Neither yields availability spans: both pages are snapshots, and a span invented around an instant credits grace nobody measured. Player counts are imported only when the source dates them — TinTin's generation stamp, MudStats' "(9 minutes ago)". 88 tests, six suites green, and one end-to-end import into a real PostgreSQL through Testcontainers. Verified once against the live page: 115 hosts, 144 endpoints, 114 dated counts (tools/live-tintin-import). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (50)
Comment |
Both PRs added a project, a package and a CI step, so the three shared registration files collided. The resolution is additive on every hunk -- the crawler needs hosting abstractions and a console sink, the importer needs the DI abstractions it shares with it, and both suites want a real Postgres on the Linux leg for the same reason the Catalog suite does. Six suites green after resolving: 711 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Plan 04 — backfill importers (spec §7.6):
src/MUI.Import, a new class library thatreads the existing MU* directories so the catalogue has a day-one seed, and splits everything it
finds by whether the source measured or merely recorded.
Composition is not wired here —
MUI.Web/Program.csis owned by another branch.AddMuiImporters(…)is exposed and ready to call.
The rule, enforced rather than remembered
imported_measuredimported_assertedpresence_sample/availability_intervalAssertedHistorySinktakes no constructor parameters and holds no store, no writer and no clock,so the asserted tier's "no history, no presence, no grace" is a fact about the type rather than a
rule at eight call sites. It still counts what it refused — a silent refusal reads identically to a
source that had nothing to offer, and only one of those is worth an email.
MeasuredHistorySinkwrites throughIImportedAvailabilityWriter, neverIAvailabilityStore.OpenAsync, whoseorigincolumn defaults tofirst_party. Reaching for thewrong write path would credit a third party's decade at full weight, and it would look correct in
every fake — so the stored value is asserted against a real PostgreSQL as well as in memory.
import_provenance(migration0100) carries §7.6's other requirement, the originating site andthe import date on every value, for the provenance chip and the about page. It is not on the grace
path; a second calculator reading it would count the same history twice.
An import may not mint a game. Identity is one signal — an endpoint we already measured — so below it
a listing becomes a
CrawlTargetand stops. That is §7.2 arrived at by having no mechanism to breakit, and the seeded target never mentions
IsOperatorSeed, so a stranger's list gets the guardeddefault without anybody having to remember.
Etiquette as code
ScrapeUriisgenuinely unreachable while a better route exists — not merely unreached by a correctly written
importer.
robots.txtbefore the first content fetch, and the gate is shut until it is adopted. Longestmatch,
AllowoverDisallow,*/$, and a group naming us beats the wildcard.TimeProvider. A longerCrawl-delayreplaces our minimum; a shorterone does not licence us to go faster. (
tintin.mudhalla.netsaysCrawl-delay: 0. It does not meanwhat a naive reader would do with it.)
MUIndex-Importer/1.0 (+https://muindex.org/crawler)on every request, and an anonymous useragent is refused before any route is even considered.
with that gate closed, and the run says so out loud instead of running.
The two sources, behind
IDirectorySourceTinTin++ MSSP Mud Crawler —
imported_measured, read as a bulk export (one page, one request).Its fixed-width box table is parsed with the markup's colours and links kept, because both carry
data the text does not: a website URL exists only in an
href(the cell's text is the game's nameagain), and the crawler's own bright-red invalid data verdict exists only in a colour. A value its
author distrusted is not imported. Capabilities land as
capability.x.declaredand never.measured— TinTin measured that the server answered; the game declared what was in the answer.
MudStats —
imported_measured, a scrape and therefore gated. Values are read by element id,never by label: the page puts a label and its value in separate elements, so a regex anchored on
Address:captures the whitespace between two closing tags and every field comes back empty.Neither yields availability spans. Both pages are snapshots: they say the host answered at an instant
and nothing about for how long, and a span invented around that credits archive grace nobody measured.
Player counts are imported only where the source dates them — TinTin's
generated onstamp,MudStats'
(9 minutes ago)— because dating somebody else's measurement to the moment we read thepage puts a fabricated timestamp into the day × hour heatmap.
Verification
Six suites green and a warning-free build. 88 tests in
MUI.Import.Tests, six of them a realend-to-end import into PostgreSQL 17 via Testcontainers (they skip by name where no container runtime
exists; CI's Linux leg demands one). Every byte any test parses comes from a committed fixture — there
is no
HttpClientin the suite that can reach the internet.HistoryTierTests.AnAssertedSourceStuffedWithHistoryWritesNoneOfItAndIsCountedForTryingis the pinfor the two-tier rule, with
TheAssertedSinkHoldsNothingItCouldWriteWithholding the enforcementmechanism in place by reflection and
ImportAgainstPostgresTests.AnImportedReachableSpanIsStoredWithTheOriginThatEarnsItHalfWeightpinning the column the half weight is actually computed from.
Run live once, politely (
tools/live-tintin-import, in the solution so it cannot rot, never runby CI): 115 hosts found, 144 crawl targets, 114 dated player counts, 88 codebases, 69 websites.
Against a real PostgreSQL with one matching game seeded, that game received 25 fields, 4 endpoints and
1 presence row, every one labelled
imported_measured, plus 30 provenance rows.Notes for review
at
/mudlist/mudlist.tsvand filed itimported_asserted. The real artefact is the MSSP crawler'sown HTML listing, and a site that connects and negotiates MSSP has measured. Tier corrected.
MUI.Backfill→src/MUI.Import, and the plan'sDirectorySource(the HTTP surface) is
DirectoryFetcher, freeingIDirectorySourcefor the source abstraction.MUI.Catalog,MUI.CrawlandMUI.Discoveryare untouched. The only file added outside the twonew projects is the migration.
🤖 Generated with Claude Code