Join the crawler to its storage, and make a measured hour tell you which of three it was - #7
Merged
Merged
Conversation
…ich of three it was
Every piece existed and nothing connected them: IProbe produced a ProbeResult, the
three writers consumed one, the scheduler and the registry knew when to dial, and no
code path ran a probe into a database. MUI.Crawler is that path.
ProbeIngestor is spec §6.5's seam and holds the four rules that matter most here. A
probe that answered with a count writes one presence row — including a measured zero,
which is a filled cell and a real fact about a game. `who` outranks `mssp` and the
choice is made in PresenceChoice *before* the writer, because presence_sample is keyed
(game_id, at) and there is nowhere to keep both. A probe that answered and could not be
counted writes count = NULL with a reason: never a zero, and never nothing, because
nothing is indistinguishable from not having probed and renders as downtime. A probe
that failed writes an availability transition, no presence row, and no field
confirmations — a dial that never got in confirmed nothing, and bumping
last_confirmed_at would record our crawler having run as the game having answered. A
scope refusal cannot reach the ingestor at all: ProbeOutcome has two members and both
mean the socket was opened, so §7.2's "a refusal writes no availability sample" is
satisfied structurally rather than by a check somebody has to remember.
The measured half of the capability matrix is written `true` only where a protocol was
observed, and the one honest negative is MSSP — because MSSP is the one option this
probe asks for. Writing `false` for the rest would be a lie we can already measure:
alteraeon.com:23 plainly implements MSDP, GMCP, MXP and MCCP, and one probe of it
yields an offered set of exactly { MSSP }. That is our instrumentation, not their game.
CrawlerService is a BackgroundService behind a Postgres advisory lock, so N web
replicas run exactly one crawler. The lease is session-level and owns its connection —
and releases the lock explicitly on the way out, because Npgsql pools and a disposed
connection returns to the pool with the session, and therefore the lock, still alive. A
replica standing down would have stopped the crawl for the whole deployment until its
process exited. CrawlCycle bounds concurrency with a semaphore, serialises per host,
honours both rate floors, and hard-bounds every probe by timeout and token on top of
whatever the probe promises: the crawler shares a process with the web tier, so a
wedged probe starving request threads is a correctness bug. CRAWL DELAY is read off
MSSP with -1 kept as "no preference" and composed by ProbeSchedule, which already knows
that politeness outranks the weekly floor.
0006_crawl_registry.sql adds the registry, the referral graph and the review queue —
the discovery half of the schema, which nothing had written. crawl_target has no
retirement column and no way to acquire one.
Verified against four live games into a real PostgreSQL, twice. All three rungs of the
count ladder fired: WHO on mush.pennmush.org (13) and eldertaleonline.com (a measured
0), MSSP on alteraeon.com (46), and the connect screen on aardmud.org (205), which
supports no MSSP and answers no pre-login WHO and which no MSSP-only directory can
count at all. eldertaleonline.com publishes NAME "PennMUSH" — an unedited codebase
default — and was listed under its own HOSTNAME rather than as one more game called
PennMUSH, which is the auto-merge failure §7.3 warns about, met in the wild.
That run also found a defect no fixture could have. Aardwolf's connect screen states
its own live player count, so reconciling connect_screen wrote "changed from ####… to
####…" on every probe, for ever, at one row per game — the exact cost §5.1 exists to
avoid. The banner is upserted now, beside the fingerprint it is the other half of. A
redesign is still noticed; that is what banner_hash is for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 41 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (37)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every piece existed and nothing connected them:
IProbe/TelnetProbeproduced aProbeResult, the three writers consumed one,MUI.Discoveryknew when to dial and who a probe was, and no code path ran a probe into a database.MUI.Crawleris that path, andAddMuiCrawler(…)is the one call that wires it into the deployable.MUI.Web/Program.csis untouched.ProbeIngestor— spec §6.5's seamThe four rules, each pinned by a test named after it:
AProbeThatAnsweredWithACountWritesThatCount,AMeasuredZeroIsAStoredCountAndNotAnAbsencewhooutranksmssp, and the choice is made before the writerWhoOutranksMsspAndTheChoiceIsMadeBeforeTheWriter,AZeroFromWhoStillBeatsANonZeroFromMsspcount = NULL+ a reason. Never a zero, never nothingAnAnsweredProbeWithNoCountObtainableWritesANullCountAndAReasonAndNeverAZeroAFailedProbeWritesAnAvailabilityTransitionAndNoPresenceRowAtAllAScopeRefusalCannotBeExpressedAsAProbeOutcome,ADialWeRefusedIsNotInAnyGamesRecordpresence_sampleis keyed(game_id, at), so the ladder —who→mssp→banner— is applied inPresenceChoicebeforePresenceWriteris called. A failed probe also confirms no fields: a dial that never got in confirmed nothing, and bumpinglast_confirmed_atwould record our crawler having run as the game having answered.The measured half of the capability matrix is written
trueonly where a protocol was observed, and the one honest negative is MSSP — because MSSP is the one option this probe asks for (IAC DO 70on every dial), so silence is an answer. Writingfalsefor the rest would be a lie we can already measure:alteraeon.com:23plainly implements MSDP, GMCP, MXP and MCCP, and one probe of it yields an offered set of exactly{ MSSP }. That is our instrumentation, not their game.CrawlerService— §12A
BackgroundServicebehind a Postgres advisory lock, so N web replicas run exactly one crawler.CrawlCyclebounds concurrency with a semaphore, serialises per host throughHostGate, honours bothCrawlRateLimiterfloors, and hard-bounds every probe by timeout andCancellationTokenon top of whatever the probe promises — the crawler shares a process with the web tier, so this is correctness rather than hygiene.CRAWL DELAYis read off MSSP with-1kept as "no preference" and composed byProbeSchedule, which already knows politeness outranks the weekly floor.The lease releases the lock explicitly on the way out, and that is not belt-and-braces: Npgsql pools, so a disposed connection returns to the pool with its session — and therefore its session-level lock — still alive. A replica standing down would have stopped the crawl for the whole deployment until its process exited.
ReleasingTheLeaseLetsTheNextReplicaTakeItfailed exactly that way before the fix.mui-crawlmui-probeprints what one server said;mui-crawlprints what a cycle wrote, read back out of the database rather than tallied in memory. It builds the same graphAddMuiCrawlerbuilds.Verified against live games, twice
Four real servers into a real PostgreSQL 17. All three rungs of the ladder fired:
Aardwolf supports no MSSP and answers no pre-login
WHO, so no MSSP-only directory can count it at all; the connect screen can.eldertaleonline.compublishesNAME "PennMUSH"— an unedited codebase default — and was listed under its ownHOSTNAMErather than as one more game called PennMUSH, which is §7.3's auto-merge failure met in the wild. A second cycle wrote four more presence rows, zero new fields, zero field changes and zero availability transitions, which is the shape §5.1 promises.That run found a defect no fixture could have. Aardwolf's connect screen states its own live player count, so reconciling
connect_screenwrotechanged from ####… to ####…on every probe, for ever, at one row per game — the exact cost §5.1 exists to avoid, burying every real event under an unreadable diff. The banner is upserted now, beside the fingerprint it is the other half of; a redesign is still noticed, because that is whatbanner_hashis for.New schema
0006_crawl_registry.sqladdscrawl_target,referral_edgeandduplicate_review— the discovery half of the schema, which nothing had written — plus the folded functional indexIGameFieldIndex's case-insensitive predicate needs.crawl_targethas no retirement column and no way to acquire one, held by a test that readsinformation_schemaas well as the interface.Worth a reviewer's eye
MUI.Crawler.Cliis a project the plan did not name. The brief allowed "amui-crawlentry point or an option onMUI.Probe"; a new project keepssrc/MUI.Probeuntouched.MsspCrawlDelaybelongs besideMsspReferralsinMUI.Discovery, which is where CLAUDE.md files this class of reading. It is inMUI.Crawlerto respect the scope this branch was given; move it when scopes reconverge.ICrawlTargetRepository.RecordAttemptAsynchas two arms and a refusal needs a third. A scope refusal is not the host failing, so it must not accrue backoff — but there is no "leave the failure count alone" arm, so the refusal path takessucceeded: trueand clears a count nothing will act on, since a refused target is never dialled.IGameStorehas no rename. A game'sgame.nameis set when it is listed; a laterNAMEchange moves thegame_fieldrow and the change feed, and thegamerow keeps its original. §5.7's deliberate slug stability does not extend to the display name.ArchiveSweeper.SweepAsyncstill has no caller. Un-archiving is wired (one successful probe restores a game, immediately); archiving needs a once-a-day cadence decision that did not belong in this branch.tests/MUI.Crawler.Tests/Support/PostgresFixture.csis a copy of the Catalog suite's. Referencing that project would pull its source-generated tests in and run the storage suite twice; the right fix is a small shared testing project.Five suites, all green, warning-free: Catalog 133, Crawl 125, Crawler 76, Discovery 182, Web 71.
🤖 Generated with Claude Code