Serve the real catalogue, and walk a referral graph in the wild

.gitignore

@@ -4,3 +4,8 @@ obj/
 .idea/
 *.user
 appsettings.*.local.json
+
+# Backfill artifacts. The importer is code and is committed; what it harvests is not (spec §7.6).
+imports/
+*.import.json
+*.import.ndjson

CLAUDE.md

@@ -29,17 +29,26 @@ poisoning the next command we sent. **Probe something before you theorise about
    disagreement is the interesting fact and must not be hidden. This is why `GameField` is keyed
    `(game, field, **source**)` — one row per field cannot hold both sides of a disagreement.
 2. **An hour has three states, not two.** Probed-with-a-count is a filled cell (*including* a
-   measured zero — we got in and nobody was there). Probed-but-uncountable is hatched. Not reachable
-   is empty. Collapsing the middle case into either neighbour is the worst bug this codebase could
-   ship: a game whose `DOING` header is customised past our parser would render as permanently dark
-   while running perfectly well.
+   measured zero — we got in and nobody was there). Probed-but-uncountable is hatched. **Not
+   measured** is empty. Collapsing the middle case into either neighbour is the worst bug this
+   codebase could ship: a game whose `DOING` header is customised past our parser would render as
+   permanently dark while running perfectly well. The third state is *not measured*, never *not
+   reachable* — a failed probe writes no presence row at all (`PresenceWriter`), so an empty cell
+   covers an hour we could not reach and an hour we never probed alike, and it may not name a cause.
+   It said "the game was not reachable" until a real crawl put it beside a game measured once and
+   found perfectly reachable, described as down for 167 hours of the week. Reachability is the
+   strip's question and comes from intervals, which can tell the two apart.
 3. **Nothing is ever deleted.** Archiving removes a game from the default listing, the rankings and
    the "active today" figure — and from nothing else. Its page, URL, history and change feed survive,
    it keeps being probed forever, and one successful probe restores it.
 4. **Parsers never fabricate.** An unreadable `WHO` yields unknown, never zero.
 5. **Never record a decision of ours as a measurement of theirs.** A scope refusal is not downtime.
    An unparseable `WHO` is not zero players. Our security policy and our parser's limits must never
-   appear in a game's public record as facts about the game.
+   appear in a game's public record as facts about the game. **The same rule reads forward into every
+   sentence a surface writes**: a percentage whose denominator is what we observed may not be
+   presented as a fraction of a 90-day window, and a graphic's empty state may not be given a cause.
+   Both shipped, and both were found the first time the site rendered a real crawl instead of the
+   fixture — a fixture is written by someone who already knows what each panel is supposed to say.
 
 ## Vocabulary
 
@@ -62,6 +71,16 @@ our host is unreachable and perfectly alive.
   A refusal happens *before* a probe exists, and `FailureCause.Refused` already means the far end
   sent an RST — a real measurement of a real host. Conflating them is unrecoverable downstream. The
   guard belongs to whatever owns the dial.
+- **Commit a harvested dataset.** The backfill importer is code and belongs here; its output does
+  not. No mirrored listing file, no seed list of real hosts scraped from a directory, no snapshot of
+  anyone's catalogue. The import is a *one-time* operation an operator runs once against the one
+  deployment — not a startup step, not a scheduled job, not something a clone reproduces (spec §7.6).
+  Test fixtures are exempt: a handful of hand-written rows exercising a parser is a test input, and a
+  copy of a third party's catalogue is not, whatever it is named or how small it starts.
+- **Ship invented data without saying so on the page.** `MUI.Web` reads Postgres when
+  `MUI_POSTGRES` (or `ConnectionStrings:MUIndex`) is set. With neither it still starts, on the
+  fixture — and then every page carries the demo banner, because a reader who cannot tell a
+  measurement from a fixture is being misled by exactly the mechanism this project exists to replace.
 - Publish an absolute "how many people play MU\*" figure. Shares ship; totals do not (spec §15.7).
 
 ## Security: the gate is on the address, not the name

README.md

@@ -6,10 +6,15 @@ is that **its data is measured rather than asserted**.
 Every fact on a game's page carries how it was obtained and how old it is. The catalogue is a
 by-product of continuous measurement, not a form somebody filled in once.
 
-> **Status:** the truth engine works. A probe reads real servers, storage keeps what it measured with
-> the three-state discipline intact, discovery walks referrals behind a resolved-address gate, and the
-> site renders both a graphical and a plain surface from one set of view models. 511 tests, Postgres
-> exercised in CI. Not yet joined end to end, and not yet deployed.
+> **Status: joined end to end.** `mui-crawl` probes live servers into PostgreSQL and the site renders
+> that database — sixteen real games, measured, in [`docs/screenshots/`](docs/screenshots/). Referral
+> discovery has been walked in the wild: two of 141 live servers publish `REFERRAL`, and the endpoints
+> they name became depth-1 targets and were probed without being seeded. Not yet deployed.
+>
+> A run against real servers immediately found three sentences the fixture could not: a percentage
+> claiming ninety days of evidence from one probe, an empty heatmap cell asserting downtime it had not
+> measured, and an internal digest presented as something a game said about itself. That is the
+> argument for measuring rather than asserting, turned on the site itself.
 
 Short form **MUI**, which is also the assembly prefix.
 
@@ -83,7 +88,10 @@ One telnet connection per probe, yielding four independent layers — not a fall
    not here — and every game we found that answers it answers option 70 too.
 
 Discovery walks the MSSP `REFERRAL` graph, honours `CRAWL DELAY`, and verifies rather than trusts —
-a referred host is a candidate hostname until it answers for itself.
+a referred host is a candidate hostname until it answers for itself. Measured on the live graph:
+`REFERRAL` is rare (two publishers in 141 servers) and the gate bites — the game they both name
+publishes only its codebase's name, so it is discovered, probed, kept, and *not* listed until it says
+who it is. Referral is a bonus path, not the primary one; that is what the backfill importer is for.
 
 ## Shape
 
@@ -111,12 +119,17 @@ states, not two:
 |---|---|
 | Probed, count obtained | Filled cell — including a measured zero, which means we got in and nobody was there |
 | Probed, no count obtainable | Hatched cell — the WHO was unparseable and MSSP had no `PLAYERS` |
-| Probe failed | Empty cell — not reachable |
+| Not measured | Empty cell — we have no measurement for that hour, which is not a claim the game was down |
 
 Collapsing the middle case into either neighbour is the worst bug this system could ship: a game
 whose `DOING` header is customised past our parser would otherwise render as permanently dark while
 running perfectly well.
 
+The third case is stated as *ours*, not theirs. A failed probe writes no presence row at all, so an
+empty cell covers both an hour we could not reach and an hour we never probed — and a game found an
+hour ago would otherwise have a perfect week described as 167 hours of downtime. Reachability is the
+90-day strip's question, and it is derived from intervals that can tell the two apart.
+
 The vocabulary is **reachable**, never *uptime* — we measure a socket from one vantage point, and a
 game with a routing problem to our host is unreachable and perfectly alive.
 

docs/screenshots/README.md

@@ -1,53 +1,64 @@
-# Screenshots — 30 July 2026
+# Screenshots — 31 July 2026
 
-Captured from the running site at `31f5864`, rendering from `FixtureGameQueries`. Every value in the
-fixture came off a real server, which is why the hard states appear at all.
+Captured from the running site against **a real PostgreSQL catalogue**, populated by `mui-crawl`
+against live servers: sixteen games, one probe each. Nothing here is a fixture. The site prefers a
+database and falls back to the demo fixture only when none is configured — and when it does, every
+page carries a banner saying so, because a reader who cannot tell a measurement from a fixture is
+being misled by exactly the mechanism this project exists to replace.
 
 | File | What it shows |
 |---|---|
-| `01-home-feeds.png` | The three liveness feeds. The *came back* card is the one place the site raises its voice. |
-| `02-games-listing.png` | The listing, with a measured zero, an unknown count and an archived game distinguishable at a glance. |
-| `03-game-page.png` | The whole game page: ANSI frame, heatmap, reachable strip, capability matrix, provenance. |
+| `01-home-feeds.png` | The three liveness feeds, off a first crawl: sixteen games *newly discovered*. |
+| `02-games-listing.png` | The listing. Measured counts, measured zeroes, and games whose handshake offered nothing. |
+| `03-game-page.png` | Virtustan MUD — the whole game page, and the game that led the crawler to two more endpoints. |
 | `04-plain-mode.png` | `?plain=1` — the same facts as words. |
-| `05-archive.png` | The archive as a section rather than a bin. |
+| `05-archive.png` | The archive. Empty, and saying so: nothing has been dark long enough. |
 | `06-mobile-game.png` | 390px. Single column, heatmap keeps all 168 cells. |
 
 ## What to look for
 
-**The game page leads with the game.** Mark, name, description, address — then the connect screen
-under *"what you see when you connect"*, which is what it is: the first piece of evidence, not the
-masthead. Leading with the art would hand the top of every page to whatever a stranger's server sends.
+**Every page is one probe old, and the page says so rather than pretending otherwise.** This is what
+makes the set worth keeping: it is the site's *first day*, which is the state every real game enters
+in, and it is where three sentences turned out to be lying.
 
-**`GMCP` sorts to the top of the capability matrix and is labelled `DISAGREES`.** Its MSSP says
-`claimed`, six years old; the handshake has never offered it. That disagreement is the single most
-useful thing the matrix can say, so it cannot be scrolled past.
+**"Reachable 100.0% of the 1 day we have measured."** The fraction's denominator has always been
+observed time; the sentence used to widen it to "of the last 90 days". Right number, claim
+eighty-nine days wider than the evidence.
 
-**The heatmap has three states and they are told apart by shape.** Filled is counted — *including a
-measured zero*. Hatched is probed-but-uncountable (see Friday). Empty is not reachable (Wednesday's
-twelve-hour band). Greyscale printing is a valid rendering of this grid.
+**The heatmap's empty cells read *no measurement in that hour*.** They used to read "not reachable —
+no measurement at all", and the summary said "167 hours across the week could not be measured — the
+game was not reachable", about a game measured once and found perfectly reachable. A failed probe
+writes no presence row, so silence there cannot tell an outage of theirs from a gap of ours.
+Reachability is the strip's question, and the strip is derived from intervals that can.
 
-**The reachable strip has four states, not three.** The fourth is *not measured* — the strip is 90
-days wide, and a game found last Tuesday has no history before that. Painting those days unreachable
-would record our ignorance as their measurement.
+**"What the game says about itself" contains only things the game said.** `banner_hash` — a digest
+*we* compute, 64 hex characters wide — used to sit at the top of that panel, off the edge of its
+column.
 
-**Ages are relative and stale ones are marked.** `created 2009 ◇ 6y` is not wrong, it is old, and the
-page says so rather than presenting it as current.
+**The three states are still three states.** Filled is counted, including a measured zero (see
+`eldertaleonline.com`, which answered and had nobody on). Hatched is probed-and-uncountable. Empty is
+no measurement. Greyscale printing is a valid rendering of this grid.
 
 **It is called *reachable*, never *uptime*.** We measured a socket from one vantage point; we did not
 measure whether the game was up.
 
 ## The plain rendering is the test
 
 `04-plain-mode.png` is not a courtesy. If a fact cannot survive there, its graphic on the main page
-was decoration. Compare:
+was decoration — and the plain surface carried the same wrong sentence, in the same words, which is
+what makes it a real parity check rather than a second implementation.
 
 ```
-Wed — peak 13 at 20:00, 12 hours not reachable
-Fri — peak 16 at 20:00, nobody on 05:00-11:59, 1 hour probed but uncountable
-
   counted   = we got in and read a number, including a measured zero
   uncounted = we got in and no number could be read
-  no data   = we could not reach the game in that hour at all
+  no data   = we have no measurement for that hour
 ```
 
 Three states, in words, with no colour to carry them.
+
+## Reproducing
+
+```bash
+mui-crawl --connection "…" --seed mush.pennmush.org:4201 --seed mud.kharkov.org:3000 …
+MUI_POSTGRES="…" dotnet run --project src/MUI.Web
+```

docs/specs/2026-07-30-mu-directory-design.md

@@ -256,7 +256,7 @@ count".** The heatmap has three renderings, so the store must distinguish three
 |---|---|---|
 | Probe succeeded, count obtained | `PresenceSample(count = n)` | Filled cell, including a measured zero |
 | Probe succeeded, no count obtainable | `PresenceSample(count = NULL, unmeasurable_reason)` | Hatched cell — *probed, unmeasurable* |
-| Probe failed | `AvailabilityInterval` transition, **no** presence row | Empty cell — *not reachable* |
+| Probe failed, or no probe at all | `AvailabilityInterval` transition, **no** presence row | Empty cell — *no measurement for that hour* |
 
 The middle row is the one the first cut of this spec missed: it said only that a *failed* probe
 writes no sample, which left a successful probe with an unparseable WHO writing nothing either —
@@ -266,6 +266,20 @@ have rendered as permanently dark while running fine.
 A measured zero is a filled cell, not an absence. It means we got in and nobody was there, which is
 a real and useful fact about a game.
 
+**The empty cell is a statement about us, and must be worded as one.** Because a failed probe writes
+no presence row, the third case covers both an hour we could not reach and an hour we never probed —
+and the store cannot tell them apart from presence alone. Every surface therefore says *no
+measurement*, never *not reachable*: it shipped the other way, and a game the crawler had measured
+once, and found perfectly reachable, had 167 hours of its week described as downtime. Whether the
+game was reachable in an hour is §5.8's question, answered from intervals, which carry that fact
+directly.
+
+The same reading applies to any figure derived from a partly-observed window. A reachability
+percentage divides by *observed* time, so a surface may print it as a fraction of the last 90 days
+only when all ninety were measured; otherwise it names the days it measured. One successful probe an
+hour ago is not "reachable 100% of the last 90 days", and the arithmetic being right does not make
+the sentence true.
+
 ### 5.5 Endpoints
 
 ```
@@ -427,6 +441,20 @@ A referred host must independently answer MSSP with its own `NAME`/`HOSTNAME` be
 Depth and fan-out per source are capped. The referring game is recorded on the discovered entry so
 that a hostile or careless `REFERRAL` list can be traced and its whole subtree pruned.
 
+**Measured, and rarer than the protocol implies.** Of 141 live servers probed on 30 July 2026 — the
+codebase survey's set plus every entry in TinTin's MSSP crawler list — exactly **two** publish
+`REFERRAL`, and both name the same game: `mud.virtustan.net:8888` and `mud.kharkov.org:3000` (one
+operator, two servers) point at `tbamud.com`, one of them naming two ports. Both referred endpoints
+became depth-1 targets, were probed on their own schedule, and are **not listed** — tbaMUD's only
+self-description is `NAME "tbaMUD"`, its codebase's name, which §7.3's placeholder rule reads as
+unset. That is the gate above working exactly as written, and it is worth knowing that the first
+real graph this crawler walked ended that way.
+
+Two consequences. Referral is a *bonus* discovery path and cannot be the primary one at this
+density, which is what §7.6's backfill is for. And the rule has a cost worth restating: a real,
+reachable game stays unlisted because its operator never edited one line — recoverable the moment it
+publishes a name, and the target is kept and re-probed for ever in the meantime.
+
 #### The gate is on the resolved address, not the name
 
 **Checking the hostname is not enough, and treating it as enough is a server-side request forgery
@@ -484,6 +512,17 @@ The same caution applies to `CONTACT` and `WEBSITE`, which are shared across eve
 provider more often than they are unique, and to `CREATED`, which is a year and therefore collides
 freely on its own.
 
+**The banner signal has the same failure mode one layer down, and the probe is where it is fixed.**
+A connect screen is a good fingerprint because operators edit it; a *placeholder* connect screen is
+the codebase's, shared by every install. tbaMUD sends `Attempting to Detect Client, Please Wait...`,
+pauses for about a second and a half, and then paints the real screen — so a probe that settles on a
+gap between lines stored that one placeholder line as the banner, and two unrelated tbaMUDs
+fingerprinted identically. That is `NAME "PennMUSH"` again in a different field, and the answer is
+not to discount the signal but to stop truncating the evidence: the connect-screen phase waits
+longer when what it has is slight and has not reached a prompt (`ProbeOptions.BannerPatience`).
+Found by a referral crawl, which is the only thing that had put two servers of one codebase side by
+side.
+
 Above threshold: auto-merge into the existing game, recording the endpoint change as a
 `FieldChange`. Middling: open a suspected-duplicate pair for review — **both pages stay live and
 link to each other reciprocally**, because a wrongly hidden game is worse than a visible duplicate.
@@ -583,6 +622,24 @@ every source on the about page and in the API. These sites are run by people in
 hobby, and several of them are the reason any of this data exists at all. A short email first is
 both the decent move and the one most likely to get better data than scraping would.
 
+**The import is a one-time operation against one deployment, and its output is not part of the
+source tree.** MUIndex is deployed to a single place; the backfill exists to give *that* database a
+day-one population, and once it has run the crawler keeps the catalogue true by measuring. So it is
+one command, run once, by an operator, pointed at the production connection string — not a startup
+step, not a scheduled job, and not something a fresh clone reproduces.
+
+The consequence is a repository rule: **the importer is code and is committed; the imported data is
+not.** No harvested catalogue is checked in — no mirrored listing file, no seed list of real hosts
+scraped from a directory, no snapshot of anyone's database. Test fixtures are exempt because they are
+inputs to a test rather than the dataset: a handful of hand-written rows exercising a parser is a
+fixture, and a copy of a third party's catalogue is not, whatever it is named. A run that produces an
+artifact writes it to a path the operator chose, and those paths are ignored by git.
+
+Two reasons beyond tidiness. Republishing another site's catalogue as a file in our repository is a
+redistribution nobody agreed to, whatever the etiquette above secured for *ingesting* it. And a
+committed dataset is asserted data with a git history — the exact thing the front page says this
+project does not do — which would rot in place while the live catalogue moved on.
+
 ### 7.7 Scheduling
 
 A single scheduler picks due targets by `next_probe_at`, feeding a bounded worker pool. Interval is