Roll an hour up without letting it lose one of its three states (#28)

* Roll an hour up without letting it lose one of its three states

presence_sample has been partitioned monthly since 0003 for rollups and
retention that nobody had written. This writes them.

The rollup keeps the tally and not the conclusion: presence_rollup_hour and
presence_rollup_day carry counted and uncountable sample counts separately,
min/max/mean over the counted ones alone, and no row at all for an hour nobody
measured. A measured zero is a filled cell, a probe that got in and could not
count is a hatched one, and an hour we never reached is the absence of a row --
the same three states the raw table keeps, after aggregation. The CHECK
constraints refuse a bucket with a zero it did not measure and a bucket that
claims to be a measurement of nothing.

Partitions are now made ahead of need as well as before every append, so a
month ending is never the first thing to discover a database we cannot write
DDL to. The pass runs behind its own Postgres advisory lock, like the crawl
loop -- CrawlLease is renamed AdvisoryLease, since two workers now share it.

Retention is configuration, because 15.4 is open. It keeps everything at every
grain by default, with 5.2's own figures a preset away, and it never runs ahead
of the rollup: raw months go whole, and only once both grains have consumed
them. 15.4 records the choice as ship conservative and tune.

Salt rotation, which 11 promised and nothing implemented: a per-epoch HMAC salt
derived from a deployment secret, an epoch label recorded beside every
aggregate, and a rollup that refuses to combine an estimate across a rotation
rather than publishing a bigger number. Names go in and sixteen bytes come out;
nothing persists a name.

Testcontainers 4.13 pulls SSH.NET 2025.1.0, which now has an advisory, and
TreatWarningsAsErrors turns that into a broken restore for everyone. Bumped.

* Let each grain remember for itself how far it has read

The daily rollup resumed from the hourly watermark, which is fine right up
until the two statements do not both run. A pass that rolled the hours,
committed their watermark and then died left the days unwritten -- and the next
pass, reading the hourly mark, skipped everything older than the overlap, wrote
its own watermark as though it had read it, and let retention drop the raw
months behind it. Six days of history came back as six hour buckets and one day
bucket, and then the raw rows went. The grain 5.2 keeps for ever, missing, with
the only other copy deleted.

Each grain now resumes from its own watermark and writes that watermark only
after its own aggregation, so an interrupted pass resumes instead of skipping,
and retention still waits for whichever grain is behind.

An estimate with no salt epoch also cost more than itself: the invariant ran
inside JSON deserialisation, so one such row threw out of the middle of a read
and took the whole window with it, while the aggregation read the same row
without complaint. The reader now drops the estimate and keeps the buckets, and
the SQL filters the same case, so both paths agree that an unlabelled estimate
is not one anything may compare with another.

The maintenance service starts beside the crawler's migration run rather than
after it, so it asks whether the schema is there instead of finding out as a
42P01 and standing down for five minutes.

11's salt machinery is in place and nothing feeds it: the WHO parser counts
rows and never extracts a name, so no probe produces aggregates. Said plainly
in 5.2 and beside the registration, rather than left to read as delivered.

---------

Co-authored-by: Harry Cordewener <nekomercutio@gmail.com>

Author: HarryCordewener

docs/specs/2026-07-30-mu-directory-design.md

@@ -226,10 +226,36 @@ called, by whatever assembles the probe's reading. It cannot be applied afterwar
 and choosing later, because there is nowhere to keep both. This is a constraint on the ingestor, not
 a limitation of the store.
 
-**Rollups are not built and have no owner.** §5.2 states the retention shape — raw 90 days, hourly
-two years, daily forever — and nothing implements it. Until it does, the heatmap reads raw samples
-over an eight-week window. Monthly range-partitioning is in place, which is what makes a rollup a
-cheap addition rather than a rewrite.
+**Rollups are built; the retention beside them is configuration.** `PresenceMaintenance` runs on a
+Postgres advisory lock of its own — the same gate §12 puts on the crawl loop, a different key — and
+does three things per pass: makes the monthly partitions ahead of need, aggregates every complete
+hour into `presence_rollup_hour` and `presence_rollup_day`, and then applies whatever retention a
+deployment configured. **The three states survive the aggregation**: each bucket carries a tally of
+counted and of uncountable samples rather than a conclusion, min/max/mean are over counted samples
+alone and are null when there were none, and an hour nobody measured produces no row — which is
+§5.4's empty cell, unchanged in shape. The `CHECK` constraints refuse a bucket that claims a count it
+did not measure and one that claims to be a measurement of nothing at all.
+
+Retention is `PresenceRetentionOptions` and defaults to **keeping everything at every grain**, for
+the reason in §15.4. The shape above — raw 90 days, hourly two years, daily forever — is available as
+`PresenceRetentionOptions.AsDesigned` and is one setting away. Raw samples go by whole partitions and
+never row by row, and never before both rollups have consumed them: **each grain resumes from its own
+watermark** and writes that watermark only after its own aggregation, and retention is clamped to the
+older of the two. A pass that rolled the hours and then died cannot leave the daily grain believing
+it had read a year it never saw, and cannot drop the raw months that are then the only copy.
+
+The heatmap still reads raw samples over its eight-week window, which is why the shortest raw
+retention this accepts is that window. Pointing it at the hourly rollup is the remaining work.
+
+**§11's salt machinery is in place and nothing feeds it yet, which is worth saying plainly.**
+`RotatingSaltProvider` derives a per-epoch salt from a deployment secret, `PresenceAggregates`
+refuses an estimate that does not name its epoch, the rollup carries an estimate only where a
+bucket's samples share one, and the reader and the aggregation agree that an unlabelled estimate is
+not readable. But no probe produces aggregates: the `WHO` parser counts rows and never extracts a
+name, so `PresenceAggregates` is constructed nowhere in `src/`, and the `aggregates` column, the
+`salt_epoch` columns and the constraints around them are unreached in the shipped pipeline. That is
+the safe half to have built first — the rule that names are never persisted is kept by there being
+no path that handles one — but the unique-player estimate is designed, not delivered.
 
 **Activity band**, the facet §9 exposes, is derived here and defined once: `players now` (a non-null
 count above zero in the most recent hourly rollup), `active this week` (any such count in 7 days),
@@ -1095,7 +1121,21 @@ webhooks beyond RSS; non-English UI (listings carry `LANGUAGE` from day one).
    decision and need not match.
 3. **Hosting and cost envelope**, which bounds probe frequency and retention.
 4. **Retention policy** for `PresenceSample` before rollup, and the salt rotation period for §11
-   aggregates.
+   aggregates — both unsettled, and both now **shipped conservative to be tuned**, because the
+   machinery could not wait for the answer and neither question has data behind it yet. §5.2's
+   rollups and partition maintenance are built; what is deliberately not compiled in is how long
+   anything is kept. `PresenceRetentionOptions` therefore **keeps everything by default** at all three
+   grains, with §5.2's own figures available as a named preset. §5.2 does authorise dropping raw
+   samples once they have been aggregated — it is the only data in the system that is ever deleted —
+   but the period it names has never been checked against a real deployment's storage, and §15.3, the
+   cost envelope that would bound it, is open too. Between an unvalidated number and a deletion, the
+   default is not to delete: turning retention on later costs one setting, and turning it on too early
+   costs measurements that cannot be taken again. The salt period defaults to **seven days** on the
+   same reasoning read the other way round — the two errors are not symmetrical. An epoch that proves
+   too short costs precision in an estimate and can be lengthened from the next epoch onwards; an
+   epoch that proves too long has already linked a season of observations together, and no later
+   setting un-links them. Tune both once there is a year of data and a hosting bill to read them
+   against.
 5. **Auto-merge threshold** in §7.3 — needs calibration against real data, so ship conservative and
    tune.
 6. **The archive grace factor** in §7.5 — the quarter, the 60-day floor and the 365-day ceiling are

migrations/0003_presence_sample.sql

@@ -1,6 +1,8 @@
 -- spec §5.2 — the only table in this schema that grows linearly with games × time, which is why it is
--- the only partitioned one. RANGE on `at`, monthly, so §5.2's retention and rollups can later work on
--- whole partitions rather than row-by-row deletes over hundreds of millions of rows.
+-- the only partitioned one. RANGE on `at`, monthly, so §5.2's retention and rollups work on whole
+-- partitions rather than row-by-row deletes over hundreds of millions of rows. Migration 0011 is what
+-- reads that promise back: it adds the two rollup tables and the watermark that retention checks
+-- before it drops a month.
 --
 -- There is no DEFAULT partition, deliberately: rows landing in one would block the later creation of
 -- the partition that should have held them. NpgsqlPresenceStore creates the month's partition before

migrations/0011_presence_rollup.sql

@@ -0,0 +1,129 @@
+-- spec §5.2 — the rollups the monthly partitioning in 0003 was put there for, and the retention that
+-- can then work on whole partitions instead of row-by-row deletes over hundreds of millions of rows.
+--
+-- The rule these two tables exist to carry forward is §5.4's: an hour has THREE states, and a rollup
+-- that turns "probed but uncountable" or "not measured" into a zero has destroyed the only thing this
+-- schema is careful about. So the tally is kept, not the conclusion:
+--
+--   counted_samples > 0                             -> a filled cell, INCLUDING a measured zero
+--   counted_samples = 0 AND unmeasurable_samples > 0 -> a hatched cell: probed, could not count
+--   no row at all                                    -> not measured, which is the absence of a row
+--                                                       here exactly as it is in presence_sample
+--
+-- min/max/sum are computed over counted samples only and are NULL when there were none, so a
+-- reader that asks for a number gets nothing rather than a zero it could mistake for a measurement.
+CREATE TABLE presence_rollup_hour (
+    game_id              uuid NOT NULL REFERENCES game (id),
+
+    -- The start of the hour, in UTC. Truncation is done in UTC on both sides of the wire so that a
+    -- session's TimeZone setting can never move an hour into its neighbour.
+    hour                 timestamptz NOT NULL,
+
+    -- §5.4's tally. Both of these are counts of rows in presence_sample, never of hours.
+    counted_samples      integer NOT NULL,
+    unmeasurable_samples integer NOT NULL,
+
+    -- Over the counted samples alone. The mean is stored as sum/count rather than as an average so
+    -- that the daily rollup can be computed from the hourly one exactly, rather than by averaging
+    -- averages over hours that saw different numbers of probes.
+    min_count            integer,
+    max_count            integer,
+    sum_count            bigint,
+    mean_count           numeric GENERATED ALWAYS AS (sum_count::numeric / NULLIF(counted_samples, 0)) STORED,
+
+    -- §11. The salt rotates, and an estimate may only ever be read within one epoch: the epoch is
+    -- recorded so that nothing downstream can combine two of them. Where an hour's samples spanned a
+    -- rotation both columns are NULL — the honest answer, because the hashes that would have been
+    -- needed to merge the two sets are never persisted.
+    salt_epoch             text,
+
+    -- The largest single-sample estimate inside that one epoch, and deliberately not a union: a union
+    -- of distinct players needs the hashes, and the hashes never leave the probe. A lower bound that
+    -- says so beats a total that cannot be justified.
+    peak_distinct_estimate integer,
+
+    PRIMARY KEY (game_id, hour),
+
+    -- A rollup row is a statement that something was measured in that hour. A row of zeroes would be
+    -- "not measured" wearing a measurement's clothes, which is §5.4's third state collapsed into the
+    -- other two — so the schema refuses to hold one.
+    CONSTRAINT presence_rollup_hour_measured_something CHECK (
+        counted_samples > 0 OR unmeasurable_samples > 0),
+
+    CONSTRAINT presence_rollup_hour_tallies_are_not_negative CHECK (
+        counted_samples >= 0 AND unmeasurable_samples >= 0),
+
+    -- The other half of the same rule: an hour with no counted sample has no count, and an hour with
+    -- one has all three. Nothing may write an uncountable hour with a zero in it.
+    CONSTRAINT presence_rollup_hour_counts_iff_counted CHECK (
+        (counted_samples > 0) = (min_count IS NOT NULL)
+        AND (counted_samples > 0) = (max_count IS NOT NULL)
+        AND (counted_samples > 0) = (sum_count IS NOT NULL)),
+
+    CONSTRAINT presence_rollup_hour_counts_are_ordered CHECK (
+        min_count IS NULL OR (min_count >= 0 AND max_count >= min_count AND sum_count >= max_count)),
+
+    CONSTRAINT presence_rollup_hour_estimate_names_its_epoch CHECK (
+        peak_distinct_estimate IS NULL OR salt_epoch IS NOT NULL),
+
+    CONSTRAINT presence_rollup_hour_is_on_the_hour CHECK (
+        hour AT TIME ZONE 'UTC' = date_trunc('hour', hour AT TIME ZONE 'UTC'))
+);
+
+-- §9's heatmap reads one game over a window, which the primary key serves. This is for the ecosystem
+-- dashboard and the activity band, which read every game over one.
+CREATE INDEX presence_rollup_hour_hour_idx ON presence_rollup_hour (hour);
+
+-- The same shape a day at a time, and the one §5.2 keeps for ever. Computed from the hourly rollup
+-- rather than from the raw table, so it survives the raw rows being dropped and stays exact.
+CREATE TABLE presence_rollup_day (
+    game_id              uuid NOT NULL REFERENCES game (id),
+    day                  timestamptz NOT NULL,
+
+    counted_samples      integer NOT NULL,
+    unmeasurable_samples integer NOT NULL,
+
+    min_count            integer,
+    max_count            integer,
+    sum_count            bigint,
+    mean_count           numeric GENERATED ALWAYS AS (sum_count::numeric / NULLIF(counted_samples, 0)) STORED,
+
+    salt_epoch             text,
+    peak_distinct_estimate integer,
+
+    PRIMARY KEY (game_id, day),
+
+    CONSTRAINT presence_rollup_day_measured_something CHECK (
+        counted_samples > 0 OR unmeasurable_samples > 0),
+
+    CONSTRAINT presence_rollup_day_tallies_are_not_negative CHECK (
+        counted_samples >= 0 AND unmeasurable_samples >= 0),
+
+    CONSTRAINT presence_rollup_day_counts_iff_counted CHECK (
+        (counted_samples > 0) = (min_count IS NOT NULL)
+        AND (counted_samples > 0) = (max_count IS NOT NULL)
+        AND (counted_samples > 0) = (sum_count IS NOT NULL)),
+
+    CONSTRAINT presence_rollup_day_counts_are_ordered CHECK (
+        min_count IS NULL OR (min_count >= 0 AND max_count >= min_count AND sum_count >= max_count)),
+
+    CONSTRAINT presence_rollup_day_estimate_names_its_epoch CHECK (
+        peak_distinct_estimate IS NULL OR salt_epoch IS NOT NULL),
+
+    CONSTRAINT presence_rollup_day_is_on_the_day CHECK (
+        day AT TIME ZONE 'UTC' = date_trunc('day', day AT TIME ZONE 'UTC'))
+);
+
+CREATE INDEX presence_rollup_day_day_idx ON presence_rollup_day (day);
+
+-- How far each rollup has consumed the raw table. Two jobs read this: the rollup itself, which
+-- resumes from it rather than re-aggregating history on every pass, and retention, which may not drop
+-- a raw partition that has not been rolled up yet — the one ordering that makes dropping raw samples
+-- recoverable-in-shape rather than a loss.
+CREATE TABLE presence_rollup_state (
+    scope             text PRIMARY KEY,
+    rolled_up_through timestamptz NOT NULL,
+    updated_at        timestamptz NOT NULL DEFAULT now(),
+
+    CONSTRAINT presence_rollup_state_scope_vocabulary CHECK (scope IN ('hour', 'day'))
+);