Materialize a window in one commit instead of one round trip per occurrence

[GiampaoloGabba]

Rescoped after measurement. The original issue proposed two changes; one of them was measured and
rejected, so what remains is the bulk write. Numbers, method and reproduction steps are in the
measurement comment below.

The problem

A materialization pass writes one occurrence per round trip: MaterializeOccurrence is called once per
granted slot, each with its own command against the store (OccurrenceMaterializer.cs, the grant loop).
A schedule that materializes a real window pays that once per child, sequentially.

What a batched write is worth

OMB compares a 500-slot window written one round trip at a time against the same window in one commit,
on Postgres, same table shape, same unique index, same FOR UPDATE on the schedule row, same
compare-and-swap on version and cursor. The bulk arm is the provider's own writable CTE with the single
row replaced by an unnest of arrays.

Arm	Throughput	Service time per occurrence
single, today's path	7,681/s	2,014 µs
bulk, one commit per window	85,274/s	177 µs

11x the throughput, 91% off the per-occurrence service time.

That is a conflict-free happy-path upper bound, not a gain an implementation would reach:

• The bulk arm returns one count for the window. A real member owes a per-slot outcome, the per-slot
  metadata and the audit rows.
• On a partial conflict it advances the cursor past the whole window and only then learns that fewer
  rows went in. A real member cannot do that.
• The single arm goes through EF (a pooled context and command per occurrence) while the bulk arm holds
  one raw connection per lane, so part of the gap is per-call storage overhead rather than round trips.

Before implementing, re-measure the single arm over the same raw connection the bulk arm uses. That
separates the batching win from the EF overhead, and only the first is what this issue is about.

Who this helps

Only schedules that have opened their window. MaxPendingOccurrences defaults to 1, meaning strictly
serial (MisfireSettings.cs:36), and capacity is MaxPendingOccurrences - activeOccurrences
(DueSlotEnumerator.cs:111). With the default, a pass grants one occurrence, there is nothing to batch,
and this change is worth nothing. It pays for applications that deliberately raised that cap, and it
scales with how far they raised it.

What was measured and rejected

The other half of the original proposal was to prepare an immutable template once per pass, so that the
DI scope, the handler resolution and the payload serialization are not repeated per child. Measured, the
whole preparable part is 631 ns with no payload and 818 ns at 1 KB, against 793.5 µs per occurrence on
SQLite and 2,089 µs on Postgres: about 0.1%, and under 1% even with a 64 KB payload, where the cost
is writing the bytes rather than serializing them. With the default window it saves nothing at all,
since one grant per pass has nothing to amortize. An end-to-end subtraction could not resolve it above
noise at either concurrency.

That half is not worth writing. This issue does not cover it.

Design constraints for whoever picks it up

• Per-slot compare-and-swap semantics have to survive: the unique index on (schedule, slot) is what
  decides duplicates today, and a batched write still has to report which individual slot lost a race.
• ITaskStorage is public and already carries capability flags with default implementations
  (SupportsDurableOccurrences, SupportsScheduleVersioning). A bulk member should follow that pattern
  so custom storage implementations keep compiling and fall back to the per-slot path.
• Each provider writes at its own tier today (procedures on SQL Server and MySQL, a writable CTE on
  Postgres, one SaveChanges on the EF base). A bulk member should keep that shape rather than force a
  common denominator.

[GiampaoloGabba]

Measured

Measure-first pass on 2070fe6. No production behaviour changed: the only edit under src/ is two
InternalsVisibleTo attributes so the benchmarks can reach internal types. AMD Ryzen 9 7950X (16C/32T), .NET 10.0.11,
Postgres 16 in Docker, SQLite in WAL. Two independent adversarial reviews went over these numbers; the
first version of them had the wrong denominator and is not what follows.

The issue proposes two things. They turn out to be worth very different amounts, so they are measured
separately.

First, the size of the window

capacity = MaxPendingOccurrences - activeOccurrences (DueSlotEnumerator.cs:111) and
MaxPendingOccurrences defaults to 1 (MisfireSettings.cs:37). In the default configuration the
grant loop runs once per pass: there is no repetition to hoist out of it. A 500-child window is
something an application opts into. (MaxServedSlotsPerRun = 500 in the materialiser is a different
bound: it limits walking over already-served slots, not how many children a pass creates.)

Part one: preparing DI and serialization once per pass

Per-occurrence build cost, by payload:

Step	none	1k	64k
build + map, the whole preparable part	631 ns	818 ns	36,080 ns
DI scope + handler resolve + executor	180 ns	173 ns	177 ns
runtime info serialize	186 ns	196 ns	191 ns
payload serialize alone	65 ns	186 ns	34,062 ns

Against what an occurrence actually costs end to end:

Cell	Lanes	Service time per occurrence	Throughput
SQLite, window 500	1	793.5 µs	1,260/s
SQLite, the write alone (LDM)	1	794.0 µs	1,259/s
Postgres, window 500	16	2,089 µs	7,501/s
Postgres, the write alone (LDM)	16	2,169 µs	7,253/s
Postgres, window 500, 64k payload	16	3,965 µs	3,503/s

The subtraction does not resolve the build cost at either concurrency. Single-lane: 793.5 against
794.0 µs, so the whole pass came out 0.5 µs cheaper than the write alone. At 16 lanes it comes out
negative too. Both are below the noise, in the wrong direction, which is itself the answer: whatever
the build costs, it is smaller than what this measurement can see. The micro pins it independently at
0.6-0.8 µs with a small payload.

So hoisting DI and serialization out of the loop has a ceiling of about 0.1% of an occurrence with a
small payload. With a 64k payload the payload serialization is 34 µs of a 3,965 µs occurrence, under
1%: most of what a large payload costs is writing 64 KB to the database, not turning it into JSON.
In the default configuration the ceiling is zero, since one grant per pass has nothing to amortise.

One case is worth separating out, because a per-pass template would miss it: a per-schedule cache of
the serialized payload would help even at window 1, where every pass re-serializes the same bytes. It
does not help enough. Service time was 6,934.1 µs in the first run and 6,956.9 µs in the nine-iteration
redo, so the 34 µs serialization cost is about half a percent. Throughput remained unstable (CV 13.6%
over three measured iterations and 18.2% over nine) and is not used for this conclusion.

Part two: the bulk write

This is the half the build cells say nothing about, and it is where the issue is right.

OMB compares a window written one round trip at a time against the same window in one commit: same
table shape, same unique index, same FOR UPDATE on the schedule row, same compare-and-swap on version
and cursor. The bulk arm is the Postgres provider's own writable CTE with the single row replaced by an
unnest of arrays. The payload is serialized once and shared by both arms, so serialization is out of
the comparison.

Arm	Throughput	Service time per occurrence
single, today's path	7,681/s	2,014 µs
bulk, one commit per window	85,274/s	177 µs

11x the throughput, and 91% off the per-occurrence service time.

Read that as a conflict-free happy-path upper bound, not as a gain an implementation would reach.
Three things make the bulk arm cheaper than a real member could be:

• It returns one count for the window. A real member owes a per-slot outcome, the per-slot metadata and
  the audit rows.
• On a partial conflict it advances the cursor to the end of the window and only then discovers that
  fewer rows went in. A real member cannot do that.
• The single arm goes through EF, a pooled DbContext and command per occurrence, while the bulk arm
  holds one raw NpgsqlConnection per lane. Part of the gap is per-call storage overhead, not round
  trips. The two arms also ran as separate processes against separate containers.

What survives all of that is the shape of the result: an order of magnitude, against a part-one ceiling
of a tenth of a percent. Even a fraction of this gap is worth more than everything in part one.

Reading

The two halves are independent changes, and the measurements point in opposite directions. The per-pass
template is about 0.1% of an occurrence with a small payload and stays under 1% at 64 KB; it saves
nothing in the default one-grant configuration and was below noise at the measured 16 lanes. The bulk
write is where the money is for applications materialising real windows.

With MaxPendingOccurrences at its default of 1, neither half does anything for a single schedule's
pass, so this is an optimization for applications that have deliberately opened the window, and its
value scales with how far they opened it. A batched write across many schedules, in a recovery sweep,
is a different question these cells do not answer and which might well be worth its own look.

Reproduce

# the build split, per payload
dotnet run -c Release --project benchmarks/EverTask.Benchmarks -- --filter "*OccurrenceBuild*"

# the whole pass against the write alone, at the SAME effective concurrency
dotnet run -c Release --project benchmarks/EverTask.LoadHarness -- OMP --storage sqlite --backlog 500 --count 5k --parallelism 1 --warmup 2 --measured 5
dotnet run -c Release --project benchmarks/EverTask.LoadHarness -- LDM --storage sqlite --parallelism 1 --count 5k --warmup 2 --measured 5

# the bulk ceiling, one arm per line
dotnet run -c Release --project benchmarks/EverTask.LoadHarness -- OMB --storage postgres --arm single --backlog 500 --count 8k --parallelism 16 --warmup 2 --measured 5
dotnet run -c Release --project benchmarks/EverTask.LoadHarness -- OMB --storage postgres --arm bulk   --backlog 500 --count 8k --parallelism 16 --warmup 2 --measured 5

OMP and LDM print both a service time (summed operation time over occurrences, the latency
denominator, and the one to subtract between cells) and a wall time (the reciprocal of aggregate
throughput). Never subtract the throughputs of two cells that ran at different concurrency; the first
version of this measurement did, and it was wrong.