Skip to content

materialize: delete superseded blobs (quota blowout fix) #421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 1 commit into from
Jun 12, 2026
Merged

materialize: delete superseded blobs (quota blowout fix) #421

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions src/lib/materialize/store.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,18 +66,32 @@ function contentHash(s: string): string {
return (h >>> 0).toString(16);
}

const BLOB_TTL_SEC = 24 * 3600;
// Short safety-net TTL only. Every sweep produces a new hash (dataAsOf
// changes), so blobs MUST be deleted on pointer swap: with a 24h TTL the
// per-minute tier-A cadence accumulated ~4300 orphan blobs in one night
// and blew the Upstash 256MB quota, turning every write into an http 400
// (worker looked alive, store silently froze).
const BLOB_TTL_SEC = 2 * 3600;

/** Atomic publish: blob first, pointer swap second. */
/** Atomic publish: blob first, pointer swap second, then delete the
* previously-pointed blob (readers that already fetched the old hash
* finish their GET; new readers see the new pointer). */
export async function publishSnapshot(
snap: MaterializedSnapshot,
): Promise<void> {
const json = JSON.stringify(snap);
const hash = contentHash(json);
const blobKey = matKeys.blob(snap.slug, snap.sig, hash);
const ptrKey = matKeys.pointer(snap.slug, snap.sig);
const prevHash = await redis(["GET", ptrKey], 5_000).catch(() => null);
await redis(["SET", blobKey, json, "EX", BLOB_TTL_SEC], 15_000);
await redis(["SET", ptrKey, hash], 8_000);
if (typeof prevHash === "string" && prevHash && prevHash !== hash) {
await redis(
["DEL", matKeys.blob(snap.slug, snap.sig, prevHash)],
5_000,
).catch(() => {});
}
}

export async function heartbeat(now = Date.now()): Promise<void> {
Expand Down
Loading