Skip to content

Periodic incremental image_scales refresh: rewrite metadata for objects changed since the last run #20

Description

@jensens

Problem

image_scales catalog metadata freezes whatever URL was resolvable at index time. Several routine situations leave entries that are stale or suboptimal, and nothing brings them back on its own:

  1. Fresh uploads can never bake a direct Thumbor URL. reindexObject() runs inside the creating transaction, where the image blob has no committed TID yet — ZODB assigns OID/serial during commit. get_blob_ids() therefore returns None (it explicitly rejects z64 serials), _build_thumbor_url() returns None, and the uid fallback URL gets indexed instead.
  2. Out-of-request indexing produces @@images URLs rather than Thumbor URLs — see image_scales metadata gets @@images URLs (not thumbor) when indexed outside a request #14.
  3. Blob replacement changes a blob's TID. Any already-indexed direct Thumbor URL for that object points at the previous TID.

Since #17, uid URLs heal on traversal, so none of this is broken — but every healed request costs a 302 through Plone, which is precisely the hop image_scales exists to avoid.

What is missing

A maintenance job that periodically refreshes image_scales for only the objects that changed since the last run, rather than the all-or-nothing full walk that exists today.

Rough shape:

  • select candidates by modification watermark (not a full catalog walk)
  • reindex image_scales only — no other indexes, no object rewrite
  • chunked and resumable, with the watermark persisted per run
  • safe to run unattended on a site with ~139k objects

Relationship to other issues

Out of scope

Changing when or how URLs are generated. This is purely about keeping already-correct generation reflected in the catalog over time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions