Skip to content

Option to Create Only One Preservation Event per Ingest #3752

Description

@006627

Ingest runs as a configurable sequence of processing steps, and each step currently writes its own preservation event, so a single SIP produces one event per step. There should be an option to consolidate this into a single preservation event per ingest, covering whichever steps actually ran, with enough detail to tell whether each one succeeded, failed, or was skipped, along with the agents involved. A SIP update goes through the same ingest process, so it should behave the same way.

This should be opt-in and off by default, must not affect ingests that already happened, and must not change how the same processing steps behave when run on their own outside of an ingest.

Acceptance Criteria

  • An ingest can be configured to produce a single preservation event instead of one per step.
  • That single event covers whichever steps actually ran as part of the ingest, recording for each one whether it succeeded, failed, or was skipped, and identifying the agent responsible.
  • A SIP update produces a single consolidated event the same way a new ingest does.
  • The option is off by default; existing behavior (one event per step) is unchanged unless it's turned on.
  • Preservation events already written before this option existed are left untouched.
  • The same processing step run on its own, outside of an ingest, still produces its own individual event regardless of this setting.

Notes

Every ingest step is implemented as its own plugin, and each one currently writes its own event independently — there's no single place today that already sees everything that happened across an entire ingest and could just be changed once. Delivering this means changing each step's own event-writing individually, not flipping one central switch.

The single event's structure needs to work within what a PREMIS event can actually represent and what the existing event-handling code can already provide, added to without breaking it — this is new, additive behavior, not a replacement of how events are built today for anything that isn't using this option.

Possible Approach (reasoned starting point, still needs validation on the specifics)

The mechanism every plugin already uses to write a preservation event (PluginHelper.createPluginEvent) is used far beyond ingest, so it shouldn't be changed directly — doing so would put every one of its callers at risk for the sake of one feature. Instead:

  1. A new, off-by-default option, set by the ingest workflow on its own parameters. It reaches every step automatically through the parameter-merging already done today when a plugin runs as part of an ingest — no new propagation code needed.
  2. A new method is added on PluginHelper, alongside the existing one, reusing the same underlying logic (agent resolution, outcome messaging). When the option is off, it simply delegates to the existing method, unchanged. When the option is on, it captures a snapshot of what would have been written — into a small, purpose-built registry, grouped by job and by the specific object the snapshot belongs to — instead of writing it.
  3. Each step plugin is migrated to the new method individually, one at a time — a small, mechanical change per plugin, not a coordinated rewrite, since the existing method is never touched and keeps working for anything not yet migrated. During a partial rollout, a migrated step contributes to the one consolidated event, while a step not yet migrated keeps writing its own individual event alongside it — an expected interim state on the way to full coverage, not a bug.
  4. The decision to actually write the one consolidated event is made explicitly by the ingest workflow, at the exact point in its own existing loop where it already writes today's per-object "ingest end" event — never guessed at by the writing mechanism itself. At that point, for that one object, it reads back what was captured, builds one real event from it, writes it, and clears that object's entry immediately.
  5. When the option is off, or for a step plugin run on its own outside any ingest, nothing changes anywhere — the option is never set on that plugin's parameters in the first place.

What's still open, and should be validated against the code rather than assumed: the exact shape of the registry and its concurrency handling, and precisely how an abandoned entry gets cleaned up if an ingest fails partway through.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions