Log EventDataIds through the SND event and attribute data ETL steps - #985
Open
labkey-martyp wants to merge 4 commits into
Open
Log EventDataIds through the SND event and attribute data ETL steps#985labkey-martyp wants to merge 4 commits into
labkey-martyp wants to merge 4 commits into
Conversation
The _SND Event Data step clears attribute values and only the _SND Attribute Data step restores them, and each step picks its rows independently, so a mismatch silently leaves event data with no attributes. Logging the ids on both sides makes the gap visible in the job log.
The exp.Object lookup added for this branch's id logging ran unguarded inside mergeRows, so a failure reading it would abort a merge that previously did no reads there at all. It now runs inside logAttributeDataToBeCleared, which logs a warning and lets the merge continue. The lookup is also skipped once the batch exceeds MAX_LOGGED_IDS, the same threshold above which logIds suppresses the id list, capping the chunked queries at ten round trips instead of the hundreds a full merge batch would spend to produce a bare count. MAX_LOGGED_IDS is public now so both sites share one constant.
These two lines carried the property name but nothing identifying the row, so a single attribute dropped from an otherwise healthy EventDataId could not be joined back to the id sets the rest of this branch logs. Per-attribute loss was invisible to every one of them, since they all reconcile at EventDataId granularity and an id with any resolving value counts as written.
The event data step logged ids only from mergeRows, and only the subset that already carried attribute values, so there was nothing to diff against the id sets the attribute data step logs. importRows and deleteRows now log theirs as well: a newly inserted row the _SND Attribute Data step misses ends up just as stripped as a cleared one, and a deleted row otherwise reads as an attribute data miss. Everything this branch adds now logs at debug, which the ETL job logger runs at by default. The pre-existing Begin/End updating exp.ObjectProperty lines are back at info, and the two diagnostics that cost real work - the exp.Object lookup for values about to be cleared, and the source ordering check - are skipped entirely when debug is off. MAX_LOGGED_IDS drops to 2000, below the 5000 row ETL batch size, so the full batches of an initial load suppress the id lists while the smaller batches of an incremental run keep them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
Make it possible to tell from the ETL job log which event data rows ended up holding no attribute values. The _SND Event Data step clears attribute values as a side effect of its merge and only the _SND Attribute Data step restores them; because the two steps decide independently which rows to process, a mismatch leaves event data silently stripped. Newly inserted rows carry the same exposure, arriving with no attribute values and depending on that same second step to supply them. Until now nothing in the log identified the rows either step handled, so the problem could only be found by querying the database after the fact.
Related Pull Requests
None.
Changes