Revisions and enable_async_checksum=True: double checksum check for files in revision

[rwiermer]

Is there an existing bug report for this?

• I have searched the existing bug reports

Current Behavior

Currently, new revision copies seemed to be checksummed twice - once inline during the data_object.copy() operation and during an check if enable_async_checksum is True.

Result: long job queues if many files are added at the same time.

In policies.py

# RW: this call is guarded by revision_eligible() check 
    revisions.resource_modified_post_revision(ctx, instance_name, zone, path)

# RW: this call is not guarded - checksum is calculated independent of whether the file is the result of a 
revision copy or not
# Option to fix: check whether this file is the result of a revision copy or not. Alternatively: only calculate checksum for files in the revision_eligible check (which would also exclude temp and other blocked files) 
    if config.enable_async_checksum:
        # Base64 encode the path.
        encoded_path = base64.b64encode(path.encode()).decode()
        # Calculate and verify checksum of data object.
        ctx.delayExec(
            f"<PLUSET>{config.async_checksum_delay_time}s</PLUSET><INST_NAME>irods_rule_engine_plugin-irods_rule_language-instance</INST_NAME>",
            f"rule_verify_checksum('{encoded_path}')",
            "")

In utils/data_object.py

#RW: checksum is enforced during copy
    msi.data_obj_copy(ctx,
                      path_org,
                      path_copy,
                      'numThreads=1++++verifyChksum={}'.format('++++forceFlag=' if force else ''),
                      irods_types.BytesBuf())

Expected Behavior

Data object resulting from a revision copy should only be checksummed once

Steps to Reproduce

Config:

• revision enabled
• enable_async_checksum=enable

Enable logging, check for which files a async checksum job is run.

Environment

Yoda 2.1 (stable)

Additional Context

No response

[stsnel]

Thank you for reporting this issue. We've logged this issue in our internal tracker as YDA-7164. I'll check with our product owner if we can get a fix in for one of the next bugfix releases.