Add per-shard CLI and Metadata_Imaging_ImageID propagation - #49
Closed
d33bs wants to merge 1 commit into
Closed
Conversation
ZedProfiler is the feature extractor the NF1 pipeline dispatches per
well/FOV shard via SLURM sbatch. This commit adds the command that process
runs and the identifier column that makes shards warehouse-joinable.
CLI (src/zedprofiler/cli.py):
- "ZedProfiler run" subcommand (argparse): repeatable --image/--label
NAME=PATH flags, --anisotropy-spacing Z Y X, identifier fields
(--patient-tumor/--plate/--well/--field), --out-dir, a --features
selector, repeatable --feature TYPE[,key=value,...] advanced requests,
--skip-existing, and --force.
- Reuses the six compute_* featurizers; builds the shared loader via
ImageSetLoader.from_image_dict with the identifier fields.
- Restartable/idempotent: deterministic output paths, --skip-existing
filters before any image is read (a finished shard re-run skips I/O
entirely), and atomic writes (temp + os.replace) so a crashed shard
never leaves a partial file that --skip-existing would mistake for
complete.
- Fixes the orphaned/mis-cased console script to zedprofiler.cli:trigger.
Identifiers (src/zedprofiler/identifiers.py):
- build_image_id(patient_tumor, plate, well, field) -> deterministic
Metadata_Imaging_ImageID; single source of truth for the format.
Loaders (src/zedprofiler/IO/loading_classes.py):
- ImageSetConfig carries patient_tumor/plate/well/field with an image_id
property; ImageSetLoader exposes image_id (falls back to image_set_name).
- New from_image_dict classmethod builds a multi-channel loader from an
in-memory {key: ndarray} dict (the path the CLI needs); it derives
compartments/image names directly from the declared label keys so it is
self-contained and correct independent of get_compartments.
Featurizers (6 modules):
- Each emits Metadata_Imaging_ImageID before Metadata_Experiment_ImageSet.
Feature values are unchanged; only a metadata column is added.
Feature writing:
- save_features_as_parquet gains an opt-in atomic flag used by the CLI.
End-to-end CLI tests self-skip when the CellProfiler 3D tutorial data is
absent (it lands via a separate data commit), so the CLI test module stays
green everywhere and runs in full wherever the data is available.
Member
Author
|
Wrong branch 😓 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR focuses on productionizing the CLI for use with orchestration for the NF1 project. We add an image_id arg in many spots to help enforce data provenance, which is important for warehousing and may fit in nicely with task parallelization.
What kind of change(s) are included?
Checklist
Please ensure that all boxes are checked before indicating that this pull request is ready for review.