Add per-shard CLI and Metadata_Imaging_ImageID propagation - #50
Merged
Conversation
ZedProfiler is the feature extractor the NF1 pipeline dispatches per
well/FOV shard via SLURM sbatch. This commit adds the command that process
runs and the identifier column that makes shards warehouse-joinable.
CLI (src/zedprofiler/cli.py):
- "ZedProfiler run" subcommand (argparse): repeatable --image/--label
NAME=PATH flags, --anisotropy-spacing Z Y X, identifier fields
(--patient-tumor/--plate/--well/--field), --out-dir, a --features
selector, repeatable --feature TYPE[,key=value,...] advanced requests,
--skip-existing, and --force.
- Reuses the six compute_* featurizers; builds the shared loader via
ImageSetLoader.from_image_dict with the identifier fields.
- Restartable/idempotent: deterministic output paths, --skip-existing
filters before any image is read (a finished shard re-run skips I/O
entirely), and atomic writes (temp + os.replace) so a crashed shard
never leaves a partial file that --skip-existing would mistake for
complete.
- Fixes the orphaned/mis-cased console script to zedprofiler.cli:trigger.
Identifiers (src/zedprofiler/identifiers.py):
- build_image_id(patient_tumor, plate, well, field) -> deterministic
Metadata_Imaging_ImageID; single source of truth for the format.
Loaders (src/zedprofiler/IO/loading_classes.py):
- ImageSetConfig carries patient_tumor/plate/well/field with an image_id
property; ImageSetLoader exposes image_id (falls back to image_set_name).
- New from_image_dict classmethod builds a multi-channel loader from an
in-memory {key: ndarray} dict (the path the CLI needs); it derives
compartments/image names directly from the declared label keys so it is
self-contained and correct independent of get_compartments.
Featurizers (6 modules):
- Each emits Metadata_Imaging_ImageID before Metadata_Experiment_ImageSet.
Feature values are unchanged; only a metadata column is added.
Feature writing:
- save_features_as_parquet gains an opt-in atomic flag used by the CLI.
End-to-end CLI tests self-skip when the CellProfiler 3D tutorial data is
absent (it lands via a separate data commit), so the CLI test module stays
green everywhere and runs in full wherever the data is available.
ZedProfiler is the feature extractor the NF1 pipeline dispatches per
well/FOV shard via SLURM sbatch. This commit adds the command that process
runs and the identifier column that makes shards warehouse-joinable.
CLI (src/zedprofiler/cli.py):
- "ZedProfiler run" subcommand (argparse): repeatable --image/--label
NAME=PATH flags, --anisotropy-spacing Z Y X, identifier fields
(--patient-tumor/--plate/--well/--field), --out-dir, a --features
selector, repeatable --feature TYPE[,key=value,...] advanced requests,
--skip-existing, and --force.
- Reuses the six compute_* featurizers; builds the shared loader via
ImageSetLoader.from_image_dict with the identifier fields.
- Restartable/idempotent: deterministic output paths, --skip-existing
filters before any image is read (a finished shard re-run skips I/O
entirely), and atomic writes (temp + os.replace) so a crashed shard
never leaves a partial file that --skip-existing would mistake for
complete.
- Fixes the orphaned/mis-cased console script to zedprofiler.cli:trigger.
Identifiers (src/zedprofiler/identifiers.py):
- build_image_id(patient_tumor, plate, well, field) -> deterministic
Metadata_Imaging_ImageID; single source of truth for the format.
Loaders (src/zedprofiler/IO/loading_classes.py):
- ImageSetConfig carries patient_tumor/plate/well/field with an image_id
property; ImageSetLoader exposes image_id (falls back to image_set_name).
- New from_image_dict classmethod builds a multi-channel loader from an
in-memory {key: ndarray} dict (the path the CLI needs); it derives
compartments/image names directly from the declared label keys so it is
self-contained and correct independent of get_compartments.
Featurizers (6 modules):
- Each emits Metadata_Imaging_ImageID before Metadata_Experiment_ImageSet.
Feature values are unchanged; only a metadata column is added.
Feature writing:
- save_features_as_parquet gains an opt-in atomic flag used by the CLI.
Tests:
- test_real_world_data.py colocalization loader switched to from_image_dict
so the new image_id column is populated for the existing colocalization
end-to-end test.
- CLI end-to-end tests run against the CellProfiler 3D tutorial data present
on main and self-skip if that data is absent on other branches.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #50 +/- ##
=======================================
Coverage ? 94.47%
=======================================
Files ? 17
Lines ? 1648
Branches ? 0
=======================================
Hits ? 1557
Misses ? 91
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d33bs
marked this pull request as ready for review
August 6, 2026 13:51
MikeLippincott
approved these changes
Aug 6, 2026
Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com>
Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com>
Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com>
Integrates the rewritten origin/main (9bab886: per-shard CLI + Metadata_Imaging_ImageID propagation) into the cli branch. No content change: cli already contained all of upstream's current content (image_id work, fov rename, get_compartments fix, real-world data) plus the CLI feature-spec/coverage work, so the merge only records the integration.
Integrate the benchmarking framework and featurization performance refactor from upstream (PR WayScience#48) with the per-shard CLI and Metadata_Imaging_ImageID propagation work on this branch. The two lines of work touched disjoint regions and merged cleanly, but the upstream BenchmarkImageSet mock lacked the image_id attribute that the cli featurization functions now read. Add image_id to BenchmarkImageSet (mirroring ImageSetLoader's fallback to image_set_name) and regenerate the six accuracy-lock signatures in test_benchmark_contracts.py, which now include the Metadata_Imaging_ImageID column.
Member
Author
|
thank you @MikeLippincott! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR focuses on productionizing the CLI for use with orchestration for the NF1 project. We add an image_id arg in many spots to help enforce data provenance, which is important for warehousing and may fit in nicely with task parallelization.
What kind of change(s) are included?
Checklist
Please ensure that all boxes are checked before indicating that this pull request is ready for review.