Python Pipeline

---

title: "Python Data Processing Pipeline" version: "1.0" last_updated: "2026-05-18" status: "active"

Python Data Processing Pipeline

Context: Codebase-Context

1. Orchestration (run_lidc.py, run_lungx.py)

The pipeline uses the Ruffus library to orchestrate a directed acyclic graph (DAG) of computational tasks. This ensures reproducibility and allows for parallel execution.

LIDC Pipeline Stages (run_lidc.py)

1. task_originate: Initializes the pipeline by loading patient lists and setting up directory structures.
2. task_image_resample: Resamples the CT images to isotropic resolution (1mm x 1mm x 1mm) using a Dockerized Python script.
3. task_check_nodules: Identifies and organizes nodule annotations.
4. task_extract_nodule_labels: Extracts nodule masks based on physician annotations (Phy1-Phy4).
5. task_staple_comparison: Applies STAPLE (Simultaneous Truth and Performance Level Evaluation) to combine multiple physician annotations into a single consensus mask.
6. task_segment_nodule: Generates automated segmentations (GrowCut and CIP) using the consensus mask.
7. task_feature_extraction: Extracts radiomics features from the segmentations using a Dockerized tool.
8. task_feature_organization: Merges all extracted features into a final comprehensive CSV file.

2. Data Handling (tasks/data.py)

• dicom_to_nrrd_convert: Uses a Docker container (wookjinchoi/radiomics-tools DICOM-RT2NRRDConverter) to convert raw DICOM files to the NRRD format.
• load_scan_list / originate_pylidc: Integrates with the pylidc library to query the LIDC database, extract bounding boxes, and construct the initial metadata tables.

3. Segmentation (tasks/segmentation.py)

Implements sophisticated segmentation combinations:

• GrowCut (_growcut_segmentation): A cellular automaton-based interactive segmentation method.
• CIP (Convexity/Concavity Preserving): Specialized segmentation to handle lobulated and spiculated margins.
• Merging: Features logic (_merge_gc_cip) to combine outputs from multiple segmentation engines.

4. Feature Extraction (tasks/feature_extraction.py)

• Standard radiomics feature extraction is offloaded to a Docker container (wookjinchoi/radiomics-tools FeatureExtraction), which wraps PyRadiomics.
• It operates on the resampled 1mm isotropic NRRD files and segmentations.