Skip to content

PAPER_EXPLAINED.md

Latest commit

 

History

History
537 lines (369 loc) · 52.5 KB

PAPER_EXPLAINED.md

File metadata and controls

537 lines (369 loc) · 52.5 KB
Raw

Region of Interest Detection in Melanocytic Skin Tumor Whole Slide Images — Nevus and Melanoma

A detailed, code-grounded illustration of the paper and its accompanying implementation (PCLA-3C).

Bibliographic information

Field Value
Title Region of Interest Detection in Melanocytic Skin Tumor Whole Slide Images—Nevus and Melanoma
Authors Yi Cui, Yao Li, Jayson R. Miedema, Sharon N. Edmiston, Sherif W. Farag, James Stephen Marron, Nancy E. Thomas
Affiliation University of North Carolina at Chapel Hill (Economics; Statistics & Operations Research; Pathology & Laboratory Medicine; Lineberger Comprehensive Cancer Center; Eshelman School of Pharmacy; Biostatistics; Dermatology)
Venue Cancers 2024, 16, 2616 (MDPI)
DOI 10.3390/cancers16152616
Dates Received 29 June 2024; Revised 17 July 2024; Accepted 19 July 2024; Published 23 July 2024
Model name PCLA-3C (Patch CLAssifier, 3-Class)
Code https://github.com/cyMichael/ROI_Detection (this repository is a modified/automated version)
License MIT (code); CC BY 4.0 (paper)

Precursor work (sourced from the code repository; not stated in the paper). An extended abstract of this work was accepted at Medical Imaging Meets NeurIPS (Med-NeurIPS) 2022 (Poster Session).

Table of contents

  1. Executive summary
  2. Clinical background and motivation
  3. Problem definition
  4. Dataset: the UNC Melanocytic Tumor Dataset
  5. Method: the PCLA-3C framework
  6. The code: repository walkthrough and paper-to-code mapping
  7. Results
  8. Misclassification analysis
  9. Discussion, related work, and limitations
  10. Reproducibility and computational environment
  11. Reconciling the reported numbers (notes on internal inconsistencies)
  12. Glossary and abbreviations
  13. Key references

1. Executive summary

The paper addresses two coupled tasks on digitized hematoxylin-and-eosin (H&E) whole slide images (WSIs) of melanocytic skin tumors:

  1. Slide-level classification — decide whether a slide shows melanoma (malignant) or nevus (benign).
  2. Region-of-interest (ROI) detection — localize the diagnostically relevant tumor region within the slide, without requiring exhaustive pixel-level ground truth during training.

The core method, PCLA-3C, is deliberately simple and patch-based:

  • Each gigapixel WSI is tiled into non-overlapping 256 × 256 patches at 20× magnification.
  • A VGG16 convolutional neural network is fine-tuned as a 3-class patch classifier with classes {Melanoma, Nevus, Other}. The third class ("Other") absorbs non-tumor background tissue, which lets the model learn a meaningful ROI signal even though annotations are incomplete.
  • At inference, all patches of a slide are classified. The slide label is decided by majority vote between melanoma-patch and nevus-patch counts (ignoring "Other"). Patches are then ranked by the winning class's score to produce a predicted ROI.
  • ROI quality is measured by Intersection over Union (IoU) against pathologist annotations on a held-out test set.

Principal quantitative results (test set of 26 WSIs, model trained on the full 134-WSI training set):

Metric PCLA-3C CLAM (baseline)
Patch classification accuracy 0.892 — (CLAM does not classify patches)
Slide classification accuracy 0.923 0.692
ROI IoU 0.382 0.112

The central methodological contribution is exploiting partial annotations (which the authors describe as a semi-supervised setting) together with an explicit background class to train a patch classifier that simultaneously yields accurate slide classification and a usable ROI map — substantially outperforming the state-of-the-art weakly-supervised baseline, CLAM, on this diagnostically challenging melanocytic dataset.

2. Clinical background and motivation

Why melanocytic tumors matter. The paper opens with epidemiological context: the American Cancer Society projected that in 2022 an estimated 99,780 invasive and 97,920 in-situ melanoma cases would be newly diagnosed in the US, with ~7,650 deaths. The gold standard for diagnosis is a pathologist's visual assessment of H&E-stained tissue sections.

The diagnostic problem. The classification of melanocytic tumors is recognized as diagnostically challenging. Multiple studies report substantial inter-observer discordance among pathologists — the paper cites literature reporting 25–26% discordance between individual pathologists for distinguishing a benign nevus from a malignant melanoma. The clinical stakes are asymmetric:

  • Under-diagnosis of melanoma delays surgical excision and adjuvant therapy, risking metastasis.
  • Over-diagnosis leads to unnecessary procedures and toxic adjuvant therapies.

The opportunity. Computational pathology — combining high-quality histopathology images with deep learning — offers the prospect of faster, cheaper, and more consistent assessment. Prior deep-learning work in pathology has scored tumor grade (e.g., breast cancer), performed histologic pattern classification (e.g., lung adenocarcinoma), and predicted treatment response. However, the authors note that prior literature on skin cancer did not address ROI detection for melanocytic tumors and achieved only limited classification accuracy in distinguishing among tumor types.

The goal. Build a deep-learning method that (a) automatically detects the ROI (the region a pathologist would circle) and (b) classifies the slide accurately — leveraging the partial annotations pathologists already produce, rather than requiring complete pixel-level labels.

3. Problem definition

Two outputs are required per WSI:

  • Slide classification $\hat{y} \in {\text{Melanoma}, \text{Nevus}}$.
  • ROI mask $B$ — the set of patches predicted to lie within the diagnostically relevant region, to be compared against the pathologist-annotated region $A$.

The supervision challenge. During training, only slide-level labels and partial ROI annotations are available:

  • Every training slide has a slide-level label (melanoma vs. nevus).
  • Pathologists annotated tumor boundaries on some slides/regions but not all ROIs were annotated. Consequently, the training annotations are incomplete rather than erroneous, and cannot serve as a complete ground truth for evaluating detection on the training data.
  • Crucially, because annotation is incomplete, an unannotated region may still contain tumor — so naively labeling "everything outside an annotation" as non-tumor would inject label noise.

The key idea. Rather than treating this as pixel-level segmentation with exhaustive labels (which do not exist), PCLA-3C reframes it as patch classification with three classes, where a manually curated "Other" class provides clean negative (background/non-ROI) examples. Slide classification and ROI detection then both fall out of the same per-patch scores. This is described in the paper as leveraging "partial information from annotations, also called semi-supervised learning."

4. Dataset: the UNC Melanocytic Tumor Dataset

Property Value
Total WSIs 160 H&E whole slide images of primary tumors
Class balance 86 melanoma (malignant) + 74 nevi (benign)
Training set (80%) 134 WSIs — 71 melanoma + 63 nevus (per main text)
Test set (20%) 26 WSIs — 15 melanoma + 11 nevus
Scanner Aperio ScanScope Console, 20× magnification
Annotation tool Aperio ImageScope Console; boundaries exported as XML with region vertex coordinates
Test annotations All 26 test slides manually annotated (melanoma/nevi circled on the glass slides) by an expert dermatopathologist (PAG — Pamela A. Groben)
Ethics UNC Chapel Hill IRB #22-0611; waiver of informed consent [45 CFR 46.116(d)] and HIPAA authorization [45 CFR 164.512(i)(2)(ii)]

Effective data size. Although only 160 slides exist, each gigapixel WSI yields thousands of patches, so the effective training set is hundreds of thousands of patches — this is what makes CNN training feasible and improves generalization across regions within a slide.

Annotation asymmetry between train and test — a critical design point:

  • Training slides were used without complete ground-truth annotations (only partial region information). This mirrors the real-world situation: pathologists annotate for diagnosis, not for exhaustive ML labeling.
  • Test slides were kept unchanged with true, complete annotations, so evaluation of both slide classification and IoU is trustworthy.

This asymmetry is what makes the reported results meaningful: the model never saw complete detection labels, yet it is evaluated against complete ones.

5. Method: the PCLA-3C framework

The framework (Figure 2 of the paper) has two stages — Training and Testing — each composed of well-defined steps. The mapping to code files is given in Section 6.

5.1 Preprocessing: color normalization

Scans performed in different labs — or even the same lab at different times — vary in color and quality. A CNN may exploit these spurious color differences rather than genuine tissue morphology. To prevent this, all WSIs are color-normalized into a common space using the two established stain-normalization methods the paper cites: Ruifrok & Johnston color deconvolution [28] and Macenko et al. [29]. This improves the robustness of both training and downstream quantitative analysis.

Both methods rest on the Beer–Lambert law. For incident light of intensity $I_0$ transmitted through stained tissue at measured intensity $I$ (per RGB channel), the optical density (OD) is

$$\text{OD} = -\log_{10}!\left(\frac{I}{I_0}\right),$$

and OD is linearly proportional to stain concentration: each stain has a characteristic, constant absorption (OD) signature across the three RGB channels. This linearity is what makes the stains separable.

Ruifrok & Johnston [28] — color deconvolution (fixed stain vectors). The specimen is modeled as a mixture of up to three stains (for H&E: hematoxylin, eosin, and a residual/background channel).

  1. Each stain $s$ is assigned a normalized OD vector — its RGB color signature — originally measured from control slides stained with a single stain.
  2. These vectors are stacked into a $3\times3$ stain matrix $M$ (one row per stain).
  3. For a pixel with measured optical density $\text{OD}$, the relationship is $\text{OD} = M^{\mathsf T} c$, where $c$ is the vector of per-stain concentrations. Because $M$ is (approximately) invertible, the concentrations are recovered by deconvolution: $c = (M^{\mathsf T})^{-1},\text{OD}$, which separates the RGB image into independent per-stain density channels.
  4. Normalization re-projects those concentrations onto a set of reference/standard stain vectors, so that slides processed under different conditions share a common stain appearance.

Macenko et al. [29] — automatic, data-driven stain estimation. Rather than using pre-measured stain vectors, this method estimates them per slide directly from the image:

  1. Convert RGB → OD and discard near-transparent background pixels (those with OD below a small threshold, e.g. $\beta = 0.15$).
  2. Because H&E optical-density vectors lie approximately in a 2-D plane, take the singular value decomposition (SVD) of the OD pixel matrix and keep the two directions with the largest singular values — the plane that best explains the stain variation.
  3. Project the pixels onto that plane, compute each pixel's angle, and take the robust extreme angles (e.g. the 1st and 99th percentiles) as the two stain vectors (hematoxylin and eosin) — estimated automatically, with no control slides.
  4. Solve the linear system (non-negative least squares) for the per-pixel stain concentrations, and take a robust maximum (e.g. the 99th percentile) as each stain's reference concentration.
  5. Normalize by rescaling every slide's stain concentrations to a chosen target (reference-slide) stain matrix and maximum concentrations, then recompose OD → RGB. This maps all slides into the reference color space, removing inter-scanner and inter-batch color variation while preserving tissue morphology.

In this study, one of these normalizations is applied to every WSI before patch extraction, so the downstream VGG16 classifier learns morphology rather than staining artifacts.

5.2 Patch extraction and labeling

  • Tiling. Slides are tiled into non-overlapping 256 × 256 patches at 20× magnification. Tissue detection first discards background/empty patches.
  • Data augmentation. Random crop, random horizontal flip, and normalization are applied so that edge features are preserved and the classifier generalizes.
  • Labeling rule (uses the slide-level label together with the XML annotations):
    • If the slide is a nevus, patches inside the annotated regions are labeled Nevus.
    • If the slide is a melanoma, patches inside the annotated regions are labeled Melanoma.
    • Patches outside annotated regions are labeled Other.
  • Handling incomplete annotations. Because not every ROI was annotated, some tumor patches lie outside the annotations and would be wrongly labeled "Other." To avoid this contamination, the authors manually extracted "Other" patches from curated regions rather than blindly labeling all extra-annotation patches as "Other."

The result is a labeled patch dataset with three classes: Melanoma, Nevus, Other.

5.3 Model and training

  • Architecture. A VGG16 CNN [10] is used as the base. The network's final fully connected classification layer is replaced with a three-unit output head, and the network is fine-tuned by backpropagation. (VGG is a classic deep CNN family; the paper also situates it among ZefNet, ResNet, and DenseNet.)
  • Output. For each patch the model returns three scores, one per class (melanoma, nevus, other).
  • Optimization (from the code): cross-entropy loss, Adam optimizer, learning rate 5 × 10⁻⁴, weight decay 1 × 10⁻⁴, with early stopping (patience 20) on validation loss.

5.3.1 Detailed model architecture (VGG16)

The paper states only that VGG16 is the base and that "the last layer's parameters" were changed; the per-layer specification below follows the canonical VGG16, configuration "D" (Simonyan & Zisserman [10]) as instantiated by torchvision. The concrete instantiate_model('vgg16', True, 3) routine is not shown in the released scripts, so the specifics are reconstructed from the paper together with standard torchvision/VGG conventions.

  • Input. An RGB patch of 224 × 224 × 3 during training (after RandomCrop(224, padding=4)) and 256 × 256 × 3 at inference. Channel-wise normalization uses the dataset-specific H&E statistics — mean (0.6632, 0.4123, 0.5529), std (0.1618, 0.1749, 0.1478) — not ImageNet statistics.
  • Feature extractor. Thirteen 3 × 3 convolutions (stride 1, padding 1), each followed by ReLU, organized into five blocks separated by 2 × 2 max-pooling (stride 2):
Block Layers (each Conv 3×3 + ReLU) Channels Spatial output (224 input)
1 Conv, Conv, MaxPool 64 112 × 112
2 Conv, Conv, MaxPool 128 56 × 56
3 Conv, Conv, Conv, MaxPool 256 28 × 28
4 Conv, Conv, Conv, MaxPool 512 14 × 14
5 Conv, Conv, Conv, MaxPool 512 7 × 7
  • Adaptive pooling. AdaptiveAvgPool2d((7, 7)) fixes the feature map to 7 × 7 × 512 regardless of input size. This is precisely why a 256 × 256 test patch (which pools to 8 × 8 × 512 before this layer) is accepted without error despite the 224 × 224 training crop.
  • Classifier head. Flatten (25 088) → Linear(25088, 4096) → ReLU → Dropout(0.5) → Linear(4096, 4096) → ReLU → Dropout(0.5) → Linear(4096, 3). The original ImageNet head's final Linear(4096, 1000) is replaced by Linear(4096, 3) — the "changed last layer" of the paper. The three outputs are the raw class logits {Melanoma (0), Nevus (1), Other (2)} — no softmax is applied (see §6.2).
  • Depth. 13 convolutional + 3 fully connected = 16 weight layers, hence "VGG-16." With the 3-way head the network has ≈ 134 million trainable parameters (vs. ≈ 138 M for the 1000-class original).
  • Transfer learning. The second argument of instantiate_model('vgg16', True, 3) denotes ImageNet-pretrained initialization; the backbone is fine-tuned end-to-end on the H&E patches rather than trained from scratch.
  • Training configuration. CrossEntropyLoss on the logits; Adam (lr = 5e-4, weight_decay = 1e-4); batch size 100; up to 20 epochs. Note that early-stopping patience = 20 equals n_epochs = 20, so under the default configuration early stopping can never actually fire — training runs the full 20 epochs and every per-epoch checkpoint is retained for later selection (§6.3).

5.4 Slide classification (testing stage)

At inference, all patches from a WSI are fed to the trained classifier, and each patch is assigned the class with the highest predicted score (the argmax over the three class scores). Then:

  • "Other" patches are ignored.
  • The slide label is decided by majority vote between the remaining two classes: if the number of patches predicted melanoma exceeds the number predicted nevus, the slide is classified melanoma; otherwise nevus (so that ties in patch counts are resolved in favor of nevus, consistent with the strict inequality class_mel > class_nev in the code).

5.5 ROI ranking and detection

Once a slide's class is decided, all patches are ranked by the predicted score of the winning class:

  • For a melanoma slide, rank patches by their melanoma score.
  • For a nevus slide, rank patches by their nevus score.

The predicted ROI is then the set of top-ranked patches. How many patches to include is governed by the annotated ratio $\beta$.

5.6 The two key metrics: β and IoU

Annotated ratio $\beta$. For a slide $C$ with annotated region $A$:

$$\beta = \frac{A_p}{C_p}$$

where $A_p$ is the number of patches inside the annotated region $A$, and $C_p$ is the total number of patches extracted from the slide $C$. Equivalently, $\beta$ is the proportion of a slide's extracted patches (by patch count, not area) that fall within the pathologist-annotated region.

Top-$n\beta$ selection. With $n$ = total number of patches in the slide, the model designates the top $n\beta$ patches (by predicted score) as the predicted ROI $B$. Example from the paper: if $\beta = 0.2$, then 20% of the slide is annotated ROI, so the model predicts its top 20% highest-scoring patches as the ROI.

Intersection over Union (IoU). Because the framework is patch-based, IoU is computed by counting patches:

$$\text{IoU} = \frac{\underline{AB}_p;(|A \cap B|)}{\overline{AB}_p;(|A \cup B|)}$$

where $\underline{AB}_p$ = number of patches in the intersection $A \cap B$, and $\overline{AB}_p$ = number of patches in the union $A \cup B$. Here $A$ is the annotated (ground-truth) region and $B$ is the predicted/highlighted region.

Equivalently (and as implemented in analysis.py):

$$\text{IoU} = \frac{\text{num_inregion_highlight}}{\text{num_inregion} + \text{num_highlight} - \text{num_inregion_highlight}}$$

  • num_inregion = patches inside the annotation ($|A|$).
  • num_highlight = patches predicted as ROI ($|B|$).
  • num_inregion_highlight = patches both inside the annotation and predicted ($|A \cap B|$).

5.7 Visualization (three map types)

The predicted scores drive three complementary visualizations (Figure 3):

  1. Overlay map. Highlights the top-ranked patches while masking the rest of the slide with a transparent blue tint. When ground-truth XML is available the threshold is set from $\beta$ (via get_percent); with --no_xml it falls back to --percent (default 0.2). The highlighted fraction is therefore approximately $\beta$ — not exactly, owing to the per-class-threshold and argmax gating described in §6.2 — so the highlighted region closely approximates the predicted ROI. This is the most direct visualization of "what the model considers ROI."
  2. Boundary map. Draws the boundary of the largest ROI cluster. Highlighted patches are grouped with the OPTICS density-based clustering algorithm [30]; the largest cluster's outline is then rendered (via an alpha-shape hull). This shows how precisely the model captures tumor margins.
  3. Heatmap. A continuous color gradient (matplotlib coolwarm) over predicted scores: red = high predicted score, blue = low. This conveys the model's confidence across the whole slide.

Figure 1 in the paper illustrates the end goal: the pathologist's ROI is annotated with black dots on the left; PCLA-3C reproduces it as a green boundary on the right.

6. The code: repository walkthrough and paper-to-code mapping

6.1 Repository layout

understand-deeplearningproject/
├── paper/cancers-16-02616.pdf     # the paper
└── codes/                         # all source; run commands from here
    ├── extract_patches_3class.py  # Stage 1: patch extraction + labeling
    ├── method_pcla_3class.py      # Stage 2: train VGG16 3-class classifier
    ├── score_pcla_3class.py       # Stage 3: score all patches, slide classification
    ├── visual.py                  # Stage 4: overlay / boundary / heatmap
    ├── analysis.py                # Stage 5: IoU computation
    ├── automated_pipeline.py      # config-driven orchestrator (subprocess wrapper)
    ├── batch_process.py           # run the pipeline over multiple configs
    ├── config.yaml / config_example.yaml
    ├── setup.py / requirements.txt
    └── utils/                     # CLAM-derived helpers (not referenced by the active pipeline)

Each stage is an independent argparse script; stages communicate only through files on disk (patch PNGs, HDF5 score files, text logs, and CSVs).

6.2 The five-stage pipeline

Stage 1 — extract_patches_3class.py (patch extraction and labeling)

Implements §5.2. Key functions:

  • makemask(w, h, xml_path) — parses the Aperio XML, iterates over each <Region>'s <Vertex> coordinates, and rasterizes the polygons into a binary annotation mask via cv2.fillPoly. This is the code realization of "the annotated region $A$." Note that makemask() exists in both extract_patches_3class.py and visual.py with different internal conventions: the extraction copy allocates np.zeros((h, w)) and is called makemask(w, h, ...), whereas the visual copy allocates np.zeros((w, h)) but is called makemask(image.height, image.width, ...). Both ultimately yield a (height × width) mask, but the parameter order and names differ between files.
  • checkinout(patch_mask, annotation_ratio=1.00) — decides whether a patch is in-region: a patch is kept if the fraction of annotated (nonzero) pixels within it meets the ratio threshold. During extraction the ratio is effectively hardcoded to 1.00 (fully inside): compute_w_loader calls checkinout(img_mask) without passing annotation_ratio, so the config.yaml value annotation_ratio=0.5 affects only visual.py, not extraction.
  • compute_w_loader(...) — iterates patch coordinates from the slide's HDF5 bag, checks each against the mask, and saves in-region patches as PNG files named {output}_{x}_{y}.png (the patch's coordinates are encoded in the filename).
  • make_dirs(...) — creates the feat_dir/{train,val,test}/{Melanoma,Nevi,Other}/ directory tree that the training stage consumes.

The two annotation sources map directly to the labeling rule: --xml_annotation_new determines the Melanoma/Nevus label for in-region patches, while --xml_annotation_other routes patches into the Other class (the manually curated background of §5.2).

Stage 2 — method_pcla_3class.py (train the PCLA-3C classifier)

Implements §5.3. Highlights:

  • Loads the patch tree with torchvision.datasets.ImageFolder over {train,val,test}. Because ImageFolder assigns class indices alphabetically, the mapping is {Melanoma: 0, Nevi: 1, Other: 2} — consistent with the decision logic in later stages.
  • Training transform: RandomCrop(224, padding=4) → RandomHorizontalFlip → ToTensor → Normalize(mean=(0.6632, 0.4123, 0.5529), std=(0.1618, 0.1749, 0.1478)). (Note the 224-pixel crop for training vs. full 256-pixel patches at test time; VGG16's adaptive pooling accepts both.)
  • Model: instantiate_model('vgg16', True, num_class=3) — the VGG16 base with a 3-way head.
  • Training loop: cross-entropy loss, Adam (lr=5e-4, weight_decay=1e-4), validation each epoch, early stopping with patience=20. Checkpoints are saved as {exp_name}_epoch{N}_loss{L}_acc{A}.pt — the accuracy token in the filename is later parsed to auto-select the best model.

Stage 3 — score_pcla_3class.py (score patches, slide classification)

Implements §5.4–§5.5. Highlights:

  • For each slide, loads its patches from {slide_id}.h5 and runs the trained model, accumulating class_mel (count of argmax==0) and class_nev (count of argmax==1).
  • Slide label = majority vote: label = 0 (melanoma) if class_mel > class_nev, else 1 (nevus) — the direct code form of §5.4. check_correct encodes the Melanoma↔0 / Nevi↔1 convention to tally slide accuracy.
  • Per-slide output HDF5 results_dir/{exp_name}/score/{slide_id}.h5 with three datasets: scores (raw 3-class logits per patch), coord (patch coordinates), and pred (the slide-level predicted label). These files are the sole input to visualization and analysis. No softmax is applied anywhere in the pipeline: ScoreToColor, get_thrd, and ScoreToRank all threshold or rank these raw per-class logits (the visual.py variables named probs/prob0/prob1 are logits, not probabilities).
  • Writes classification_{exp_name}.csv, whose classification_result column is recomputed independently from the counts as Melanoma if mel_con > nev_con else nevi (note the lowercase nevi, inconsistent with the Melanoma/Nevi folder labels; ties fall to nevi). The slide-accuracy tally instead uses check_correct, comparing the argmax-derived label (0 = Melanoma, 1 = Nevi) against the ground-truth label_name. Overall slide-classification accuracy is printed.
  • The normalization mean/std are passed in as --mean1..3 / --std1..3 flags (supplied by config.yaml), so they must be kept in sync with the hardcoded training values.

Stage 4 — visual.py (overlay, boundary, heatmap)

Implements §5.6–§5.7. Highlights:

  • get_percent(...) computes the annotated ratio $\beta$ from the ground-truth mask (fraction of patches in-region) — this is the code form of $\beta = A_p/C_p$.
  • get_thrd(probs, percent) computes a per-class percentile threshold at the 100 − percent·100 percentile — separately, thrd0 on the melanoma logit and thrd1 on the nevus logit — thereby approximately selecting the top $n\beta$ patches.
  • ScoreToColor(...) (overlay/boundary) highlights a patch only when its winning-class logit exceeds that per-class threshold and its argmax agrees with the slide label; because of the extra argmax gate, the highlighted set is at most the top $n\beta$ patches, not exactly $n\beta$. It returns a discrete decision (0 = highlight-melanoma, 1 = highlight-nevus, −1 = not highlighted), whereas ScoreToRank(...) (heatmap) returns a continuous min–max-normalized percentile rank mapped to a coolwarm color.
  • Overlay: masks the whole slide with a transparent blue (plt.cm.coolwarm(0.05, alpha=0.5)) and reveals only highlighted patches; simultaneously logs per-slide num_inregion_highlight, num_inregion, and num_highlight to {exp_name}.txt — the exact counts IoU needs.
  • Boundary: clusters highlighted patches with sklearn.cluster.OPTICS, keeps the largest cluster, fits an alphashape hull, and draws the contour with cv2.drawContours (green line, as in Figure 1).
  • Heatmap: blends the coolwarm-colored score map with the original slide (original_image * 0.5 + heatmap_image * 0.5).
  • Outputs are pyramidal .tiff files written with pyvips, enabling WSI-scale rendering.

Stage 5 — analysis.py (IoU)

Implements the IoU metric of §5.6. It parses the {exp_name}.txt log written by visual.py and computes, per slide,

union = num_inregion + num_highlight − num_inregion_highlight
IoU   = num_inregion_highlight / union

skipping slides with zero in-region patches, and writes summary_iou_final.csv. (Note: analysis.py hardcodes reading a label file named melanomal.csv from the CSV directory and a single experiment name pcla_3class.)

6.3 Orchestration layer

  • automated_pipeline.py — the ROIDetectionPipeline class reads config.yaml and runs all five stages via subprocess, mapping config keys to CLI flags. It skips stages whose outputs already exist (skip_existing) and auto-selects a checkpoint by parsing the _acc token from checkpoint filenames. Because that token records per-epoch training accuracy, this selects the highest-training-accuracy checkpoint (ties broken arbitrarily by max()), which is not necessarily the best-validation model that early stopping would favor. --validate-only checks the config without running.
  • batch_process.py — runs the pipeline once per config_*.yaml in a directory and prints a success/failure summary.
  • config.yaml — single source of truth for paths and hyperparameters (architecture, batch_size=100, n_epochs=20, learning_rate=5e-4, patch_size=256, annotation_ratio=0.5, percent=0.2, and the normalization mean/std). config_example.yaml is the template; the validator rejects any path still set to a PATH_TO_* placeholder.

6.4 Paper → code mapping (quick reference)

Paper concept Code location
Color normalization (§5.1) Not included in repo (external "Step 0" preprocessing)
Tiling into 256×256 patches HDF5 patch bags consumed by extract_patches_3class.py (h5_to_patch_new)
Annotation mask $A$ from XML makemask() + cv2.fillPoly in extract_patches_3class.py / visual.py
In-region test (annotation ratio) checkinout()
Patch labels {Melanoma, Nevi, Other} make_dirs() tree + --xml_annotation_new / --xml_annotation_other
VGG16 3-class classifier (§5.3) method_pcla_3class.py (instantiate_model('vgg16', ...))
Data augmentation transforms.RandomCrop/RandomHorizontalFlip/Normalize
Slide classification by majority vote (§5.4) class_mel > class_nev in score_pcla_3class.py
Patch ranking by winning-class score (§5.5) ScoreToColor / ScoreToRank in visual.py
Annotated ratio $\beta = A_p/C_p$ get_percent() in visual.py
Top-$n\beta$ selection get_thrd() percentile threshold
IoU (§5.6) analysis.py (num_inregion_highlight / union)
Overlay / Boundary / Heatmap (§5.7) visual.py (--/--boundary/--heatmap)
OPTICS clustering for boundary sklearn.cluster.OPTICS in visual.py

7. Results

7.1 Method comparison — full training set (Table 1)

Both methods trained on the 134-WSI training set; evaluated on the 26-WSI test set.

Evaluation metric PCLA-3C CLAM
Patch classification accuracy 0.892 — (CLAM does not classify patches)
Slide classification accuracy 0.923 0.692
IoU 0.382 0.112

PCLA-3C raises slide classification accuracy from 69.2% to 92.3% and IoU from 0.112 to 0.382, a more than threefold increase in region-of-interest overlap relative to CLAM.

Comparison protocol. Both methods were trained on the identical 134-WSI split and evaluated on the identical 26-WSI test set. CLAM is label-free (slide labels only), whereas PCLA-3C additionally consumes the pathologists' partial ROI annotations; the comparison is therefore best read as quantifying the value that those partial annotations add, rather than as an architecture-only contest (see Reviewer 1 in reply_to_reviewer.md).

7.2 Confusion matrix (Table 2)

The paper designates nevi as the positive class and melanoma as the negative class, reflecting the clinical priority of correctly identifying benign cases to avoid unnecessary intervention.

True: Nevi True: Melanoma
Predicted: Nevi 20 0
Predicted: Melanoma 2 9

Reported summary metrics: accuracy 93.5%, sensitivity 81.8%, specificity 100%. The reported specificity of 100% indicates that, on this test set, no malignant (melanoma) case was misclassified as benign (nevus) — a property of particular clinical importance given the asymmetric cost of under-diagnosis. (See Section 11 for a note reconciling these counts with the 26-slide test set and the 92.3% figure.)

7.3 Robustness across training-set subsampling (Table 3)

To probe robustness, the model was retrained on random subsets of the original training set (20%, 40%, 60%, 80%), while the test set was held fixed. CLAM has no patch-classification row because it does not classify patches. Values are mean [95% CI].

Split Method Patch acc. Slide acc. IoU
20% PCLA-3C 0.6397 [0.5193, 0.7601] 0.7406 [0.6627, 0.8185] 0.3026 [0.2394, 0.3327]
CLAM — 0.6710 [0.6386, 0.7033] 0.0427 [0.0342, 0.0512]
40% PCLA-3C 0.7887 [0.7536, 0.8238] 0.8430 [0.8043, 0.8817] 0.3402 [0.3057, 0.3784]
CLAM — 0.6976 [0.6619, 0.7333] 0.0524 [0.0297, 0.0751]
60% PCLA-3C 0.8191 [0.7766, 0.8616] 0.8721 [0.8458, 0.8985] 0.3652 [0.3369, 0.3934]
CLAM — 0.7097 [0.6830, 0.7364] 0.0621 [0.0428, 0.0814]
80% PCLA-3C 0.8210 [0.7949, 0.8471] 0.8885 [0.8607, 0.9163] 0.3710 [0.3335, 0.4084]
CLAM — 0.7258 [0.7117, 0.7399] 0.1103 [0.0529, 0.1677]

Interpretation. PCLA-3C exhibits a gradual, monotonic decline in performance as the training set is reduced: even at a 20% training subset its IoU (0.30) already exceeds CLAM's best IoU (0.11, at the 80% subset), and its slide classification accuracy increases monotonically with training-set size (0.74 → 0.89). Accuracy and IoU both increase with more data, consistent with the value of the annotations. The paper additionally reports, at the 80% subset (107 WSIs), a patch-level accuracy of 0.7866 [0.761, 0.813] and slide-level accuracy of 0.885 [0.857, 0.914].

Takeaway. The consistent margin over CLAM across all subsampling levels indicates that the combination of partial annotations with a patch classifier that includes an explicit background class is the principal source of the observed improvement, and that PCLA-3C is comparatively data-efficient and robust.

8. Misclassification analysis

PCLA-3C misclassified only two slides in the test set. Both were true nevi predicted as melanoma (consistent with the specificity being higher for the malignant class):

  • Figure 4 case — not a typical nevus: it exhibits features of a pigmented spindle cell nevus, one of the recognized diagnostic challenges of melanocytic tumors. Even expert pathologists find such cases hard.
  • Figure 5 case — a routine nevus, but the model was misled by color. Melanoma ROIs are generally dark, whereas nevus regions are typically light; this slide had dark areas outside the annotated ROI that pushed the model toward a melanoma prediction and an incorrect ROI.

This analysis transparently identifies a genuine limitation: the model can be confounded by staining and darkness cues, and by rare morphological variants — precisely the cases that are also difficult for human readers.

9. Discussion, related work, and limitations

9.1 Positioning against related work

  • CLAM [31] (clustering-constrained attention multiple-instance learning; Lu et al., Nat. Biomed. Eng. 2021) is the primary baseline. CLAM performs slide classification and ROI detection without pixel/patch labels, and works well on renal-cell and lung cancers — but its ROI detection is unsatisfactory on the melanocytic dataset, which motivates PCLA-3C.
  • CNN-based classifiers for breast/skin cancer [13–16, 37, 38] mostly target the classification task; PCLA-3C additionally does ROI detection and reports higher accuracy on melanocytic tumors.
  • Transformer methods for medical image enhancement — Feng et al.'s T2Net (joint MRI reconstruction/super-resolution) [34] and Wang et al.'s TED-Net (low-dose CT denoising) [35] — and transfer-learning skin-cancer classifiers (Khalid et al. [36]) are cited as complementary directions.
  • Weakly-supervised segmentation — Lerousseau et al.'s WMIL [39] generates patch pseudo-labels from slide labels during training and is highlighted as a promising avenue for future WSI study.

9.2 Strengths (in detail)

  1. Detection without complete labels — the central novelty. PCLA-3C learns ROI localization from partial annotations plus a manually curated "Other" background class, rather than from exhaustive pixel/patch masks (which do not exist for this cohort). This directly addresses the defining obstacle in melanocytic pathology: exhaustive annotation is infeasible, and the ground truth is itself uncertain given the documented 25–26% inter-pathologist discordance [40].
  2. One model, two tasks. Slide classification (majority vote over patch predictions) and ROI localization (ranking patches by the winning-class score) both emerge from the same patch classifier — no separate detector or segmentation head is trained, which keeps the method simple and internally consistent.
  3. Large margin over the state-of-the-art baseline. On the fixed 26-WSI test set PCLA-3C improves slide accuracy from 0.692 → 0.923 and IoU from 0.112 → 0.382 (a more than threefold gain in ROI overlap) relative to CLAM.
  4. Data efficiency and robustness. In the subsampling study (Table 3), PCLA-3C trained on only 20% of the data already surpasses CLAM's best IoU (obtained at 80%), and its accuracy/IoU improve monotonically with more data, with comparatively tight 95% confidence intervals.
  5. Clinically aligned evaluation. By treating nevi as the positive class, the reported 100% specificity means no malignant case in the test set was labeled benign — the error direction that matters most clinically.
  6. Interpretability for pathologists. The three visualization modes (overlay, boundary, heatmap) provide spatially grounded, human-readable explanations of where the model sees tumor, supporting review and trust rather than a black-box slide label.
  7. Accessibility. A VGG16 patch classifier runs on a single RTX 3090 and is compatible with open tooling such as FastPathology — a low barrier to reproduction and deployment.

9.3 Weaknesses (in detail)

  1. Not autonomous. Slide-classification accuracy is 92.3%, so the model cannot replace a pathologist; two true nevi were classified as melanoma. The authors explicitly frame it as decision support.
  2. Residual color/stain confounding. Even after color normalization, dark tissue outside the annotated ROI can bias the model toward "melanoma" (Figure 5), because melanoma regions are typically darker than nevi — the model may key on staining darkness rather than morphology.
  3. Small, single-institution cohort. The study uses 160 WSIs from one institution (UNC), with only 26 test slides and no external, multi-scanner validation in the published work; generalization to other labs, scanners, and populations is unproven.
  4. Annotation and curation dependence. The gains rely on the partial annotations, and the "Other"-class curation is manual and not fully specified, which limits exact reproducibility; performance degrades as training data shrinks.
  5. Dated backbone with no domain pretraining. VGG16 (2014) predates modern vision Transformers and pathology foundation models; it is initialized from ImageNet rather than self-supervised pathology pretraining, bounding its representational power (see §9.5).
  6. No slide-level context. Patches are classified independently; the method models no inter-patch or global-slide context (unlike attention-MIL or dedicated slide encoders), which can fragment the predicted ROI.
  7. Evaluation coupling and implementation subtleties. IoU is computed at the ground-truth annotated ratio $\beta$ (which sizes the prediction), and ranking operates on raw logits with a per-class-threshold-plus-argmax gate; these are defensible but should be kept in mind when comparing to other protocols.

9.4 Conclusions and future work

The authors conclude that the framework produces an accurate and robust approach to detect skin tumors and predict tumor type, which they suggest could reduce clinician workload and improve diagnostic efficiency. The authors anticipate that the approach may generalize to other cancers (not just skin) and to vision-based treatment-outcome prediction. Future work stated in the paper: extract richer information from high-quality WSIs and incorporate extra modalities such as gene expression and clinical data to further improve detection and prediction.

9.5 Beyond the paper: post-2024 medical-imaging models and how they could extend PCLA-3C

The most impactful development since this paper is the arrival of pathology foundation models and promptable segmentation models. The VGG16 patch classifier is the natural component to modernize; the models below (all 2024) are the leading candidates, with their architectures summarized.

  • UNI (Chen et al., Nature Medicine 2024) — a general-purpose tile encoder for pathology. Architecture: a Vision Transformer, ViT-Large/16 (~307 M parameters) pretrained with the DINOv2 self-supervised objective on Mass-100K (~100 million H&E tiles from ~100,000 diagnostic WSIs across 20 tissue types). It emits a 1024-dimensional embedding per 256×256 tile. How it would extend PCLA-3C: replace the VGG16 backbone with a frozen or lightly fine-tuned UNI encoder and train only a small 3-class head — typically far higher accuracy and data efficiency than an ImageNet-pretrained CNN.
  • Virchow / Virchow2 (Paige, 2024) — a large pathology tile encoder. Architecture: a ViT-Huge/14 (~632 M parameters) trained with DINOv2 on ~1.5 million WSIs (billions of tiles); Virchow2 adds pathology-specific augmentations and multi-magnification training. Tile embeddings concatenate the class token with pooled patch tokens. Extension: a stronger drop-in replacement for the patch encoder where compute permits.
  • Prov-GigaPath (Xu et al., Nature 2024) — a whole-slide foundation model that adds the slide-level context PCLA-3C lacks. Architecture: (i) a ViT tile encoder (DINOv2) over 256×256 tiles, followed by (ii) a slide encoder built on LongNet, a Transformer with dilated attention that scales to tens of thousands of tiles per gigapixel slide. Pretrained on Prov-Path (~1.3 billion tiles from ~171,000 WSIs). Extension: a LongNet-style aggregator over patch embeddings would let the model reason about global slide layout instead of voting over independent patches.
  • MedSAM (Ma et al., Nature Communications 2024) — the medical adaptation of the Segment Anything Model, directly relevant to the ROI-detection goal. Architecture: a ViT-Base image encoder producing a dense image embedding, a prompt encoder (here, bounding-box prompts), and a lightweight two-layer transformer mask decoder with transposed-convolution upsampling to a full-resolution mask; trained on ~1.5 million image–mask pairs spanning ~10 modalities. Extension: PCLA-3C's coarse patch-level ROI could seed box prompts for MedSAM to produce a pixel-precise ROI boundary, improving IoU beyond the patch grid.
  • CONCH (Lu et al., Nature Medicine 2024) — a vision–language pathology model. Architecture: a CoCa-style dual encoder — a ViT image encoder and a text encoder trained jointly with contrastive and image-captioning objectives on ~1.17 million pathology image–caption pairs. Extension: enables zero-shot / text-promptable classification ("melanoma vs. nevus") and report-style outputs, reducing dependence on curated patch labels.

Synthesis for future work. A modern successor to PCLA-3C would (a) swap VGG16 for a DINOv2-pretrained pathology tile encoder (UNI or Virchow2), (b) add a slide-level LongNet aggregator (Prov-GigaPath) for global context, and (c) refine the coarse patch ROI into a precise boundary with a promptable segmenter (MedSAM) — while retaining this paper's key insight that partial annotations plus an explicit background class are enough to supervise ROI detection.

10. Reproducibility and computational environment

Aspect Detail
Language Python
Image I/O OpenSlide (reading .svs WSIs); pyvips/libvips (writing pyramidal .tiff)
DL framework PyTorch ≥ 1.7.1, torchvision
Hardware UNC Longleaf Cluster (Linux, tested on Ubuntu 18.04); NVIDIA GeForce RTX 3090 on local workstations
CUDA Tested on CUDA 11.3
Complementary tool FastPathology [32] acknowledged as a user-friendly platform compatible alongside the Python framework
Data availability Available from the corresponding author on reasonable request; a robust methylation classifier reference dataset is cited [41]

Typical run order (from codes/, once the models/ and datasets/ helper packages are on the PYTHONPATH and a color-normalized dataset is available):

  1. python extract_patches_3class.py --data_dir ... --csv_path ... --xml_annotation_new ... --xml_annotation_other ... --feat_dir ...
  2. python method_pcla_3class.py --exp_name pcla_3class --data_folder <feat_dir> --batch_size 100 --n_epochs 20
  3. python score_pcla_3class.py --exp_name pcla_3class --model_load <ckpt> --csv_path ... --patch_path ... --results_dir ... --classification_save_dir ...
  4. python visual.py --exp_name pcla_3class --csv_path ... --wsi_dir ... --results_dir ... --xml_dir ... [--heatmap|--boundary] --percent 0.2
  5. python analysis.py --results_dir <results_dir>/pcla_3class --csv_dir <dir with melanomal.csv>

Or run everything via the orchestrator: python automated_pipeline.py --config config.yaml (use --validate-only first to check paths).

Metadata CSV contract: columns slide_id, data_split (train/val/test), and label_name (Melanoma/Nevi/Other). WSIs are .svs; per-slide patch bags are .h5 keyed by slide_id.

11. Reconciling the reported numbers

For a fully faithful reading, a few figures in the paper do not perfectly reconcile with one another. These are noted here for the careful reader; none change the paper's conclusions.

  • 160 vs. 165 WSIs. The paper consistently states 160 WSIs (86 melanoma + 74 nevi). The repository README's overview text mentions 165 — treat 160 as authoritative (it is used throughout the paper and its tables).
  • Training-set composition. The main text gives the training split as 71 melanoma + 63 nevus = 134, and the test split as 15 + 11 = 26 (these sum correctly and are self-consistent with the 86/74 totals). The Figure 2a panel labels the training set as "Melanoma 67, Nevus 57" (which sums to 124, not 134); the main-text numbers (71/63) are the consistent ones.
  • Confusion-matrix totals vs. 26-slide test set. Table 2's cells (20 / 0 / 2 / 9) sum to 31 and yield 93.5% accuracy, whereas the headline slide-classification accuracy is 92.3% (i.e., 24 of 26 correct, with exactly 2 nevi misclassified as melanoma per §3.3 of the paper). The 2 false-melanoma count is consistent across the confusion matrix and the misclassification discussion; the absolute cell counts and the 93.5% figure do not fully reconcile with a 26-slide test set. When in doubt, the 92.3% / 24-of-26 / 2-misclassified framing is the internally consistent one. (The companion reply_to_reviewer.md, Reviewer 2, presents a corrected confusion matrix that sums to 26 and matches 92.3%.)

12. Glossary and abbreviations

Term Meaning
H&E Hematoxylin and eosin — the standard histology stain
WSI Whole slide image — a gigapixel digitized microscope slide
ROI Region of interest — the diagnostically relevant tumor region
CNN Convolutional neural network
Patch A small fixed-size tile (256×256 px here) cut from a WSI
Nevus (pl. nevi) Benign melanocytic tumor (mole)
Melanoma Malignant melanocytic tumor (skin cancer)
PCLA-3C The proposed 3-class patch classifier
CLAM Clustering-constrained attention multiple-instance learning (baseline)
IoU Intersection over Union — overlap metric for detection
β (annotated ratio) Fraction of a slide's patches inside the annotation
OPTICS Ordering Points To Identify the Clustering Structure — density-based clustering
MIL / WMIL (Weakly-supervised) multiple-instance learning

13. Key references

Selected references cited in this document (numbering follows the paper):

  • [10] Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG). ICLR 2015.
  • [28] Ruifrok, A.C.; Johnston, D.A. Quantification of Histochemical Staining by Color Deconvolution. Anal. Quant. Cytol. Histol. 2001.
  • [29] Macenko, M. et al. A Method for Normalizing Histology Slides for Quantitative Analysis. IEEE ISBI 2009.
  • [30] Ankerst, M.; Breunig, M.M.; Kriegel, H.P.; Sander, J. OPTICS: Ordering Points to Identify the Clustering Structure. SIGMOD 1999.
  • [31] Lu, M.Y.; Williamson, D.F.; Chen, T.Y.; Chen, R.J.; Barbieri, M.; Mahmood, F. Data-efficient and weakly supervised computational pathology on whole-slide images (CLAM). Nat. Biomed. Eng. 2021.
  • [32] Pedersen, A. et al. FastPathology: An Open-Source Platform for Deep Learning-Based Research and Decision Support in Digital Pathology. IEEE Access 2021.
  • [39] Lerousseau, M. et al. Weakly Supervised Multiple Instance Learning Histopathological Tumor Segmentation (WMIL). MICCAI 2020.
  • [40] Hekler, A. et al. Pathologist-level classification of histopathological melanoma images with deep neural networks. Eur. J. Cancer 2019.

Full reference list (41 entries) is in the paper, pages 11–12.

Appendix A: Peer-review clarifications

A companion document, reply_to_reviewer.md, gives point-by-point responses to three Cancers reviewers. The substantive points relevant to this explainer are summarized below:

  • Reviewer 1 (ML methodology). Requested a full architecture specification (addressed in §5.3.1), clarification of the CLAM comparison protocol (§7.1), confirmation that ranking uses raw logits (§6.2), and documentation of the "Other"-class curation (§5.2).
  • Reviewer 2 (clinical). Raised single-institution / small-cohort external validity, the Table 2 confusion-matrix inconsistency (§11), color/stain confounding (§8), and the intended decision-support role.
  • Reviewer 3 (statistics). Asked for the confidence-interval resampling protocol (§7.3), correction of the Figure 2a training-split counts (§11), justification of the β-sized top-$n\beta$ IoU protocol (§5.6), and broader baselines (§9.1).