Clothes Segmentation - Technical Report

CyShield assessment CVCY002, Computer Vision Engineer

A deep-learning pipeline that takes a photo of a person and segments the clothes they are wearing, producing a mask, an overlay, and a transparent garment cut-out ready for a virtual fitting room. This document is both the required report and the setup/usage instructions - there is no separate README.

Note on numbers: the results table in Section 5 is from an actual training run (report/train_log_a100.txt, checkpoints/history.csv), not a placeholder. Everything in this report describes a real run or a final design decision.

---

Setup & usage

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Get ATR from Hugging Face - no manual download or unzipping:

pip install datasets
python scripts/prepare_hf.py --out data/atr                    # full ~17.7k images

Train, evaluate, predict:

python -m src.train --config configs/config_a100.yaml           # A100 preset, see below

python -m src.evaluate --checkpoint checkpoints/best_model.pth --out report/metrics.txt

python -m src.predict --checkpoint checkpoints/best_model.pth --input my_photo.jpg

Trained checkpoints aren't committed to git; the best_model.pth from the run described in Section 5 is published on Hugging Face: IsmaelElsharkawi/clothes-segmentation-task.

predict.py writes into outputs/: <name>_overlay.png (the photo with garment classes painted on top, for eyeballing), <name>_cutout.png (RGBA, only garment pixels opaque - the actual fitting-room asset).

Training configuration. config_a100.yaml trains at batch size 64 (with a sqrt-scaled learning rate) for 30 epochs on the full dataset, per the epoch-count finding in Section 5/Section 9. You can use --epochs N on the command line to extrapolate

Reproducibility. train.seed seeds Python/NumPy/PyTorch; the train/val split is written once by prepare_hf.py from a fixed --seed, so it's identical on every machine; the full config is saved inside every checkpoint, so evaluate.py/predict.py always rebuild the exact architecture that trained.

1. Dataset choice and reason

Chosen: ATR (human parsing), ~17,700 images with pixel-level labels for 18 body and clothing parts.

Candidates considered:

Dataset	Size	Why
ATR (human parsing)	~17.7k	Full-body shots of single people, dense pixel labels, garment classes already separated from skin/hair. Exactly the framing a fitting-room photo has.

Why collapse 18 labels into 4. ATR separates upper-clothes, skirt, pants, dress, belt, scarf, bag, shoes, and body parts. For "segment the clothes worn by a person" most of that is either not a garment or too rare to learn reliably. Collapsing to background / upper_body / lower_body / full_body:

• removes classes with only a few hundred examples (belt, scarf as a standalone), which would otherwise contribute noise and near-zero IoU;
• balances the label distribution, so the Dice term in the loss is well behaved;
• matches the downstream product need. A fitting room replaces an upper garment, a lower garment, or a dress, not "a belt".

Why four classes and not two. A plain garment-vs-background mask would be easier to learn and would score higher, but it cannot answer "which pixels are the top", so it could only ever swap an entire outfit at once - the three-way garment split is the minimum that supports replacing just one item. full_body has to be its own class rather than an upper_body + lower_body pair, because a dress covers the same region as a top and bottom together yet can only be swapped as a single piece. Nothing is given up by measuring the binary case anyway: Section 5 reports foreground IoU (all garment classes merged) alongside the per-class numbers.

The mapping is a single readable table (ATR_GROUPS in src/dataset.py) and can be changed without touching any other code.

Split. 90 % train / 10 % validation, shuffled with a fixed seed by scripts/prepare_hf.py, so the split is identical on every machine.

2. Preprocessing and augmentation

• Letterbox resize to 256×256. The long side is resized and the short side is zero-padded. Plain square resizing would stretch the person and distort garment proportions, which is precisely the signal we want the model to learn.
• Geometric augmentation (applied to image and mask): horizontal flip (p=0.5), shift ±5 %, scale ±15 %, rotation ±15°. This covers the framing and pose variation of real user photos.
• Photometric augmentation (image only): brightness/contrast ±20 %, hue and saturation jitter, occasional motion/Gaussian blur and sensor noise. Phone photos taken in a bedroom look nothing like the studio shots ATR is dominated by; this is the cheapest way to close part of that gap.
• ImageNet normalisation, because the encoder is ImageNet-pretrained.

Validation and inference use resize + normalise only, so evaluation is deterministic.

3. Model architecture

U-Net with an ImageNet-pretrained MobileNetV3-Large encoder, built with segmentation_models_pytorch (smp 0.5.0) in src/model.py. 6.69 M parameters total; maps a (B, 3, 256, 256) image batch to (B, 4, 256, 256) raw logits.

The network in full

Encoder - timm-mobilenetv3_large_100, depth 5. A 3x3 stride-2 stem convolution (3 → 16), then 15 bottleneck blocks (1 depthwise-separable + 14 inverted residual / MBConv), of which 8 carry squeeze-and-excitation channel attention, with hard-swish activations in the deeper stages; a final 1x1 convolution widens the last stage to 960 channels. Six feature maps are tapped, five of which feed the decoder:

Stage	Stride	Channels	Map @ 256 px	Role
0	1/1	3	256×256	input, unused
1	1/2	16	128×128	skip → decoder block 3
2	1/4	24	64×64	skip → decoder block 2
3	1/8	40	32×32	skip → decoder block 1
4	1/16	112	16×16	skip → decoder block 0
5	1/32	960	8×8	bottleneck

Decoder - 5 U-Net blocks, 256 → 128 → 64 → 32 → 16 channels. Each block does a nearest-neighbour 2× upsample, concatenates the matching encoder skip, then applies two 3x3 Conv → BatchNorm → ReLU layers. No attention gates (smp's attention slots are Identity in this configuration).

Block	Upsamples to	Input channels	Output
0	16×16	960 + 112 = 1072	256
1	32×32	256 + 40 = 296	128
2	64×64	128 + 24 = 152	64
3	128×128	64 + 16 = 80	32
4	256×256	32 (no skip)	16

Head. A single 3x3 convolution, 16 → 4 channels, with no activation: the network emits raw logits and softmax is applied inside the loss and at inference.

Part	Parameters
Encoder (MobileNetV3-Large)	2,971,952
Decoder (5 U-Net blocks)	3,713,728
Segmentation head	580
Total	6,686,260

Note that the decoder is the larger half of the model - the 1072-channel concatenation at block 0 dominates. With the encoder frozen for the first 3 epochs, 3,714,308 parameters are trainable.

Why this design

• Transfer learning is not optional at this data scale. ~17.7k images is small for dense prediction. A pretrained encoder already encodes edges, texture and shape, so training only has to learn "which pixels are garments".
• U-Net's skip connections recover boundary detail, which is the property that matters most for a cut-out: the asset is judged on how clean its edges are, and each decoder block gets a direct path back to a full-resolution feature map rather than having to reconstruct edges from a coarse bottleneck.
• A MobileNetV3-Large encoder keeps the model cheap. Depthwise-separable convolutions hold the whole encoder under 3 M parameters, which is what makes it trainable at this budget and fast enough to serve in real time. The encoder name is a config option (model.encoder), so scaling up needs no code change if more budget appears.
• Full-resolution output. The fifth decoder block upsamples back to the input resolution with no skip (stage 0 is the raw image), so the head predicts at 256×256 directly rather than at a coarser stride that would need upsampling afterwards.
• Encoder frozen for the first 3 epochs. The pretrained features are already good; skipping their gradients while the randomly-initialised decoder warms up saves memory and time at no measurable cost.

Training setup. AdamW (lr 0.0028, sqrt-scaled for batch size, weight decay 1e-4), cosine LR decay over 30 epochs, batch size 64, 256 px, fp32, on the full ~17.7k-image dataset. The checkpoint with the best validation mIoU is kept.

What this costs

This architecture is an explicit accuracy-for-cost trade. Dataset size and epoch count are no longer part of it - both are already at their ceiling (full dataset, 30 epochs; see Section 9) - so what remains is resolution and model capacity. Against a heavier setup (384 px, a larger encoder and decoder) the expected loss is several points of mIoU, concentrated in the places this model is already weakest:

• Thin boundaries (sleeve edges, collars, straps) from the 256 px resolution. This is the most visible failure in a cut-out.
• The rare full_body class, which has the fewest training examples of the four and so remains the weakest regardless of resolution or decoder.
• Global garment-extent decisions (dress vs. shirt+skirt), which need context wider than this decoder's effective receptive field.

If the budget later allows, the highest-value knobs to turn back up are, in order: resolution (256 → 384), then encoder and decoder capacity. Resolution is a config option; more capacity means swapping segmentation_models_pytorch arguments.

4. Loss function selection and reason

total = 1.0 · CrossEntropy + 1.0 · Dice

Clothes segmentation is class-imbalanced: background is typically 60-80 % of the pixels and full_body is comparatively rare.

• CrossEntropy alone optimises per-pixel accuracy. A model can score well by leaning towards background, and small garment regions barely affect the gradient.
• Dice alone optimises region overlap directly (the thing IoU measures) and is insensitive to class size, but its gradients are noisy early in training when predictions are near-random, and it can stall.
• The sum gives CrossEntropy's stable, well-conditioned gradients everywhere plus Dice's pressure to actually cover each garment region. This pairing is the standard choice for imbalanced medical and human-parsing segmentation for exactly this reason.

Both weights are exposed in configs/config_a100.yaml (ce_weight, dice_weight). The Dice implementation uses a smoothing constant of 1.0 so that a class absent from a batch contributes 0 loss rather than NaN.

5. Performance analysis and evaluation metrics

Metrics used and why

Metric	Why it is reported
Per-class IoU	The standard segmentation metric. Shown per class because the aggregate hides which garment type is failing.
mIoU	Mean over classes; used for model selection (best checkpoint).
Per-class Dice (F1)	Less harsh than IoU on small regions; useful for spotting a class that is found but poorly delineated.
Pixel accuracy	Easy to read, but reported with a caveat: background dominance makes it optimistic. It is never used for model selection.
Foreground IoU	All garment classes merged into "garment vs. not garment". This is the number a virtual fitting room actually cares about, since the cut-out quality depends on the union, not the class split.

All metrics come from a confusion matrix accumulated over the entire split, not averaged per batch, so an odd-sized last batch cannot skew the result. Classes absent from the data become NaN and are excluded from the means.

Results

From an actual run: configs/config_a100.yaml, full ATR dataset (17,706 images), 30 epochs, on a Colab A100. Full per-epoch log in report/train_log_a100.txt, full per-epoch metrics in checkpoints/history.csv. The numbers below are that run's validation split at its best checkpoint (epoch 30) - the same confusion-matrix computation evaluate.py would print, since train.py's per-epoch validation pass uses the identical SegmentationMetrics class.

class                  IoU      Dice
----------------------------------
background          98.5%     99.2%
upper_body           75.3%     85.9%
lower_body           75.3%     85.9%
full_body            56.7%     72.4%
----------------------------------
mean                 76.5%     85.9%

pixel accuracy : 97.8%
foreground IoU : 86.0%

This matches the expected pattern: background, upper_body and lower_body are the strongest classes, full_body is clearly the weakest - both because dresses are the least frequent label and because they are the class most easily confused with an upper_body + lower_body pair (see Section 7). The confusion matrix in src/metrics.py makes that specific failure directly visible: check the full_body row against the upper_body/lower_body columns.

checkpoints/history.csv records per-epoch train loss, validation loss and every validation metric, so overfitting is easy to spot: it begins where validation loss turns up while training loss keeps falling. In this run that gap starts opening (train loss 0.190 vs. val loss 0.240 by epoch 30) without validation loss itself turning upward yet - see Section 9 for what that implies about further training.

6. System capabilities (task 3a)

What the system does well:

• Separates garments from skin and hair, not just "person from background". Body parts are explicitly mapped to background during training, so bare arms and legs are correctly excluded from the cut-out.
• Distinguishes upper from lower garments, which is what makes selective try-on ("replace only the top") possible.
• Handles a single, roughly centred, mostly unoccluded person in a wide range of poses, garment colours and patterns. This is the bulk of ATR and the model is strongest here.
• Preserves input resolution. Inference letterboxes to 256 px, then interpolates the logits (not the argmax mask) back to the original size, which gives visibly smoother garment boundaries than upsampling a hard mask.
• Produces a directly usable asset. The RGBA cut-out needs no further post-processing before compositing.
• Cheap enough to serve behind a simple API. At 6.69M parameters and a 256px input, inference cost is small relative to typical segmentation models, though actual GPU/CPU throughput has not been benchmarked and should be measured before making a latency claim.

7. Drawbacks and known failure modes (task 3b)

• Multiple people. The model is semantic, not instance-aware. Two people in frame produce one merged garment mask with no way to tell whose is whose - and with enough people it can fail much harder than a merged mask. examples/ayza-atgawez_overlay.png (5 people, varied pose/scale/occlusion) produces a mask on essentially one, mostly-occluded, centrally-placed person, and misses a large, unoccluded, brightly-coloured blazer on another person entirely. Contrast examples/odyssey_overlay.png, where 3 people - similarly scaled, similarly posed, all unoccluded and facing camera - are all segmented reasonably well. The failure tracks how much the frame departs from the single-person training distribution, not headcount alone.
  
• Dress vs. two-piece confusion. The most common class error. A long shirt over a skirt, or a co-ord set in a single colour, is frequently labelled full_body (or the reverse). examples/donald-trump-xi-jinping.png shows the reverse case concretely: a single-colour two-piece suit (jacket + trousers, same fabric) gets painted almost entirely upper_body, with only ragged fragments of lower_body breaking through near the ankles - the model can't find the jacket/trouser boundary when colour gives it no cue.
  
• Thin and low-contrast boundaries. Straps, thin belts, sleeve/skin edges and garments whose colour matches the background lose detail at 256 px. Output stride limits how fine a boundary the decoder can recover, and the 256 px working resolution is the single biggest contributor to this failure (see Section 3).
• Occlusion and cropping. Bags, crossed arms, held objects, and photos cropped above the knee degrade the lower-body prediction in particular. examples/the-office_overlay.png shows this within a single successful photo: one subject's shirt and trousers are segmented cleanly, while the other's skirt - similar-toned to her skin, and partly occluded by her clasped hands - gets a fragmented, misplaced lower_body patch instead of full coverage.
  
• Non-garment objects mistaken for clothing. examples/ismael-yaseen_overlay.png shows a rifle held between two people with part of its barrel painted lower_body. Texture/shape cues the model associates with garments (a long, roughly parallel-edged region between two people) aren't exclusive to clothing, and nothing in the pipeline constrains predictions to plausible garment shapes.
  
• Non-photographic input. ATR is exclusively real photographs; the model has never seen flat colour, hard outlines, or illustrated shading. examples/stewie-griffin.png (an animated character) gets a visible upper_body patch painted directly on a cheek - skin, not fabric. This is a distinct failure axis from the pose/lighting biases below: it's the image's medium, not its content, that's out of distribution.
  
• Tight headshots / portrait crops. Observed directly on a close face-and- shoulders photo where only a thin strip of the actual garment was visible: the model didn't just lose boundary precision, it missed the real garment almost entirely (predicted background) and instead hallucinated garment classes on non-garment regions - teeth/mouth painted full_body, collar creases painted lower_body. ATR has no training examples cropped this tightly, so the model has no learned prior for where in frame a garment should even be, and falls back to matching local texture instead. This is the concrete failure behind the framing requirement in Section 8.
• Layering is not modelled. An open jacket over a t-shirt is one flat upper_body region; the model cannot say which garment is on top.
• Dataset bias. ATR is dominated by full-body, front-facing, well-lit shots of standing people, skewed towards fashion photography. Expect degradation on seated poses, back views, close crops, and on body types, skin tones and garment styles under-represented in the source data. This bias should be measured on a held-out set representative of the real user population before any deployment.
• Accessories are background by design. Bags, hats, shoes and belts are deliberately not segmented. That's a product decision, not a bug, but it's still a limitation if the fitting room later wants them.
• No confidence output. The pipeline returns a hard argmax. It cannot currently say "I am unsure about this region", which a production system should do before showing a bad cut-out to a user.

8. Operating limitations: required capture conditions (task 3c)

For reliable output, the input photo should satisfy:

Condition	Requirement
People in frame	Exactly one person
Framing	Full body or at least head-to-knee; person occupies ≳ 40 % of the frame height
Pose	Standing, facing the camera, arms not fully crossed over the torso
Occlusion	Garments not substantially hidden by bags, furniture or other people
Lighting	Even, indoor or daylight; no strong backlight or hard shadows across the body
Background	Any, but contrasting with the garment colour; avoid a wall the same shade as the outfit
Resolution	≥ 512 px on the long side (below this, boundary quality drops sharply)
Blur	Sharp; heavy motion blur is not recovered
Image type	A real photo of a worn garment, not a flat-lay, product shot or mannequin

Outside these conditions the model still returns a mask, but neither the class assignment nor the boundary quality should be trusted.

9. Possible improvements

Ordered by expected return on effort:

1. Test-time augmentation (horizontal flip + multi-scale), a reliable 1-2 point mIoU gain for a few lines of code.
2. Tune regularization, not epoch count, from here. The 30-epoch A100 run (Section 5) shows why: train/val loss gap opened to 0.190 vs. 0.240 by the last epoch while val loss itself had nearly flattened - a sign this data volume is close to its ceiling for this model size at this epoch count. More epochs alone are unlikely to help much further; stronger augmentation or higher weight decay, tuned against the val-loss knee, are more likely to.
3. Higher inference resolution (512 px) to recover thin boundaries, at a linear cost in latency.
4. A stronger encoder (EfficientNet-B3, ConvNeXt-Tiny) once the data pipeline is the bottleneck rather than the model.
5. Boundary-aware loss (add a boundary/Lovász term) to sharpen garment edges, which matters more perceptually than mIoU suggests.
6. A person detector as a pre-step. Crop to one person, which both fixes the multi-person failure and increases effective resolution on the subject.
7. Fine-tune on in-domain photos. A few thousand images from the real target distribution would beat any of the above.
8. Expose per-pixel confidence (max softmax probability) so the application can reject low-quality results instead of showing them.