Take a compact NLP intent classifier from an fp32 baseline to a set of mobile-ready inference artifacts, and make the deployment choice with a reproducible
quality vs latency vs size vs memory vs energybenchmark — not a guess.
A support / assistant / productivity mobile app needs to classify short user
phrases (e.g. "I am still waiting on my card?" → card_arrival) on device,
without a mandatory cloud round-trip. Even an accurate model is often unshippable
on mobile because it is too large, too slow, memory-hungry, or battery-hungry, and
forcing cloud inference hurts latency, cost and privacy.
This project shows the engineering path from "a model that works" to "a model you can actually ship on device", and — crucially — a measured recommendation for which optimized variant to deploy.
- Use case: on-device intent classification for short user messages.
- Dataset: Banking77 — 77 fine-grained banking-support intents, ~13k short utterances. Single-domain and clean, so the optimization story is the star, not a fight for SOTA accuracy.
- Users: ML / mobile-ML / edge-AI engineers, applied scientists, and platform teams responsible for inference efficiency and on-device UX.
┌──────────────┐
Banking77 ──▶ train │ baseline fp32│ BERT-tiny (2L, H128, 4.4M params)
└──────┬───────┘
│
┌───────────┬───────┬───────┬───────────────┬──────────────┬──────────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼
ONNX export ONNX int8 torch unstructured distillation structured structured
(fp32 graph) (dynamic) int8 prune (50%) (mini→tiny) FFN prune + ONNX int8
│ │ │ + recover 512→256 (mobile pick)
│ │ │ + recover
└───────────┴───────┴───────────────┴────────────────┴──────────────┴──────────────┘
│
▼
benchmark suite → quality / latency / size / memory / energy
→ deployment recommendation
Every variant is measured on the same axes, single-threaded on CPU to emulate on-device single-core inference.
| Axis | Metric |
|---|---|
| Quality | test accuracy, macro-F1 (77 classes) |
| Size | deployable artifact size (MB) |
| Load | cold model/session load time (s) |
| Latency | single-sample p50 / p95 / mean (ms, 1 thread) |
| Memory | steady-state process RSS (MB) |
| Energy | CPU-seconds per inference — compute proxy, honestly labeled |
Energy is a proxy. Without on-device power measurement, CPU-time per inference is used as a transparent compute/energy proxy. It is labeled as such everywhere and not presented as a true battery measurement.
Single-thread CPU, Banking77 test set (3076 samples, 77 classes). Latency is single-sample (batch=1).
| variant | accuracy | macro-F1 | size (MB) | load (s) | p50 (ms) | p95 (ms) | RSS (MB) | energy-proxy (cpu-ms/infer) |
|---|---|---|---|---|---|---|---|---|
| baseline_fp32_torch | 0.8644 | 0.8610 | 16.77 | 6.08 | 1.501 | 1.963 | 426 | 1.641 |
| onnx_fp32 | 0.8644 | 0.8610 | 16.82 | 5.20 | 0.541 | 0.609 | 481 | 0.547 |
| onnx_int8 | 0.8638 | 0.8601 | 4.29 | 4.76 | 0.396 | 0.467 | 472 | 0.391 |
| torch_int8_dynamic | 0.8628 | 0.8592 | 15.60 | 4.99 | 2.048 | 2.372 | 439 | 2.109 |
| pruned_fp32_torch (unstructured 50%) | 0.8807 | 0.8801 | 16.77 | 5.48 | 1.536 | 1.912 | 428 | 1.641 |
| distilled_fp32_torch | 0.8748 | 0.8744 | 16.77 | 5.81 | 1.593 | 2.141 | 427 | 1.719 |
| structured_pruned_torch (FFN 512→256) | 0.8872 | 0.8872 | 16.27 | 5.72 | 1.489 | 2.119 | 425 | 1.562 |
| structured_onnx_fp32 | 0.8872 | 0.8872 | 16.32 | 5.32 | 0.575 | 0.660 | 468 | 0.625 |
| structured_onnx_int8 | 0.8885 | 0.8885 | 4.17 | 4.82 | 0.367 | 0.427 | 455 | 0.312 |
structured_onnx_int8 sits top-left — highest quality at the lowest latency.
Deployment pick (mobile): structured_onnx_int8 — macro-F1 0.8885 (the best
of all variants), 4.0× smaller (4.17 MB) and 4.1× faster (p50 0.367 ms) than
the fp32 baseline. It Pareto-dominates every other variant: nothing is smaller, faster,
and more accurate. Structured FFN pruning (with a recovery fine-tune) + dynamic int8.
- ONNX fp32 reproduces the torch baseline's F1 exactly (0.8610) → the export is correct; it is also ~3× faster than eager PyTorch purely from graph optimization.
- Dynamic int8 is the size lever — 4× smaller at ≤0.2 F1-point cost.
- torch dynamic int8 is slower than fp32 here — at hidden size 128, int8 dynamic-quant overhead on CPU outweighs the compute saving. Honest counter-example: quantization is not automatically faster; the ONNX int8 path is the one that pays off.
- Unstructured pruning ≠ latency. 50% of Linear weights zeroed, quality kept — but dense CPU/ONNX kernels don't exploit unstructured sparsity, so size and latency are unchanged. Its payoff is only size-if-stored-sparse.
- Structured pruning is the right latency technique — it physically removes FFN
neurons (512→256), so the matrices really shrink. But at this tiny scale single-sample
latency is overhead-bound, not compute-bound: the fp32 p50 (0.575 ms) is within
measurement noise of the unpruned ONNX (0.541 ms). The FFN FLOP saving only clearly
surfaces in the int8 path, where
structured_onnx_int8(0.367 ms) edges plainonnx_int8(0.396 ms). The bigger, reliable payoff of structured pruning here was a quality gain from the recovery fine-tune — which is why it also tops the F1 column. - Size stays embedding-bound. BERT-tiny is 3.9M of 4.4M params in the word-embedding table, so no amount of FFN/attention pruning moves fp32 size much (16.8 → 16.3 MB); quantization is what collapses size (→4.2 MB). The two levers are complementary.
- RSS is process-level (framework + libs dominate the ~400–480 MB, not the 4–17 MB model); onnxruntime rows sit higher from the ORT library footprint, so RSS is a weak cross-framework signal — size and latency are the real mobile axes.
- ONNX dynamic int8 is the size lever: ~4× smaller artifact at a small, measured quality cost.
- Unstructured magnitude pruning zeroes 50% of Linear weights and keeps quality after a short recovery fine-tune — but on dense CPU/ONNX kernels it does not reduce latency or size; its payoff is only size-if-stored-sparse. Reported as such.
- Structured FFN pruning physically removes intermediate neurons (512→256), so it is the correct latency technique — but at hidden=128 single-sample inference is overhead-bound, so the FFN FLOP saving is within noise on fp32 and only edges ahead in the int8 path. Its clearest win here was a quality gain from the recovery fine-tune.
- Distillation keeps the compact student architecture but transfers signal from a larger teacher — any quality gain at equal size is a clean win.
- Size vs latency are separate levers: structured pruning targets compute/latency; quantization targets size (embedding-bound). Combining them gives the Pareto-best artifact.
- The pruned / structured / distilled variants receive extra training epochs (recovery / distill), so their quality is not strictly epoch-matched to the baseline; the point is the deployment trade-off, not a controlled training-budget ablation.
py -3.11 -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt
.venv/Scripts/python training/prepare_data.py # splits
.venv/Scripts/python training/train_baseline.py # baseline fp32
.venv/Scripts/python optimization/export_onnx.py # ONNX fp32 + int8
.venv/Scripts/python optimization/prune.py # unstructured pruned variant
.venv/Scripts/python optimization/distill.py # distilled variant (optional, slow)
.venv/Scripts/python optimization/structured_prune.py # structured FFN prune + ONNX export
.venv/Scripts/python benchmarks/run_all.py # benchmark + report
.venv/Scripts/python benchmarks/report.py # re-render report from cached CSVAfter the repro steps above, classify any phrase with the champion model:
.venv/Scripts/python predict.py "I am still waiting on my card?"query: "I am still waiting on my card?" (model: structured_pruned)
card_arrival 0.940
card_delivery_estimate 0.025
lost_or_stolen_card 0.008
- Lower serving cost & cloud dependency — inference moves on-device; the backend stops paying per request for a task that fits in ~4 MB.
- Better UX — lower, more predictable latency (p50/p95 reported) means the assistant feels instant even offline.
- Stronger privacy — user phrases are classified locally, never leaving the device.
- I optimize models against real deployment constraints, not just accuracy.
- I understand quality / latency / size / memory trade-offs and can measure them.
- I take an ML artifact all the way to a production-friendly, exportable form.
- I can defend an optimization decision in the language of product and business.
- Built a reproducible model-optimization lab taking a compact BERT intent classifier from an fp32 baseline through ONNX export, dynamic int8, unstructured + structured pruning and distillation, benchmarked on a single-thread CPU across quality/latency/size/memory/energy-proxy (9 variants).
- Produced a Pareto-dominant on-device artifact (structured FFN pruning + ONNX int8): 4.0× smaller, 4.1× faster, and higher macro-F1 than the fp32 baseline.
- Diagnosed why techniques did or didn't pay off — unstructured pruning can't speed up dense kernels, sub-ms inference at hidden=128 is overhead- not compute-bound, and model size is embedding-bound so quantization (not pruning) is the size lever.
- Implemented an honest energy/compute proxy and bounded quality regression, avoiding the common "optimize without benchmarking" trap; packaged it tying efficiency metrics to serving cost, UX latency and on-device privacy.
- Why on-device intent classification is a size/latency problem before it is an accuracy problem.
- Unstructured vs structured pruning: why the first can't speed up dense kernels and the second can — and why even structured pruning was latency-neutral here because sub-millisecond inference at hidden=128 is overhead-bound, not compute-bound.
- Why size stayed embedding-bound (3.9M/4.4M params) so quantization — not pruning — is the size lever, making the two techniques complementary rather than competing.
- How dynamic int8 quantization trades a controlled quality delta for 4× size, and why torch dynamic-quant was slower while ONNX int8 was faster.
- How to build an energy proxy you can defend instead of overclaiming battery impact.