On-device keyword spotting on a microcontroller, from training the network to measuring the current it draws. INT8 DS-CNN on TFLite Micro, MFCC written in C against the ESP-DSP FFT, a cascaded power state machine, and a sweep of 134 architectures with energy measured on hardware for every point on the Pareto front.
96.12 % accuracy, 101 KB model, 460 ms inference, 530x lower average power than continuous monitoring on the same board.
This is the firmware and training half of my bachelor's thesis. The research write-up and the measurement methodology live in itmo-bachelor-thesis; the current logger firmware lives in precise-power-logger.
vad_cascade_showcase.mp4
- The cascade
- Hardware
- Firmware architecture
- Feature extraction
- The network
- Inference on device
- Training, quantization and the sweep
- Measurements
- Build and flash
- Repository layout
The device spends almost all of its life in deep sleep, drawing microamps. An analog sound detector pulls a pin low when the acoustic level crosses a threshold, the chip wakes on EXT1, and only then does anything expensive happen. Each stage is more capable and more expensive than the one before it, and each is started by an event from the previous one.
 |
|---|
Three stages: an always-on analog detector on the raw waveform, then MFCC feature extraction, then DS-CNN classification. Cost rises left to right; duty cycle falls.
DEEP_SLEEP ──sound──> HW_BOOT ──> RECORD ──> MFCC ──> INFERENCE ──> RESULT ──> SHUTDOWN
6.2 mA 66.6 mA 56.2 mA 81.8 mA 83.1 mA 49.2 mA
1982 ms 2519 ms 410 ms 460 ms 597 ms
 |
|---|
Currents and durations are measured, not datasheet values, for the shipped f104_b5_ptq
model. The whole cycle costs 1525.4 mJ, and the interesting part is that computation is
only 19.2 % of it; see the thesis repo
for what that implies.
The state machine is deliberately linear and single-shot: wake, record a window, classify it once, show the result, sleep. There is no always-on inference loop, because an always-on inference loop is what the cascade exists to avoid.
| Part | Role | Why this one |
|---|---|---|
| ESP32-S3 | application processor | Xtensa LX7 with vector instructions for INT8, deep sleep with EXT1 wake, 512 KB SRAM, open SDK |
| INMP441 | microphone | digital MEMS, I2S output, 24-bit samples, no analog front end or codec needed |
| LM393 + mic | stage-1 sound detector | analog comparator, always powered, hundreds of microamps instead of tens of milliamps |
| INA228 | current measurement | 20-bit, 15 mOhm shunt, ~2.6 uA per LSB, hardware averaging down to roughly 1 uA |
| ST7789 | status display | development aid only, powered down during measurement runs |
On the microphone. The INMP441 is a digital MEMS part, not an electret capsule: the transducer, preamp, sigma-delta ADC and I2S serializer all sit inside the package. Three consequences matter here. There is no analog signal path to keep clean, so no codec, no bias network and no board-level noise budget to defend. It outputs 24-bit samples over I2S, which the ESP32-S3 receives in 32-bit slots and the firmware shifts down to int16. And it draws about 1.4 mA continuously, far too much to leave running through deep sleep, which is exactly why stage 1 is an analog comparator rather than the microphone itself.
On the comparator. Stage 1 has one job: decide whether anything is happening at all,
for as little energy as possible. An LM393 comparing a microphone envelope against a
threshold does that in the analog domain with no clock, no CPU and no sampling. A digital
VAD would need the microphone and the processor awake, which spends most of what cascading
was supposed to save. Its output goes to an RTC-capable GPIO held with a pull-up, and the
chip is armed with esp_sleep_enable_ext1_wakeup(..., ESP_EXT1_WAKEUP_ANY_LOW) before
sleeping.
Two power domains. Measuring a system this quiet means the instrument must not contaminate the measurement, so the device under test and the logger run from separate batteries with no shared supply.
 |
|---|
 |
|---|
 |
|---|
ESP-IDF, built through PlatformIO. Roughly 2000 lines of application C and C++, plus a generated mel filterbank table.
 |
|---|
C4 component view: main drives the state machine, voice_engine_kws coordinates the
recognition pipeline, kws runs the network on TFLM with ESP-NN SIMD kernels, mfcc sits
on ESP-DSP for the FFT, vad_sleep owns deep sleep and EXT1 wake, sync_gpio marks state
transitions for the logger.
| File | What it does |
|---|---|
main.c |
the cascade state machine: wake check, boot, run, sleep |
vad_sleep.c |
deep sleep entry, EXT1 wake arming, RTC GPIO hold |
i2s_mic.c |
I2S master, 16 kHz mono, 32-bit slots from the INMP441, downshift to int16 |
mfcc.c / mel_matrix.c |
STFT, mel filterbank and DCT-II in C; the 513x40 mel matrix is precomputed |
kws.cpp |
TFLite Micro interpreter, INT8 quantization of the input, argmax over 12 classes |
kws_profiler.h |
per-operator timing hooks used to produce the layer profiles |
model_loader.c |
maps the model flatbuffer from its own flash partition instead of linking it in |
voice_engine_kws.c |
record, pick the highest-energy 1 s window, extract features, classify |
voice_engine_wakenet.c |
the same interface backed by Espressif WakeNet9, for comparison |
sync_gpio.c |
1 ms pulse on a dedicated pin marking every state transition for the logger |
profile_storage.c |
persists profiling runs to SPIFFS |
display.c / st7789.c |
status display |
Three decisions in here are worth calling out.
The engine is an interface, not an if-statement. voice_engine.h declares three
functions, and either the custom DS-CNN backend or Espressif's proprietary WakeNet9 is
compiled behind them, selected by USE_CUSTOM_KWS. That is what made an honest comparison
against a closed-source baseline possible on identical hardware, in the same cascade, with
the same measurement harness.
The model is not linked into the firmware. It lives in its own 1 MB flash partition
(partitions.csv, custom subtype 0x40) and is memory-mapped at runtime through
esp_partition_mmap. Swapping the network means reflashing one partition rather than
rebuilding the application, which is what makes a 134-model sweep practical to measure.
Phase boundaries are marked in hardware. Because the state machine is strictly sequential, every transition emits a 1 ms pulse on GPIO1 instead of relying on timestamps. The logger sees the pulse and knows which phase the current trace belongs to, so phase energies do not depend on clock alignment between two boards.
Audio becomes a 49x10 MFCC matrix, on the device, in C:
 |
|---|
The word "yes" through the pipeline: (a) raw waveform, (b) magnitude spectrogram, (c) log-mel spectrogram, (d) the first nine MFCC coefficients that reach the network.
16 kHz PCM
-> frame: 40 ms window, 20 ms stride, Hann, zero-padded to 1024
-> FFT (esp-dsp radix-2, float32) -> magnitude spectrum, 513 bins
-> mel filterbank, 40 bins, 20 to 4000 Hz
-> log
-> DCT-II, orthonormal, keep the first 10 coefficients
-> 49 frames x 10 coefficients
Different words produce visibly different MFCC signatures, which is the entire premise of classifying them with a small convolutional network:
 |
|---|
Every constant here is duplicated in exactly two places, src/mfcc.h and nn/config.py,
and they have to agree. If the on-device pipeline and the training pipeline disagree by so
much as a window length, the model sees a feature distribution it was never trained on and
accuracy collapses in a way that looks like a quantization bug. The DCT is built to match
tf.signal.mfccs_from_log_mel_spectrograms term for term, for the same reason.
The mel filterbank is precomputed as a 513x40 float matrix in mel_matrix.c rather than
derived on the device, trading 80 KB of flash for not recomputing transcendental functions
on every boot.
DS-CNN from Hello Edge (Zhang et al., 2017), the MLPerf Tiny keyword spotting baseline, parameterized here by filter count and block count so the whole family can be swept.
 |
|---|
A stem convolution over the MFCC matrix, then N depthwise separable blocks, then global
average pooling and a fully connected layer over 12 classes: ten commands plus _silence_
and _unknown_.
The depthwise separable block is the whole reason this fits on a microcontroller. A regular 3x3 convolution over C channels costs 9·C·C multiply-accumulates per position. Splitting it into a depthwise 3x3 (one filter per channel, 9·C) followed by a pointwise 1x1 (C·C) drops the cost by roughly a factor of 9 while keeping the receptive field. The measured profile below shows exactly what that trade does to the runtime: the 3x3 depthwise layers become almost free, and essentially all of the time moves into the 1x1 projections.
Shipped configuration: 104 filters, 5 blocks, 101.2 KB as INT8, 96.12 % accuracy.
 |
|---|
Confusion matrix of the shipped INT8 model on the Google Speech Commands v2 test split.
The only class it struggles with is _silence_, which is also the smallest.
 |
|---|
TFLite Micro with a MicroMutableOpResolver<8> carrying only the operators DS-CNN
actually uses: Conv2D, DepthwiseConv2D, FullyConnected, Mean, Reshape, Quantize,
Dequantize, Softmax. Registering the full op set would cost tens of kilobytes of flash for
kernels that never run.
The tensor arena is 140 KB and is allocated with MALLOC_CAP_INTERNAL on purpose: the
ESP-NN SIMD kernels for Xtensa do not operate on PSRAM, so putting the arena in external
RAM silently drops the vectorized path and multiplies inference time.
Input floats are quantized to int8 using the model's own scale and zero point; the output is dequantized and argmaxed over the 12 classes.
kws_benchmark() re-runs inference N times with a per-operator profiler attached and
streams a CSV over UART. That is where every profile in measurements/*/profile/ comes
from, and it answers the question that matters for optimizing an inference path: where does
the time actually go.
 |
|---|
Shipped model, 100 runs, error bars are the standard deviation. The five pointwise 1x1 convolutions take 15.5 % of the time each and the stem takes 16.7 %, while the five depthwise 3x3 layers between them take 0.6 % each. Global average pooling is 2.4 %; the classifier is not measurable.
The shape holds across the whole family, not just this model:
 |
|---|
 |
|---|
Left to right: models from 13.5 KB to 314.5 KB. As the network grows, the share taken by pointwise convolutions rises from 47 % to 88 %, and total inference time grows from 54 ms to 1290 ms. Optimizing anything other than the 1x1 projections is not worth the effort on this target.
Everything under nn/ is a single pipeline with one source of truth for hyperparameters
(config.py) and one declarative list of experiments (runs.py).
flowchart TD
A["Google Speech Commands v2<br/>12 classes"] --> B["Preprocessing<br/>MFCC 49x10, label encoding"]
B --> C["Train DS-CNN, FP32<br/>50 epochs"]
C --> D["PTQ<br/>500 representative samples"]
C --> E["QAT<br/>25 epochs @ 1e-4"]
D --> F["Serialize to TFLite INT8"]
E --> F
F --> G["export_to_c.py<br/>model_data.c/.h"]
G --> H["Flash into the model partition"]
- Dataset: Google Speech Commands v2, silence and unknown balanced at 10 % each
- Augmentation: +-100 ms time shift, background noise mixed in with probability 0.7 at up to 0.1 amplitude
- Training: 50 epochs, batch 100, cosine decay from 1e-3, L2 1e-4, label smoothing 0.1, fixed seed
- Quantization: full INT8 PTQ with 500 representative samples, and QAT for 25 epochs at 1e-4, both branching from the same FP32 checkpoint so the comparison is paired
The sweep covers 134 architectures across three groups: an inherited baseline set, a dense grid in the fast zone (32 to 128 filters), and a set of gap-filling points, ultra small models and extra depth.
 |
|---|
Accuracy over the grid: filter count vertically, block count horizontally.
 |
|---|
INT8 model size against accuracy, Pareto front in red. The reference architecture from the literature does not lie on the front: at the same accuracy, smaller models exist.
 |
|---|
The same front against measured energy per inference. The most accurate model in the sweep buys 0.17 percentage points for 3x the energy, 451.2 mJ against 149.6 mJ.
PTQ or QAT. Both were run on 63 architectures from the same FP32 checkpoints.
 |
|---|
Median difference: +0.05 pp, with 98 % of models inside a +-0.65 pp corridor, which is the Wilson confidence interval of the test set itself.
Important
For INT8 on DS-CNN keyword spotting, QAT buys nothing measurable, so PTQ was selected: same accuracy, far less work to deploy. The conclusion is deliberately narrow. At 4 or 2 bits, or on attention architectures, the literature shows QAT does pay off.
cd nn
python -m data.download # ~2.3 GB
python -m precompute_mfcc # cache features once
KWS_GROUP=B python -m train_all # train a group
python -m quantize_ptq --slug f104_b5
python -m export_to_c --slug f104_b5_ptqResults land in nn/results/all_models_final.csv: parameters, FP32 accuracy, INT8 accuracy
and size for both quantization methods, per architecture.
measurements/ holds the hardware measurements behind the thesis, in three views per
configuration:
| Directory | Contents |
|---|---|
energy/ |
current traces: timestamp, bus_voltage, bus_current, state |
profile/ |
per-operator inference timings: run_id, op_index, op_tag, ticks_us |
stats/ |
aggregated per-operator statistics: mean, std, min, max, median, % of time |
basic/ covers a reference set of models; best/ covers the 18 architectures on the
Pareto front, each measured three times with the standard deviation reported.
scripts/ turns the raw exports into figures: profile_analysis.py produces the per-layer
time chart, bulk_profile_to_stats.py aggregates runs, make_pipeline_figure.py draws the
feature pipeline.
Requires ESP-IDF 5.x and PlatformIO.
# 1. pick the model
# edit custom_model_slug in platformio.ini, e.g. f104_b5_ptq
# the slug must exist in nn/runs.py and have been exported
# 2. export it if you have not already
cd nn && python -m export_to_c --slug f104_b5_ptq && cd ..
# 3. build and flash the application
pio run -t upload
# 4. flash the model into its own partition
parttool.py write_partition --partition-name model \
--input nn/results/merged_runs/f104_b5_ptq/model.tflite
# 5. watch it
pio device monitorBuild flags of note: USE_CUSTOM_KWS=1 selects the DS-CNN backend over WakeNet9,
RUN_SELFTEST=1 runs a startup self test, BOARD_HAS_PSRAM=1 enables external RAM for the
MFCC scratch buffers.
src/ ESP32-S3 firmware, C and C++
nn/ training, quantization and export pipeline, Python
models/ds_cnn.py the network
runs.py the 134-architecture sweep definition
config.py single source of truth for hyperparameters
results/ per-architecture accuracy and size
measurements/
basic/ reference set: energy, profile, stats
best/ Pareto-front architectures, three runs each
scripts/ measurement post-processing and figure generation
components/ vendored esp-tflite-micro
docs/ datasheets, reference material and README figures
data/ layer timing exports and figures
partitions.csv flash layout, including the dedicated model partition
platformio.ini build configuration and model slug selection
- itmo-bachelor-thesis research write-up, measurement methodology, Pareto fronts, energy decomposition, defense slides
- precise-power-logger firmware for the logging domain: INA228 readout, synchronization with the device under test
Contact: boris0indeed@gmail.com, telegram @worthant.