Trick-or-Treat — keyword spotting

A wake-word model for the phrase "trick or treat", trained to run locally on resource-constrained hardware.

task ......... 3-class keyword spotting
classes ...... trick_or_treat · unknown · noise
input ........ 1 s windows, 16 kHz mono, 16-bit PCM
project ...... Edge Impulse 1150125

What was built

Three model families, trained on identical data with identical hyper-parameters, so the comparison isolates architecture rather than tuning.

window .............. 1000 ms
training split ...... 810 windows (train) / 90 windows (frozen test)
validation .......... 10 % (81 windows)
training ............ 100 cycles, lr 5e-4, float32
augmentation ........ SpecAugment (warping, freq/time masking, gaussian noise)
                      — the two conv models; the transfer-learning block refuses
                      augmentation outright ("requires a spectrogram input")

Results — frozen 90-window test set

model            accuracy   trick recall   FNR    noise recall   unknown recall
transfer KWS      97.8 %      100.0 %      0.0 %     95.7 %          95.5 %
1D conv           98.9 %      100.0 %      0.0 %    100.0 %          95.5 %
2D conv           97.8 %      100.0 %      0.0 %    100.0 %          90.9 %

The accuracy column is a tie and should be read as one. 89/90 vs 88/90 is a single window; the 95 % Wilson intervals overlap almost completely. All three caught 45/45 positives, so all three show FNR 0 %, and each model's entire error budget is one or two unknown/noise windows.

What actually separates them is confidence margin — how much room the model leaves before it stops being sure. Re-scoring the same windows at a higher firing threshold, without retraining:

Push the threshold to 0.9 and the transfer-learning model loses 5 of 45 positives (FNR 11.1 %) and more than a third of its distractor windows. The conv models hold the phrase at 0.8 with zero misses, and the 2D model still holds 100 % at 0.9. On a device that must fire despite room noise, that margin is the difference between "works" and "works when you shout".

Recommendation: 2D conv, threshold 0.8. 1D conv is a reasonable fallback; the transfer-learning model has the least headroom.

Honest limit of these numbers

The 90-window test set is a window holdout, not a voice holdout: most of the synthetic voices appear in both train and test. The figures above therefore measure utterance matching, not speaker generalisation. A model that misses a stranger at the door can still score 100 % here.

Dataset

size ......... 900 × 1.000 s  (15 minutes)
balance ...... 450 trick_or_treat · 225 unknown · 225 noise
format ....... 16 kHz mono, 16-bit PCM, exactly 16 000 samples per file
licence ...... CC BY 4.0

Positives contain one complete utterance each. Clips longer than 1000 ms were time-compressed rather than truncated — a clip that loses the "treat" but keeps the label teaches the model the wrong thing. Every compressed clip was re-transcribed to confirm the phrase survived; 28 takes were compressed, none dropped.

Unknown words are chosen as near-misses on purpose. This class is what stops the detector firing on "tree".

Noise is front-door material: traffic, wind, rain, footsteps, a doorbell, TV bleed, kitchen clatter, and silence carrying a room's own hum.

Dataset limitations

• All positives are synthetic (pocket-tts's without-voice-cloning build — no real voice, no cloned voice). A model trained only on this data can still miss a real human speaker; consented, licence-compatible real voices are the most valuable contribution this dataset can receive.
• unknown is the smallest class, and the weakest in every model — it is where false positives come from.
• All recordings share a single room's noise profile.

Attribution

Dataset distributed under CC BY 4.0. Built from pocket-tts (Kyutai, CC BY 4.0), Google Speech Commands v0.02 (CC BY 4.0, © Google LLC), and CC0-1.0 noise material including a room recording made for this project.

Please keep this attribution with the data.