Hands-free Listam: talk to the ESP32 leaf and have items added, removed, or dictated as notes — without touching a screen. The leaf only listens for a wake word and streams the utterance; everything intelligent runs on a writable host peer, and the result replicates back like any other change. Scope is deliberately small to start: three wake words, three commands, and a dictation notetaker.
The INMP441 mic feeds an on-device wake detector. A loudness (dB) gate fronts the "yo" word so only a deliberate, loud "yo" triggers. Audio stays on the device until a wake word fires.
leaf-esp32/src/voice.rsAfter wake, the leaf streams PCM16 (16 kHz mono) over a dedicated side-band TCP connection until end-of-utterance (VAD silence) — the proven replication socket is left untouched.
audio-bridge.mjsThe host runs whisper.cpp on the utterance, then a deterministic per-locale grammar
turns the transcript into {intent, slots}.
The controller maps the intent onto the existing add/remove/note write paths and appends to the writable core; the new item replicates back to the leaf.
voice-controller.mjsThe always-on peer runs the bridge under Node and shells out to a real
whisper-cli binary. It works while the desktop is closed and is the
audio sink that also captures the training dataset.
A leaf can instead stream to the desktop's audio bridge on TCP :9994
(DEFAULT_VOICE_PORT). The Pear worker starts the same shared
pipeline — audio-bridge → voice-bridge → STT → controller → LED feedback — but the
STT engine spawns whisper.cpp via bare-subprocess, and the
controller writes through the injected callBackend (RPC_ADD,
RPC_DELETE) into the desktop's own base — no separate
peer, no second base, no pairing round-trip.
One wake word is actually built — and it fires on real ESP32-S3 hardware.
A ~62 KB int8 streaming MixedNet KWS model (yo.tflite)
runs entirely on-device behind a loudness (dB) gate, reusing the mic's existing RMS/dBFS
math. Two more phrases are aspirational and not yet trained.
| Wake word | Detection | Status |
|---|---|---|
yo | on-device model + dB gate | Built & fires on-device. A clear "yo" scores max_prob ≈ 0.996 (well over the WAKE_PROB_CUTOFF of 0.90); ambient sounds that merely trip the loudness gate score ≈ 0.03 and are rejected. The dB gate fronts it so a quiet "yo" never even runs the model. |
hey listam | model (planned) | Unbuilt — aspirational. |
dai dai dai | model (planned) | Unbuilt — aspirational. |
components/mww bundles
esp-tflite-micro plus the vendored rhasspy/pymicro-features
fixed-point audio microfrontend (the exact code microWakeWord trains on), wrapped in a C++
shim so the on-device features byte-match the training pipeline. The corrected
kFeatScale (raw filterbank ÷ 25.6, was ÷ 64 — which under-scaled
features ~2.5× toward the int8 floor) is what finally made the model fire. The 64 KB
tensor arena is allocated in PSRAM (MALLOC_CAP_SPIRAM) so it
doesn't contend with the hypercore stack.
Parsed by a deterministic grammar (English complete; the other five UI locales follow, accent-folded). Wake words spoken in the pre-roll are stripped before parsing.
| Say | Does | Intent |
|---|---|---|
| "add milk to groceries" | Adds the item to the named list (resolved via the list registry). Bare "add milk" uses the default list. | add_item |
| "remove milk" | Removes every exact match across all lists. Never deletes board tickets or registry entries; ambiguous fuzzy matches are a safe no-op. | remove_item |
| "note … end note" | Saves the dictated text as a note (see Notetaker). | note |
parseIntent("add milk to groceries")
→ { intent: "add_item", slots: { item: "milk", list: "groceries" }, confidence: 0.9 }wake.fired,
the command is treated as addressed and executes. This replaced re-detecting "yo" in the
transcript, which wrongly gated clearly-addressed commands whenever whisper mis-transcribed
the spoken "yo" (e.g. as the Italian "io"). It falls back to text detection
(detectWake) only for older firmware that doesn't send the label.
Parsing is not enough to mutate a list. Each intent has a write-gate confidence floor
(DEFAULT_EXEC_FLOORS) that the parse must clear when no wake word fired, so
ambient speech that merely parses can't touch the lists. The strictest bar is on the
destructive remove:
| Intent | Floor (no wake) | Why |
|---|---|---|
add_item | 0.75 | Anchored "add milk" still works hands-free; the lenient 0.6 retry path is excluded. |
note | 0.75 | Same non-destructive bar as add. |
remove_item | 0.9 | Destructive — above the grammar's max for a remove (anchored remove tops out at 0.85), so a wake word is effectively required to delete. |
A wake word short-circuits the floor entirely (the user clearly addressed the device).
Floors are overridable via config.voice.execConfidence.
"note <stream of words> end note" captures free-form speech and saves it to a
dedicated notes destination — a new net-new list type
(NOTES_LIST_TYPE = 'notes', mirroring the text-only todo type,
no dual-read shim). Note items sync now; a dedicated Notes surface lands in the desktop
and mobile UI later.
Both engines share one args builder and one multilingual GGML model (en/es/de/fr/it/pt):
whisper-cpp-subprocess.mjs shells out under Node on the
headless peer; bare-whisper.mjs spawns the same binary via
bare-subprocess inside the desktop worker.
Tether's QVAC SDK (ASR = whisper.cpp under the hood) would run natively in the Bare worker and distribute models P2P over hyperdrive — a spike-gated upgrade for the desktop path, with an optional local LLM for multilingual intent. Not yet built.
lib/stt/qvac.mjsbuildWhisperArgs forces
a low-latency decode — -bs 1 (beam size 1), -bo 1
(best-of 1), -nf (no temperature fallback), plus -t threads
and -nt -np (no timestamps / no progress). The spoken locale is threaded
end-to-end and passed as -l <locale> (omitted, so whisper
auto-detects, only on the "auto" fallback). A per-locale default --prompt
biases whisper toward the command vocabulary as an anti-merge guard ("add milk", not
"Admilk"). This fixed the earlier medium.en bug: that model is
English-only, so it ignored -l it and silently mis-transcribed Italian —
the fix is to use a multilingual GGML model.
Every leaf utterance the audio bridge receives is persisted as a WAV + JSON sidecar, auto-labeled from the wake-labeled END frame, building a real-life "yo" corpus to retrain and improve the wake model.
[reason, fired, probMilli, featPeak], so each clip is tagged a
positive that fired the on-device "yo" wake versus a hard-negative that
only tripped the dB gate. Hard-negatives are exactly what cuts false-accepts. The headless
peer is the audio sink, so the corpus accumulates on the always-on instance; it is wired
into the headless service and on by default — set LISTAM_VOICE_DATASET=0 to
disable (directory via LISTAM_VOICE_DATASET_DIR). The filesystem is injected
and saves are fire-and-forget, so dataset capture never blocks the live pipeline; old clips
are pruned past a cap.
The board has no buzzer, so its onboard addressable RGB LED reports state (driven over RMT). The state machine is owned by the voice thread.
| State | LED |
|---|---|
| idle / normal (dB gate only) | off |
| wake word recognized / listening | blinking yellow |
| command recognized | purple |
| saved / sent | solid green |
| heard you, didn't understand | red |
| BLE provisioning (advertising, waiting to pair) | blue |
The host command core (notes list type, multilingual intent parser,
add / remove-everywhere / note controller with write-gate floors); the leaf voice
front-end (INMP441 I2S mic, dB gate, side-band PCM16 streaming, multi-addr
connect_any, RGB LED); the on-device "yo" wake word
(fires on real ESP32-S3); the host audio + STT bridge (whisper.cpp); desktop-native
hosting in the Bare worker; and training-dataset capture.
More wake words and better "yo" recall (retrain on the captured corpus); the QVAC desktop STT upgrade; and a dedicated Notes UI surface in the desktop and mobile apps.