This is a submission for the Gemma 4 Challenge: Build with Gemma 4
What I Built
I live with my wife and our 1.5 year old daughter. Both me and my wife are working and usually have busy schedules. My wife manages the stock keeping of grocery and daily essentials and our cook and maid frequently ask her to order stuff that is running out. These requests usually come on-the-fly, and my wife has to keep a mental note and then add it to her ordering list. Later when she gets time, she adds items to cart (usually the likes of Amazon, Blinkit etc). Many times she even forgets items and we end up compromising on some dishes which can't be prepared because some ingredients are out of stock.
I built Piclu, a product that captures voice notes, and installed it in our kitchen. Now our cook and maid just press a button and leave a voice note whenever something comes to their mind. No runtime interruptions to my wife. The voice notes get transcribed and structured with Gemma 4 E4B, and the shopping list stays populated for my wife to see whenever she gets time. She then orders as appropriate. Our cook and maid are also happier as they don't have to wait for my wife to be around to share stock outs and they also now have fewer things to keep in their memory.
For the voice capture - I programmed an M5Stack Atom Echo as a Push-To-Talk device and plugged it into the kitchen socket. It records the audio locally and then pushes it to the backend endpoint. Everything runs locally on a small home server, no cloud, so the audio never leaves home.
A fair question here would be: why not just ask the cook and maid to write down the requests or send a WhatsApp voice note? Two reasons. Writing it down is more friction than it sounds when your hands are busy in the kitchen, and not everyone is comfortable writing. And even if they did write it down somewhere, my wife would still have to read through it and add each item to the cart one by one. We tried WhatsApp voice notes as well but it was too much of an ask when they are in the middle of work and in a hurry.
Demo
In the video you can see the full loop - the helper presses the button by the counter and leaves a note in Hindi ("nimbu aur tamatar khatam hai", out of lemons and tomatoes), Gemma 4 transcribes it onto the list locally, a second Gemma 4 pass turns the notes into a structured shopping list, and each item shows up on its own as a tickable to-do.
Code
open-piclu
A household voice-note brain powered by local Gemma 4 E4B. Captures voice notes from anyone in the house (cook, family, helpers), transcribes them in their own language and accent, and turns them into a structured shopping list the household manager can review and act on.
Built for the Gemma 4 Challenge: Build with Gemma 4. Audio arrives over a single HTTP endpoint — bring your own push-to-talk source (the bundled web UI, a Raspberry Pi with a button, a Home Assistant blueprint, an IFTTT webhook, your phone — anything that can POST a WebM/Opus blob to /api/items/freeform_audio).
Data flow
PTT source ─► POST /api/items/freeform_audio (multipart: audio)
─► app HTTP
─► brain gRPC over Unix socket
─► recordings.Store (seals .webm on disk + emits SealedEvent)
─► processor: ffmpeg decode (WebM/Opus → 16 kHz mono PCM)
→ Gemma 4 E4B pass 1 — TRANSCRIBE (multimodal audio →
Hinglish text,…
The open sourced repo has the Gemma brain (transcription and structuring), the web UI, and the single audio endpoint that the M5Stack pushes to. It also has the full transcription and extraction eval I talk about below.
Under the hood it is a Go app and brain talking over a Unix socket, the cart and notes update live over SSE, and the whole thing runs from a Docker Compose. Gemma 4 runs locally through a libmtmd / llama.cpp build, with a Lemonade adapter for faster GPU inference (the eval uses the same adapter), and the eval is fully reproducible from the repo.
How I Used Gemma 4
Gemma powers 2 key steps in the note capturing flow - transcription and structuring. Both run as local Gemma 4 E4B passes on the home server.
Step 1: Transcription
When I started building this, I tried a few open source models and put them against each other on a small eval dataset I prepared from our own voice notes - Whisper, indic-whisper from AI4Bharat, and Gemma 4. The pure speech-to-text models had difficulty understanding the accents of our cook and maid when they spoke about our household items. Gemma 4 uniquely stands out, because of its native multimodality - it takes the audio and a text prompt together, so unlike a pure STT model I can actually tell it about what it is about to hear. Across the open-source landscape, Gemma 4 E2B/E4B are among the few models that support native audio and text input within a single prompt.
Among the Gemma variants, the bigger ones (26B MoE and 31B Dense) don't take audio at all, and the smaller E2B is not accurate enough, its word error rate comes out above 1.0 and it drops about 36% of the items during structuring. So E4B was the variant that is both multimodal on audio and accurate enough for this job. In the eval it also beat Whisper-Large-v3, the gold standard for STT with global language support, both on accuracy and on staying quiet on noise (Whisper tends to invent text on silence, which would quietly pollute the list). Here is the head to head:
| Model and prompt | WER | CER | No-speech detection |
|---|---|---|---|
| Gemma E4B, no context in prompt | 1.29 | 0.80 | 67% |
| Gemma E4B, full prompt (shipped) | 0.55 | 0.30 | 93% |
| Gemma E2B, full context in prompt | 1.48 | 0.90 | 20% |
| Whisper-Large-v3 (forced Hindi) | 0.70 | 0.36 | 7% |
Lower WER and CER are better (WER is word error rate, CER is character error rate; CER is the fairer read here because Romanized Hindi has many valid spellings). No-speech detection is how often the model correctly stayed silent or flagged the clip as noise instead of inventing words - the shipped E4B prompt teaches it those no-speech flags, which is why it handles 93%. Full method and the rest of the numbers are in the repo eval.
The context I give it is specific to our home. I live in Gurugram, India and our cook and maid are from West Bengal and Bihar respectively. I prompt E4B with what is being said, the purpose, by whom and intended for whom, so it can steer the predicted tokens keeping the intent in mind. That one change is the single biggest lever - in my eval, adding this context reduced the word error rate by more than half compared to a plain "transcribe this" (the no-context row in the table above). The relevant part of the transcription prompt looks like this:
You are an expert audio transcriber specializing in {{native_language}} dialects.
You receive voice notes from members of a household in {{city}}, {{country}} —
typically the cook, household helps, or family members — addressed to the
household manager regarding the shopping list (stock requests, brand preferences,
quantity changes, cancellations).
The full prompts (transcription and structuring) are in the repo under internal/prompts.
Step 2: Structuring
This pass turns the notes into clean items to be ordered. Notes can arrive with linked context as well, for e.g:
- Note 1: The dishwash liquid is running out.
- Note 2: The current dishwash liquid is not good, the earlier one was better.
I keep an SQL table keyed by a name_id, with kind, size, quantity, stock_state and notes. I pass the new transcript and the existing table items to Gemma and it generates a small DSL that the code then parses to insert or update items in the table:
- Note 1 returns:
UPSERT name_id="dishwash-liquid" name="dishwash liquid" stock_state="low" - Note 2 returns:
UPSERT name_id="dishwash-liquid" name="dishwash liquid" notes="current one is not good, the earlier one was better"
Both notes map to the same name_id, so the table ends up with a single merged row:
dishwash-liquid stock_state="low" notes="current one is not good, the earlier one was better"
If a note is not about the shopping list at all, Gemma emits NO_OP and nothing changes. In the eval, this structuring pass had a recall of 1.0 (it found every product) and an F1 of 0.99, and it never invented items from non-shopping notes.
Next I plan to build MCP servers for the online ordering platforms and then eventually auto populate the carts with the items to be ordered, contextualized using the order history on respective platforms. I am bullish that Gemma 4 can deliver here as well - it will come down to how well it does tool calling and how much its chain-of-thought thinking can optimize the ordering. Let's see what happens, fingers crossed.