Introduction
Every so often an idea comes along that I can’t stop thinking about, and a couple of weeks ago one of them finally made it onto my desk. A little buddy. A round face on a small screen, a microphone, a speaker and a personality. Tap it, talk to it, and it talks back. It’s nowhere near finished, but I’ve reached a good point to write up where it’s got to, so this post covers the idea, the build, the bugs that fought back, and the setback that’s paused the project while I wait for a new board.
The idea
I already run an AI agent on a server, which I wrote about in my Hermes post, and I mostly talk to it through telegram. Typing to a chatbot works well enough, but there’s something missing when there’s no face to it. I wanted a physical thing sitting on the desk that feels alive, something I can talk to like a person and that occasionally pipes up with something useful on its own.
The personality is a character called Kai, a laid back California surfer type. Short spoken replies, no markdown, no emoji, just a voice on the desk. It’s a daft idea in the best way, and it’s made me think properly about voice as an interface for the first time.
These kinds of devices have started to flood the market and are all over Aliexpress but I wanted something that I controlled which was my own. The goal was that if I build the ecosystem and wrote the firmware I could genuinely have something that feels useful and personal.
The hardware
The heart of it is a Waveshare ESP32-S3 Touch AMOLED 1.75 board, which turned out to be far more capable than I expected from the price:
- A 1.75” round AMOLED screen at 466x466. The roundness is the whole point, it makes a perfect face
- Dual microphones with echo cancellation built in
- Touch, USB-C, Wi-Fi and Bluetooth
- LiPo battery support through an AXP2101 power management chip
The only extra bit of hardware is a small speaker that plugs into a header on the board. Everything else, the brain, lives on the server. The ESP32 just captures audio, plays audio and draws the face.
How it works
The design is deliberately simple. There’s no wake word, so it’s tap to talk:
- Tap the screen, hold it, and speak
- The audio streams over a WebSocket to a bridge server on my VPS
- The bridge runs Whisper locally for speech to text
- The text goes to the AI brain, usually a fast direct API call with Kai’s personality as the prompt. Questions that need live data, like weather or surf reports, get routed to the full agent with all of its tools instead
- The reply comes back as speech and streams out of the speaker
All of it runs over TLS through Cloudflare at kai.jaymeh.co.uk, which means the buddy works from anywhere, not just my home network. The bridge runs as a systemd service on the server, so it survives reboots and restarts itself if it crashes.
Here’s a very early prototype, back when it just had a couple of buttons and a voice:
The build
The firmware is Arduino with LVGL for the face. It has a tap to talk button, state driven eyes and a setup mode. One of the best decisions early on was a settings portal, so WiFi, token and volume are all configurable from a screen on the device itself rather than being compiled into the firmware. I don’t have to reflash the thing every time I change the WiFi password or move to a new network.
The server side has grown a few extras since the basic loop worked. The one I like most is push. I can send text to the server and the buddy will speak it out loud, which means the agent can pipe replies to the desk whenever I say “tell me” in chat.
This would start to tie in nicely where Kai can actually tell me things on cue, maybe we’ve found a new car that match my requirements or a product I’m watching has dropped in price. The possibilities are endless!
The bugs
This is where it got interesting. The worst problem was Kai replying to things nobody had said. Whisper would transcribe near silence as a single word, “You”, “Thank you”, “The”, and Kai would cheerfully answer it, so I’d get responses to questions I never asked. The fix ended up being three layers of defence. A voice activity filter, a minimum clip length of a quarter of a second, and a noise gate that measures the actual volume of the clip before Whisper even runs. It’s been quiet ever since.
Latency was the next big one. The first working version took 21.9 seconds to answer. That’s fine for a demo but awful for a conversation. When showing it to people, this was always the first thing they mentioned. Calling the model API directly instead of booting up a whole command line client got it down to 9.7 seconds, and then I added a tiny classifier that sends tool heavy requests to the full agent while keeping casual chat on the fast path. Voice is unforgiving about latency in a way chat just isn’t.
Keeping it light
The biggest learning curve for me has been how low level all of this is. My day job is PHP, mostly WordPress. In PHP you write a variable and you don’t think about it again. The server has gigabytes of memory, the garbage collector sorts everything out, and if something does go wrong you get a stack trace and a log line to read.
On the ESP32 every byte matters. The board has 8MB of PSRAM and a lot of that is spoken for before my code even runs. The display, the audio buffers, the web socket and the graphics library all want their share, and if you get the balance wrong the device just crashes mid conversation. No stack trace, no error log. It reboots and you start again.
I’ve had out of memory exceptions, random reboots, and connections that dropped with no explanation. The classic one was the TLS connection failing at startup because the certificate store allocation failed. On a phone that’s a rounding error. On this board it’s fatal. The microphone produced nothing at first either, because the separate audio chip on the board needs initialising before the ESP32 can talk to it. Layers of hardware I’ve never had to think about in my working life.
I’ve learned to keep things light as a discipline. Audio is streamed in small chunks rather than buffered in big gulps. Every feature on the server side is easy to add. Every feature on the device side is a negotiation about memory.
The face
The face has been reworked more times than anything else and it’s still not where I want it to be. The whole personality lives or dies on what that little screen draws, so it matters more than any other part of the build.
There’s an idle state, listening, thinking, talking. Eyes that move and react. I’ve done numerous reworks of the animations and I keep coming back to the same problem. Making Kai feel alive on the desk means animations, and animations cost memory and CPU. Every frame is a small piece of RAM that could be the difference between a smooth conversation and a crash. I want him to blink, to look at you when you talk, to react to what’s happening, but every one of those things is a trade off against the device falling over. I even started migrating the display code to a newer version of the graphics library to get nicer effects.
It’s a balance I’m still working on, and it’s the bit of the project I think about the most.
The case
With the electronics working, the fun part started. 3D printing a body for it. The design idea is that the round screen is the face and the shell just adds the character around it. Ears, hair, a collar, a little cap.
The board has no mounting holes, so the case is a sandwich. A front shell with a circular bezel for the screen and a back shell with the cavity. I started by remixing an existing case on MakerWorld for this exact board, because remixing beats measuring, and then remodelled it as my own design. The current dimensions are a 46mm wide head at about 12mm deep, with cutouts for USB-C and the buttons, a speaker grille, and a pocket for the battery I’ll add later.
I’m five prototypes in and each one has taught me something about tolerances. Screen clearance, snap fits, the difference between 0.2mm and 0.3mm. I’m printing in PLA+ for fast iteration and will do the keeper in PETG. There are more iterations to come and the printer has been earning its keep.
The setback
A few days in, the original board died. The screen cable got caught while I was working on it and the display was damaged. The board itself is probably fine, but without a screen there’s no buddy.
I’ve got replacements on order. A cased version with a built in speaker, plus a bare backup board as a spare. They’re coming from AliExpress so it’s a waiting game. The project is paused until they land.
What’s next
- More case iterations. The body is the bit I’m enjoying most
- A wake word eventually, so I don’t have to tap it
- Battery percentage and auto dimming on the face
- Better face animation states
- Streaming TTS so the first words come out in a few seconds instead of waiting for the full reply
- Maybe a 4G HAT so the buddy works anywhere, not just on Wi-Fi
What I learnt
The biggest thing I’ve taken from this project is how much I enjoy being this close to the hardware. Working in PHP for so long you get used to a safety net, and it’s refreshing to be somewhere where every decision has a real cost. The other big lesson is that latency is the entire experience when it comes to voice. Nobody notices a delay in chat. Everyone notices a pause before a voice answers.
The boring infrastructure turned out to be half the work as well. TLS, certificates, a firewall rule, a systemd service. None of it is interesting until it stops working, and all of it is necessary before the fun part can happen.
Most of all I’m learning that restraint is a feature. On a small device the most important decision is what not to add. The buddy that stays alive and talks to you is worth more than the one with fancier animations that keeps crashing.
I’ll write a follow up when the new board lands and Kai is back on the desk. See you next time.
