In my last post I fixed a bug in one of my side projects while sitting in a doctor's waiting room in Berlin. After that, the obvious question was what's actually running at home? So this one is about that.
It's a Raspberry Pi, a tiny computer about the size of a credit card. In the last post I called the agent a waiter. It takes your order and brings back the result, while the actual cooking happens in someone else's kitchen. This post is about where the waiter lives and how it ended up there.
And to be upfront, most of what I planned in the beginning didn't survive the first week. The parts that broke taught me way more than the parts that worked.
The plan that died
My first plan was kind of over engineered. I wanted a small helper that would look at every task first, is this coding, research or just a quick question, and then pick the cheapest AI model that could still do it well. I'd pay per message, but always the lowest price. A bit like a travel app that hunts for the cheapest flight every single time.
It didn't last. The rankings I wanted to use show which models are most popular, not which ones are best for your kind of task, and popular is not the same as good. The bigger reason though was that I was already paying monthly subscriptions for ChatGPT and Claude. It's like having a monthly train pass and still comparing ticket prices before every ride. There's nothing left to save.
So that idea went, and a couple of other tools went with it. What's left is boring and cheap. The agent runs on my ChatGPT subscription, and a tiny bit of pay as you go is left for small background chores like giving conversations a title, around a cent a day.
Funny enough, the same marketplace I wanted to use for this, OpenRouter, later launched its own tool for testing models on your own work, and their pitch was basically my problem. A popularity list can't tell you how a model handles your tasks. So the idea wasn't wrong, the way I wanted to do it was.
Why a Pi is enough
If you read the last post you already know the trick. I'm not running an AI model at home. The model is the kitchen and OpenAI runs it. What lives at home is the waiter, the agent that takes my messages, remembers things, works on my projects and asks the kitchen whenever it needs to think.
And a waiter doesn't need much, so the Pi is plenty. I did add a proper drive to it, because the little memory card it starts with, the same kind you'd find in a camera, wears out if something writes to it all day long. And an agent is constantly writing, it keeps notes of everything it does.
The Pi was already running my home media server and a few other things, so the agent just moved in next to them. It runs day and night, makes no noise, and I don't notice it on the power bill.
What's actually running
Everything inside the big box runs on the Pi. I message the agent on Telegram, just like texting a person, and for the thinking part it calls OpenAI's model through my subscription. That's the only bit that happens outside my home.
When it has to do real work, like changing code on my website, it does that inside a sandbox. Picture a sandpit with walls. The sand, which is the agent's work, stays inside and can't spill into the rest of the computer. But on a windy day sand can still blow out, so there's also a cover on top. That's the gatekeeper. Anything the agent wants to send out to the internet has to pass it, and it only lets things through to places I approved. How well the sandpit really held up is a story for the next post.
Searches stay at home too. Instead of going straight to Google, the agent uses a private search engine running on the Pi.
There are also a few things that run on a timer, like a message every morning at 7 with the weather and what to wear. That one doesn't use AI at all. It's a simple fixed routine, because a task that's the same every day doesn't need any judgement, so I keep the AI for the things that do.
One agent per person in the house
Once mine was working, the next question came from the other person living here. Can I have one too?
Hermes has profiles and each profile is basically its own assistant. My better half got her own Telegram bot, her own memory and her own personality file, which she can shape however she wants. Both live on the same little computer, so adding the second one cost almost nothing.
The part I think matters most is privacy. People tell their assistant all kinds of things, work stuff, health questions, plans, half formed ideas, and everyone deserves their own space for that, the same way everyone has their own phone. So each profile keeps its own conversations and its own memory, and one assistant can't read the other's. You can connect them if you want them to work together, but only on purpose, and we haven't needed to.
What we do share is the machine and the ChatGPT subscription, so a busy week for one of us means the usage limit comes a bit sooner for both. It's just one more thing we share at home, like the Wi-Fi.
Green lights that lied
If I had to pick one lesson from the whole build, it's this. Something showing as working and something actually working are two different things, and I had to learn it twice.
The first time was search. My agent is supposed to search through the private search engine on the Pi, and searches kept giving results, so I assumed all good. It wasn't. Think of it like asking someone to always shop at your local market. The groceries keep arriving, so you never check, but they've actually been coming from the big supermarket every single time. Because of a small setup mistake my agent could never reach my private search, so it quietly used public search services instead. For days. Now a little routine checks every morning and messages me if it ever happens again.
The second time was storage. I bought that proper drive so the camera style memory card wouldn't wear out, and moved everything over to it. Or so I thought. Weeks later I found more than 10 GB still sitting on the old card. It was like moving house and finding half your boxes still in the old basement, because the movers only packed the rooms they were shown. What I thought was done was only part of the job.
Neither of these broke in an obvious way. The ones that crash, you notice. The ones that keep looking fine are the ones that get you.
Why not just use Muse or Dots?
While I was still building this, two big companies launched pretty much exactly this kind of product. Meta released Muse in early September and OpenAI released Dots at the end of the month. Both are personal agents that are always on, do things on your behalf, and each one runs on its own computer in the cloud. So the fair question is, would I still build mine today?
Where they clearly win: no Pi to look after and no hunting for where Docker hid its files. They connect to way more apps than mine, Dots talks about thousands, and they come with proper apps, avatars and safety features I would never build myself. Dots for example has a read only mode, so it can't control your browser or computer while you're not there. And there's a whole security team behind each of them, not just me.
Where mine wins, at least for me. Muse launched in the US first and Dots is only for OpenAI's Pro and Business Premium plans in some markets, while mine works here in Berlin on the subscription I already pay for. My agent's memory and files stay on a box at home, whereas Meta uses Muse conversations to train its models unless you opt out, though they say a mode where they can't see your workspace is coming before the end of the year.
When OpenAI retired the model my agent was using, I switched to a newer one in a minute, and I could switch to a different company's AI altogether if I wanted. Two separate assistants for two people on one small machine costs me nothing extra. And I can see everything. Everything it does is written down where I can read it, and its rules are simple files I can change, which is also how I found the problems I'll write about next.
So honestly, for most people I'd say try the products first. They are easier and they'll keep getting better. Build your own if you like tinkering, care about where your data lives, or want to understand what an agent is really doing on your behalf. I'm all three, so I'd build it again, but I wouldn't tell everyone to.
Before you try this
A few honest things before anyone copies this.
It takes time. It took me weeks of evenings and a lot of digging into what went wrong, so if you want an assistant and not a project, this isn't it. You're also the security team, nobody fixes the holes for you and nobody tells you when something is off. The next post is pretty much about how that went.
It also still depends on the cloud. The thinking happens at OpenAI, so if their service is down or my subscription changes, my waiter has no kitchen. And it's not free. I already had the Pi, the drive and the subscriptions, but starting from zero it costs real money.
What's next
Two more posts are coming. One is about the sandbox not being where I thought it was. I built the agent a sealed room and it turned out to have a side door, and funnily enough the agent itself was the one who showed me. The other one is about video, where the agent writes the video and my laptop renders it, and how that took my demo videos from rough to actually good.
Until then, the waiter keeps working.