Blog 19 September 2026

How I built my AI agent, so far

An assistant running on a machine in my house, keeping a knowledge base that grows every week, and publishing some of it to this site. Here's what it's made of.

Work in progress. It's a few weeks old and changing weekly. Some of this will look daft in six months. Writing it down now anyway.

Why it doesn't run on my laptop

I started by running it on my laptop. The session dies when you shut the lid, which you notice on about day one.

The useful part of an assistant like this is the context it's built up: what you decided, what you ruled out, what it already looked up. Losing that every evening is annoying. Leaving a desktop switched on permanently to avoid losing it is worse.

I wanted to start something on my laptop, add to it from my phone, and pick it back up that evening with nothing reset. So it runs in a container on a small machine that was already on in the house anyway. My laptop and phone are just windows onto it.

The container keeps the conversation going across devices. The knowledge base keeps the useful bits after the conversation ends. Two different jobs.

Four repos, not one

Everything the agent touches is in git, split across four repos:

Partly that's change rate. I didn't want a site I edit daily sharing a repo with something I touch monthly. Mostly it's blast radius: each repo has its own deploy key, so a key that leaks costs me one repo instead of all of them.

The fourth repo came out of a problem. The agent's container used to be described in writing inside the knowledge base: which base image, which packages, how it gets started. That description went out of date twice in a fortnight.

The reason was obvious once I saw it. When the agent needed a new tool installed, there was no file for it to change, only a paragraph describing a file it couldn't reach. So I made it real: the container is now defined in its own repo, as files the agent can actually edit.

If the agent can't edit it, it goes out of date.

What the machine looks like

Docker host: a machine you control Mine is at home. A VPS or dedicated box behaves identically. Your phone or laptop, anywhere Anthropic API Both sides connect to it. Neither connects to the other. GitHub Builds and smoke tests every push Private image registry Anyone on the internet Router Two ports forwarded. Nothing else. The agent in its own container Can push to git. Nothing else. Update watcher Pulls new images and restarts containers. Reverse proxy Certificates, 3 domains Password sits here, not in application code. Site containers public site the one people see preprod checked first private hub password protected outbound poll git push pulls images the only way in
inbound (the only one) started from inside, going out internal to the host
One arrow points in, and it's someone asking for a web page. Everything else starts from inside the machine.

Top left is the bit people ask about. My phone never connects to the machine, and the machine never connects to my phone. Both connect out to an API and meet in the middle. That's how I can pick a session back up from a hotel without the machine being reachable from one.

Deploys work the same way round. A change gets built and tested on GitHub, and the machine spots a new image and pulls it down. Nothing reaches in to install anything. That leaves one way in: someone requesting a page, on two ports, hitting a reverse proxy that handles the certificates and the hub's password.

Private by default, public on purpose

The knowledge base has my family's details in it, what's in our accounts, and work material that's commercially confidential. It also has the interesting parts of twenty years of building data platforms.

So the same store holds the things I'd most like to publish and the things that must never be published. That needs a real answer, and "be careful" isn't one.

The vault AUDIENCE: ME. NEVER SERVED. Everything, without editing: • Family, birthdays, the household • Accounts, capital, what it earns • Renewals, subscriptions, policies • Work material under confidentiality • Every original source, unedited • What I decided, and why Grows every week. An LLM writes and maintains all of it. Private repository. Not on the web, not reachable, no port open to it. I decide what crosses Nothing crosses automatically The hub AUDIENCE: ME, ON ANY DEVICE. PASSWORD. A daily briefing. Meal plans. Small tools that are useful on a phone and nobody else's business. Private, but on the web so I can reach it. The public site AUDIENCE: ANYONE Writing, resources, a CV, this post. Drafted from what the vault already holds, then rewritten for people who are not me, and checked for what must not appear.
One knowledge base, three audiences. The gate in the middle is me.

Getting something out is a rewrite, not a copy

You'd think the hard part is access control. It isn't. The hard part is that I can't just move something from the knowledge base to the website.

Anything from work gets rewritten first. Internal names come out, and so do figures that only make sense with context, and anything specific to my employer. Saying where I work is fine. Explaining how their systems work isn't.

Those rules are written down in the knowledge base, so the agent reads them before it drafts anything rather than me remembering to say it every time.

What the agent is allowed to do

It can commit and push. That's it. No access to Docker, no shell on the host, no deploy credentials, no way to restart anything.

Everything after that has its own gate. A push gets built, the site starts in a container, and every page is requested to check it actually loads. Only then does an image get published, and only I promote it to the public site.

That's not theoretical. A couple of weeks ago a change broke a page: a template error threw a 500 instead of content. The test caught it, nothing got published, and I fixed it before anyone saw it.

The question I found useful wasn't "what should the agent be allowed to do". It was "what happens when it does the worst thing it can do". That turned out to be an architecture question rather than a permissions one, and the answer is what you'd give any new colleague. Not root on production. A branch and a pipeline.

2f5c21b · 2026-10-29T08:59:05Z