I Built Four Second Brains in Four Months and Ended Up Where I Started

On my first day at Bandwidth, a guy on my team gave a presentation about keeping his Claude context in Obsidian. He used Markdown files, one per thing, written as he worked, so that when he came back to a model later, it had something to read and use for context. He'd call a skill at the end of a session to write the docs and another one to pull them back when he needed them.

It blew my mind at the time. The persistence of context was really cool, but after a few weeks, I began to think the whole thing was too manual. If I have to ask for the context, then the memory isn't ambient, and ambient was the entire thing I wanted. So I took his idea and went off to automate the part he was doing by hand.

Four months later I'm running his workflow in a different container, and I got there by building four systems that didn't work.

Hooks

The first one was the dumbest. I was in Claude Code all day, so I wired up start and stop hooks and had every message I sent and every message I got back appended to context automatically. It was expensive, it was slow, and frankly, it was stupid. I was paying to stuff a transcript of myself into a prompt. I killed it pretty quickly.

SQLite

Around this time, I was spearheading a project at work to build enterprise search over our data. We were calling it "talk to your data." It was the first time I'd gone deep on embeddings: how you chunk things, what the vector is actually holding, what you lose, and how retrieval behaves as the corpus grows.

So of course I thought, if I can do this to a company's data, I can do it to my own thoughts, so I built it on SQLite. sqlite-vec gives you vector storage and KNN search inside a plain SQLite file, which was exactly the shape I wanted for something personal. The catch is it stores vectors and doesn't make them, so I had to bring my own embedding model. I ran a tiny one locally on CPU. It was light enough to run locally, but weak enough that I was leaving quality on the table. And it was slow again.

MCP, and the good part

Then I put my SQLite database behind an MCP server I'd spin up every morning. It had two functions: push and retrieve. For a few weeks, it felt like I had built a little Jarvis for myself. I could sit down at home, open a session, say, "where are we on this," and most of the time it knew. Then it got worse. I was doing pure vector search, and as the store grew, the retrieval quality fell off. Semantic similarity is different from what I actually meant, and the more semantically similar entries I added, the more often I got a neighbor back instead of an answer. So I added text matching on top of the vector search and got to something like a 90% hit rate, which I was happy with. Published benchmarks put dense vector retrieval alone around 78% recall@10, keyword search alone around 65%, and the two fused together around 91%. I had rebuilt the industry's standard answer in my bedroom and landed within a point of it.

And it still didn't do what I wanted.

My whole idea about AI engineering back then was that I should be able to sit in my home directory, spawn as many Claude sessions as I want, start talking about anything, and have it pick up instantly. I had retrieval working about as well as it could, and I was nowhere near that. That should have told me retrieval was never the problem. I went and did more research and ended up on the Lance format, a columnar format built for exactly this kind of workload: fast random access to individual vectors rather than scanning whole columns, versioning, and embeddings sitting next to the rest of your data. It was good technology, and I was going to move everything onto it and build system five.

The YouTube short

What stopped me was a YouTube short from a creator with almost no following, showing off a site she'd built for very specific Claude skills. I went to Google to find the site, missed it, and landed on a much bigger skills marketplace I hadn't known existed, which promptly showed me the most popular skills on it. One guy kept coming up. Matt Pocock. His skills felt strict in a way I didn't understand at first, so I looked him up and watched a video, and it changed how I was thinking about all of this.

His argument, roughly, is that what makes AI-assisted engineering work is ordinary software engineering discipline rather than some new AI-native technique. Break the work down small enough that the model is operating where it's actually smart. Argue with it before you let it write anything. Write a spec and then work the board. That makes interacting with the model more like running functions from an IDE than having my own personal AI agent. I configured it the way I configured VS Code, exactly how I like it, and then used it to do work.

Where the context went

What I do now is put the context in the ticket. Whatever context a piece of work needs later goes in the ticket for that work. If I need to reference something I did in another repo or another domain, I don't need it floating in ambient memory. I just say, "Go read what I did on this ticket and see how I handled it there."

Scoping context to the work turns out to be the right approach, which means I spent four months trying to solve a problem that wasn't really there. Chroma published research on what they call context rot: across eighteen frontier models, output quality degrades as input grows, and it starts degrading well before the window is anywhere near full. A model with a 200K window can get measurably worse at 50K.

I'll also say this wasn't a solved problem that I was too dumb to solve. Agent memory in 2026 has its own benchmark suite and its own literature now. Memory staleness, where a stored fact stays confidently retrievable after it stops being true, is still open.

Full circle

That guy is still running his Obsidian vault. He writes his docs at the end of a session, every time, with a skill built for it, and pulls them back deliberately when the work calls for them. It works really well, and it has worked for as long as I've known him.

It's the same move I'm making. A Markdown file and a Jira ticket are the same idea in different containers: write down what matters while you still remember why it mattered, keep it scoped to the thing it belongs to, and retrieve it on purpose.

The manual part I objected to on day one, the part where I had to invoke it myself, was the whole point. I had thought of the manual work as overhead, but it turned out to be the system itself.

I just needed to build four of them to hear it.