Building an AI Assistant That Knows You

By the start of 2026, remembering became table stakes. ChatGPT remembers your preferences across chats. The other big assistants shipped their own versions within a few quarters of each other. Ask one what you talked about last week and it can usually tell you.

And yet most of them still feel like they're meeting you for the first time. They recall facts without seeming to understand you. They bring up the wrong thing at the wrong moment, or nothing at all. The memory is there. The knowing isn't.

That gap is the whole problem, and it's worth being precise about why it exists. Remembering is the easy job. Knowing someone takes three more jobs that most products skip.

The field already mapped this

The interesting work here isn't new. In 2023, a Stanford and Google team built "Generative Agents," a little town of simulated characters that planned their days and remembered their interactions. The part everyone fixated on was how lifelike it felt. The part that mattered was the architecture. Each agent had a memory stream, and on top of it, two operations: retrieval that scored memories by recency, importance, and relevance, and reflection, where the agent periodically paused to synthesize raw memories into higher-level conclusions about itself and others.

That second operation is the one to notice. The agents didn't just store what happened. They digested it.

Around the same time, the MemGPT paper made the other half of the case. It treated the model like an operating system managing memory across a hierarchy: a small fast working set in the context window, and larger, slower stores it pages in and out as needed. Karpathy had been sketching the same picture, the model as a CPU and the context window as its RAM. The framing is now common enough to be a cliché, but it carries a real instruction. Memory is not one thing. It's a stack of things with different speeds and different jobs.

Put those two ideas together and you get a working definition of what it takes to build an assistant that knows you. Not a bigger model. A loop.

The four jobs of memory

Capture is recording what's worth keeping. This is the job everyone ships, because it's the easy one. Write down the fact, the preference, the decision. The trap is capturing everything, which is the same mistake as stuffing the context window: noise crowds out signal, and you pay for it later.

Consolidation is the job almost nobody ships, and it's the one that separates a notebook from a memory. New information has to be reconciled with what's already there. Contradictions resolved. Ten scattered observations compressed into one durable conclusion. This is the Generative Agents "reflection" step, and it's not a coincidence that Anthropic has been productizing a version of memory consolidation it calls Dreaming. Without it, a memory store just grows, gets contradictory, and slowly rots. With it, the system actually learns a person instead of accumulating trivia about them.

Recall is surfacing the right memory at the right moment. The naive version dumps everything relevant into context and hopes. We saw in the last piece where that goes: more context, less performance. So recall is a relevance problem, not a retrieval problem. The skill is bringing up the one thing that matters now and leaving the rest in the drawer.

Calibration is letting the person correct the system's model of them. This is the other job nobody ships, and it might be the most important. A confident, wrong belief about you is worse than no belief at all. If the assistant decided you hate meetings before noon and you can't fix it, every helpful action built on that belief is now an annoyance. People will only trust a system with memory if they can see what it thinks it knows and push back.

I run a small version of this loop on my own setup, and the lesson from living with it is blunt. Capture and recall were a weekend. Consolidation and calibration are the whole game. They're also exactly the two jobs the splashy demos skip, because they don't show up in a thirty-second clip. They show up three weeks in, when the thing either feels like it knows you or feels like it's guessing.

Not all memory is the same

The hierarchy framing earns its keep here, because "memory" hides three very different things.

Working memory is the live conversation. Fast, cheap, gone when the session ends. Episodic memory is what happened in past sessions, the log you can page back through. Semantic memory is the durable layer: the small set of things that are reliably true about a person. Not "on Tuesday they asked about X" but "they think in analogies, they hate hedged answers, they care more about direction than speed."

That bottom layer is where knowing-you actually lives, and it's the smallest by volume and the most expensive to build. It's the output of consolidation working over months, not a transcript. An assistant that invests in the top of the stack feels like it has a good search bar over your history. An assistant that invests in the bottom feels like it gets you.

Knowing is more than remembering

Here's the part that the word "memory" obscures. Knowing someone isn't mostly facts. It's a model of their preferences and judgment, good enough to predict what they'd want without asking.

This is old territory, actually. Recommender systems have spent two decades building models of people from their behavior, and they learned two things the hard way that the assistant builders are about to relearn. First, the model of you is only as good as the signal you feed it, and the richest signal is the stuff you do, not the stuff you declare. Second, a personalization model with no correction loop drifts into a caricature. The filter bubble was a calibration failure: a system confidently wrong about you, with no good way for you to say so.

An assistant that knows you is a recommender system pointed at your whole working life instead of your next video. The same rules apply. Learn from what I do, not just what I say. And give me a steering wheel.

Proactive, not just responsive

The reason any of this is worth the effort is what it unlocks at the end. An assistant that genuinely knows you can act before you ask. Surface the thing you'd have looked for. Draft the reply the way you'd have written it. Flag the conflict you'd have caught.

But proactivity is where weak memory does the most damage, and the failure mode is already visible. One team building with agents described drowning under seventy notifications a day. Proactive suggestions from a system that doesn't really know you aren't help. They're a new inbox. The thing that makes proactivity welcome instead of exhausting is precisely the quality of the underlying model: it has to be right often enough that you stop double-checking. That bar is high, and you only clear it by doing the unglamorous loop well.

So the order matters. Memory before proactivity. Calibration before either. An assistant that acts on a shaky model of you is worse than one that waits to be asked.

I find this oddly reassuring. The hard part of building an assistant that knows you isn't a frontier-model problem we have to wait on. The model can already talk like it knows you. Making that real is mostly architecture: a capture step that's selective, a consolidation step that digests, a recall step that's relevant, and a calibration step that hands you the wheel. None of it is glamorous. All of it compounds. And the version that wins probably won't be the one with the smartest model. It'll be the one that did the boring loop, over a long time, for one person at a time.