Here’s a strange fact about how today’s AI models work. When you give one a long document and ask it questions, it is, in a quiet and very expensive way, re-reading everything it has seen so far in order to produce each new word. Not the whole text from scratch every time, but something close to it in spirit. Understanding why is the key to a trick that could make AI memory dramatically cheaper, and it turns out the trick is the same one your own brain already uses.
The notebook problem
Imagine you’re reading a long book and taking notes as you go. For every single new word you want to write, you first glance back over all the notes you’ve taken so far. On page one, that’s nothing. By page nine hundred, every new word means scanning a mountain of notes. The work per word keeps growing, and most of it is wasted, because the vast majority of those old notes have nothing to do with the sentence you’re writing right now.
That mountain of notes has a name inside an AI model: the KV cache. It’s the model’s working memory of everything it has read. And it is enormous. For a large model holding a book-length context, that memory can run to tens of gigabytes, larger than the model would like to carry, and it grows with every token. Worse, when you step away from a chat for a few minutes, most systems simply throw the whole thing in the trash. Come back later and the model has to read your entire document again from the beginning before it can say a word.
So we’re paying twice. Once to carry a giant pile of memory that’s mostly irrelevant at any given moment, and again to rebuild it from nothing every time a conversation goes cold.
What your brain does instead
You read a thousand-page novel last year. Do you remember every sentence? Of course not. You remember the shape of it: the main characters, the turns that mattered, a handful of vivid scenes. Everything else has softened into a blur. And here’s the thing, you function perfectly well like that. You can discuss the book, recommend it, argue about its ending. You compressed a thousand pages into a manageable sketch and lost almost nothing you actually needed.
That’s the whole idea. An AI’s working memory is full of “notes” that get used constantly and notes that, after a while, never get touched again. So why keep them all at full size? Keep the load-bearing ones sharp. Let the quiet ones fade.
There’s a subtlety that took the research community a while to get right, and it’s worth pausing on. There are two ways to make a note fade. You can delete it, or you can blur it. Deleting is final: if it turns out that minor character matters in the last chapter, the note is simply gone, and you’re stuck. Blurring is gentler: you fold a handful of minor notes together into one fuzzy summary note. The detail is softened, but it isn’t annihilated. If it turns out to matter, you at least have a faint trace, and you know where to look. Blurring beats deleting almost every time, for the same reason a smudged photograph is more useful than a torn one.
You will occasionally forget something. That’s the point, not the flaw.
The honest objection to all of this is: what if you blur away the one detail someone later asks about? And the honest answer is, sometimes you will. But notice that this is exactly how you operate, and it has never made human memory useless. When someone asks you a detail you’ve forgotten, you don’t panic. You go back and look it up. You re-read that one chapter.
A machine can do the same, and far more gracefully. Most questions land on the parts it kept sharp, and those it answers instantly and cheaply. The rare question that needs a blurred-away detail triggers a fallback: go re-read just that one section, or hand it off to a search tool. You pay the expensive, precise cost only at the rare moment it’s actually needed, instead of paying it on every word of every interaction forever.
This is a theme that shows up everywhere once you start looking for it. Zip files compress your documents enormously because real text is full of predictable structure; only the truly random parts resist, and those are rare. The internet runs smoothly most of the time without every server agreeing on everything every instant, because genuine conflicts are uncommon, and you handle those when they arise rather than grinding everything to a halt to prevent them. The pattern is always the same: the world is mostly smooth and predictable, the genuinely hard moments are sparse, and the smart move is to handle the easy majority cheaply and escalate only at the rare hard point.
Most of AI today does the opposite. It pays the maximum cost everywhere, on every word, just in case. That’s the waste.
Why this matters beyond saving a few dollars
There’s a second payoff that’s bigger than memory savings. If you can compress a reading session down small enough, you can keep it instead of throwing it away. Picture saving your annotated book to a shelf rather than tossing it, so that next week you pull it down and pick up exactly where you were, in seconds, instead of starting the thousand pages over.
Done naively, that’s a bad trade, because storing the full uncompressed memory costs more than just rebuilding it later. But compressed the way we’ve been describing, the math flips. A stored, shrunk-down session becomes cheaper to keep on a shelf than to rebuild from scratch, and it reloads in seconds instead of a minute of grinding. For anything that needs to resume long-running context reliably, and especially for systems where rebuilding the memory later may not even be possible, that stored, faithful memory stops being a nice-to-have and becomes the thing that makes the whole feature work.
The honest state of things
None of this is science fiction, and I won’t pretend we invented it whole. Researchers have been chipping at every individual piece, the keeping, the blurring, the storing, for a few years now, and some of it is already running in real systems. What’s genuinely underexplored is the part that matters most for the way we want to use it: what happens when you store a memory for one purpose and reopen it weeks later for a completely different one. Your notes from reading a novel for pleasure are not the notes you’d have taken reading it for a law exam. A memory frozen under one set of intentions, reawakened under another, is a real and largely unsolved problem, and it’s exactly where the blur-don’t-delete instinct earns its keep, because a blurred memory can still be squinted at from a new angle, while a deleted one is just gone.
That’s the frontier we’re poking at. The short version, if you skipped to the bottom: AI doesn’t need a perfect, photographic, infinitely expensive memory. It needs a good sketch, a cheap shelf to keep the sketch on, and the good sense to go look something up on the rare occasion the sketch isn’t enough. Cheaper, faster, and a lot more like the way you already think.
— Greg