An 80GB Wikipedia Cache Versus a 100GB Model That Holds the Internet

Sonya Huang and Shaun Maguire open Sequoia’s Training Data with Engram co-founders Dan Biderman and Jessy Lin. The company’s site says they do not see the world through pretraining or post-training: their models are always training. Jessy translates that as a claim about the bottleneck. Models are already incredibly smart. What makes them more useful now is not more raw intelligence, but understanding new and evolving context—a new task, a job, a company’s way of working—and baking that into the weights as deeply as pretraining baked in facts and skills.

That is why they treat memory and continual learning as two sides of the same coin. Sonya asks whether a separate database shoved into the context window is therefore not true memory. Jessy does not want to throw the current toolkit away. Context engineering, long prompts, multi-hour agent traces, and tool use will still play a part. The underused move, she argues, is to take the same training pipeline frontier labs use to make models good at math and code, and apply it to every domain a company actually has.

Dan’s analogy is sticky notes versus the trace you wake up with. External notes are valuable and should never be discarded. When you return to work the next day, though, your brain also holds a residue—new information about how things should be and where to look. Current systems are almost all externalized memory. Two problems follow. Individuals will soon generate tens of millions of tokens a day; storing, searching, and rereading that stream will be expensive. It will also confuse the models unless retrieval itself gets a major breakthrough.

The product, at a high level, sits inside long-lived workspaces. Engram is working with partners such as Notion, Microsoft, and Harvey, where teams already write documents and, increasingly, talk to agents and give them feedback. The goal is a model that does not merely reread the files at test time, but understands the company the way an employee of several years does: the initiatives, the way hiring is run, the local how-we-do-things. They train per-team models in those workspaces and let them improve on what people actually care about.

Technically that means writing into weights, mostly via adapter fine-tuning. LoRAs, prefixes, sparse architectures—decades of academic tools are in the kit. The harder step is data: turning raw documents and interactions into training signal, using supervised fine-tuning, RL, on-policy distillation, and the rest of the field’s stack. They need white-box weight access. Closed-source partnerships are possible; open-source models are easiest. Any transformer, Dan says, they can do their thing to.

The trade-off Sonya names is the one they want: burn compute up front so every inference pass ships less context. Dan goes further than a modest saving. Skip the research, the reread, and the monstrous system prompt, and you can see two orders of magnitude less token consumption—not 50%, but as much as 100×. Knowledge about people, teams, and priorities often lives in no single document. A model that has trained on the residue can answer in 100 tokens what a frontier model spends 100,000 tokens hunting down. There is a quality window too: a recurring three-to-six-month gap where the current generation is not yet good at some bespoke task, and lightweight continual learning can fill it before the next general model arrives.

Why start at the workspace rather than the individual? Either is fine, Dan says. Teams are more disciplined about collecting context and have years of it, so they are the easier deposit to mine. Every person’s computer and phone is a later target. For now, the big veins of information sit in knowledge-work collaboration.

Sonya raises the old fight: is it a feature or a bug that models memorize facts such as the capital of France? One school wants abstract concepts of countries and capitals, not a roster of facts in the weights. Jessy splits the answer. You need to remember things in order to compose them into more complex concepts. What is missing is a sense of what is worth remembering. Academic benchmarks that ask for the length of a particular bridge in an African country are asking models to devote capacity humans would not. Human memory is lossy on purpose: intelligence includes compressing what matters and dropping what does not.

She does not buy a clean split between fact learning and skill learning. People have tried stripping facts out of a model to leave a “pure core.” The result is unnatural; it does not know basic things. If you must recall a basic fact to take the next step in a thought, you cannot get very far. Internalizing is what lets the model compose abstractions. Dan adds the computer-science version. Databases and algorithms used to be separate curricula: store-and-query versus sample-efficient manipulation. Deep learning mushed them together so thoroughly that interpretability teams now try to pry them apart. As companies adopt AI, the split is creeping back—carefully engineered private context, a generic model that is a stranger to it. Engram wants a cadence that mixes facts, stories, and details back in. The cost is real: capitals rarely change; many other facts change all the time, and imprinting those into weights is hard.

Dreams come up as a loose inspiration, not a blueprint. Jessy sees a missing phase: everything happens at test time, thinking on the fly over the user’s context, making mistakes, never digesting the episode back into the model so the next attempt starts further along. Dan admits that “build an AI like our dreams” sounds nuts. What interests him is the experiment: in dreams we talk to ourselves and probe what we can and cannot do in the world and in social situations. They want models to retreat from the live interaction and try their affordances—including the tail cases we replay at night—offline.

Dan’s canonical motivation is a thought experiment. If OpenAI had to win a math olympiad in a week, would it catalog textbooks and hire annotators to mark chapters, or collect data, synthesize a training set, launch a job, and see where the model lands in five or six days? Anyone who has trained a model knows the second path is the superior way to integrate ideas and capabilities. That magic already happens in high-stakes math, coding, and cyber. They want more people to have it.

Shaun asks the obvious question: why isn’t the end product just owned by the foundation-model labs? Jessy’s answer is worldview. Frontier labs want one model, bigger and bigger, more intelligent across domains. Engram imagines everybody having their own. Much of what people want learned is private—it will never appear in a post-training set—or conflicting: this company wants the task done one way, that company another. The old machine-learning toolkit of clean supervision, ground-truth rewards, and a tidy environment does not fit ambiguous real-world work. Dan puts it as a P0 problem. Labs are spending talent and money on pretraining, scale, data, RL, and inference-time compute on the road to a generic AGI that can code, do math, and automate the economy. They are thinking about memory; Demis Hassabis said at a Sequoia event about a month earlier that the topic needs new breakthroughs. It is still more of a product effort. Engram is doing only this.

Three things, Jessy says, have to look different: research breakthroughs; infrastructure that trains small models for everybody rather than one giant run; and a tighter loop between research and product. Today researchers train, throw the model over the fence, and product teams prompt on top. If models are always training, user input *is* the training signal. That is why they are deploying as fast as they can, not only publishing. Dan’s other unlock is learning from many humans and organizations at scale without sending someone to sit shoulder to shoulder with each of them.

Why this problem? For Jessy it is the remainder now that models are so smart—learning at the edges—and a way to reverse-engineer the accident of pretraining. The internet was a gift of diverse examples; nobody fully understands why generalization emerged. Continual learning means taking whatever made Paris and Python stick and aiming it at a company’s private data. Dan came from neuroscience, originally interested in consciousness and, citing Amos Tversky, in natural stupidity more than artificial intelligence. He does not think machines should copy brains. He does think human memory evolved under capacity constraints into fuzzy representations that abstract, connect, and inform the next day. Current systems have almost none of that beyond generic pretraining. You store a codebase losslessly on disk; you do not need AI for that. You need something else for the residue that changes how you work tomorrow.

Shaun asks the bitter-lesson question: will memory just emerge inside the model if you scale data and compute, or does it need a stand-alone architecture? Dan splits deployment from architecture. In the wild, AI is already more than GPUs—sandboxes, other computers, models trying things. On architecture, he has worked on state-space models and other long-context designs that cut the quadratic memory cost of attention. Some of the best Chinese models now use related layers. In Engram’s hands, those methods have always traded accuracy for computer-memory savings. There is no free lunch. If you are truly bitter-lesson-pilled, you ask how to burn *more* compute, on context you have not seen. They are not betting AGI stalls. They are betting that rereading Shaun’s files will not be enough to understand Shaun. (The joke: you would need a 100-trillion-parameter model for that.)

What do customers actually want learned? Jessy says the app layer is full of people trying to make a model work for a use case—brand style on a website, a workflow, a way of writing. The methods do not really distinguish facts from skills. Dan is blunter: most app layers are a frontier model in a loop with search. What they want from Engram is firm knowledge encoded so the model does not have to research, and a targeted sense of whom to triage to. With fairly lightweight training, those things can become instinctual. In that sense, he calls it a RAG killer—then immediately adds that you can always still retrieve. The interest is interfacing with large data planes without long tool-calling loops.

What belongs in weights versus RAG? Dan says nobody has the answer; it is the core question of biological memory. You do not need last year’s hotel room number in neural tissue. You probably do want the password to your home imprinted somewhere for the next few years. They try to use as few heuristics as possible. Humans watch TikTok, absorb garbage, and do not go completely off the rails; models should be robust in the same way. Jessy’s short-term pain point is inference cost: agents running for days, rereading the same files, and many people in one company running the same queries on the same documents. An employee does not type “what was I working on yesterday?” into a search box. They just know.

Caching helps some of that. Association does not. If you only RAG, you cannot notice a teammate’s research direction and recall a related thing they did not ask about. Those links live in weights. Retrieval’s deeper limit, Dan adds, is addressing: knowing what to look for. Keyword search is popular because it is easy to RL and light on embedding infra, but “knowing what to search” is an intuition. The company itself started with a dive into KV caches. A single Wikipedia article—Taylor Swift, say—can occupy about 80GB of HBM as KV cache. A 70B Llama’s entire weights are about 100GB and, with some distortion, remember the internet. Tens of kilobytes of text become 80GB of “brain state.” Gradient descent is the existence proof that you can pack a lot into few numbers. Take those 80GB, spend compute offline, compress the cache by something like 1,000×, and you change load time, cost, and fidelity together.

What would a ChatGPT moment for memory look like? Jessy does not claim to know the product shape. The proof of concept everyone is waiting for is an intern you can teach over time who actually gets better. Context engineering, however sophisticated, is not getting there. You will feel it as: whoa, the model is different from yesterday. Dan notes that ChatGPT itself was not the product direction people had on the roadmap. His test is harsher. If you quit tomorrow and spent 24/7 prompt-and-skill engineering OpenAI and Anthropic models to serve you, you would barely move the needle; you would be better off waiting for the next version. The future they want is one where time spent on a domain actually translates into performance on the tasks you care about. That only works, they think, if you scale compute and train on private data *without destroying the model*—which is hard.

Sonya asks whether we will carry a memory wallet from app to app, the way people talk about a token wallet. Jessy is conservative. She already dislikes ChatGPT memory crossing personal and work contexts: sheets recommended because you trained a model on a GPU last week. Some of that is the memory being flawed; some of it is that tools should keep memories separate, under user control. Dan’s holy grail is different. You go to work, burn tokens, create value. IP stays with the company. Some of the skills and methods you invented travel with you to the next job, sanitized. We already do a biological version of that, bounded by NDAs and ethics. Doing it digitally would, he thinks, force people to push AI deeper—and reward them for it.

On a detour, Shaun offers a crackpot information-theory story for why language overtook vision: biology gives optics a massive bitrate advantage; computers flatten everything to electronics, nerfing vision and promoting language, after which LLMs happen to be the architecture that fits text. Dan does not endorse the theory. Humans are vision-biased, and occipital real estate probably still beats language cortex—he would have to check ChatGPT; in his personal life he is “a big RAG believer.” Brain bias toward a modality does not mean machines will be more efficient at it. For knowledge work, text is fine. Jessy expects language and vision to combine into a more unified system over the longer term.

If everything goes right in five to ten years, Jessy wants everyone to have their own model, different from their neighbor’s and from the frontier model, serving different purposes—an intelligence that helps in ways that make sense to that person or team. Dan’s variant borrows a neuroscience rhyme: memory and navigation share circuits; landmarks in space and pieces of episodic memory live together. He wants Engram to be the LLM interface to everyone’s data plane, in the spirit of Databricks or Oracle, except the memories are neural, the models are personalized, and there are hundreds of millions of them. It is not representing the file system as it is. It is representing a brain state of that file system.