HippoRAG Matches Iterative Retrieval at One-Tenth the Online Cost

Nathan Labenz opens The Cognitive Revolution with a result that is easy to remember. HippoRAG, a retrieval method inspired by the human hippocampus, roughly ties the strongest iterative retrieval of the moment on three multi-hop question-answering benchmarks—a little better on some, a little worse on others—while spending about one-tenth the time and money at query time. The trade is a more expensive offline index, a bill Labenz and his guest will unpack later.

The idea is unusual for a RAG paper. Instead of sending an LLM on repeated document hunts, the system pre-processes the corpus into an entity knowledge graph and, when a question arrives, does an associative walk on that graph. Lead author Bernal Jiménez Gutiérrez is a PhD student at Ohio State working with Yu Su; Labenz spends the next hour and twenty minutes walking from neuroanatomy to how you would actually ship this.

Gutiérrez’s path is cross-disciplinary on purpose. He left Berkeley in 2015 with an applied-math degree, joined a healthcare startup doing NLP to match cancer patients with clinical trials, and stayed in biomedical NLP for years—first with ontologies and knowledge bases such as UMLS, more recently with how to actually use large language models in that domain. He describes the work as a two-way street: biology on one side, language models on the other, ideas moving both ways.

The OSU NLP lab, led by Yu Su, has shipped things like Mind2Web, a benchmark for web agents, and Su is openly interested in biologically inspired methods. HippoRAG came out of the two of them talking through neurobiological memory. The engineering starting point was more prosaic. Gutiérrez had been thinking about domain-specific models and concluded that vertical data is too scarce to pretrain from scratch; a strong general model plus RAG is probably the best bet. Digging in, he found RAG has hard limits. At the same time he and his advisor were talking about the continual learning humans do that current LLMs basically cannot. Early in 2024 those two threads clicked: for knowledge-seeking questions, an LLM chain of reasoning is not enough. You have to consolidate the corpus first.

Labenz has a name for RAG’s job: the missing middle memory. Models have superhuman parametric knowledge that freezes at training time, and a context window that either contains a fact or does not; everything in between is retrieval. He calls himself an adoption accelerationist: rather than wait for the next scale-up, take the systems we have, dial them in, and get utility now. Medicine is his first candidate domain; he has talked often with the Med-PaLM team, now Med-Gemini. Gutiérrez’s biomedical background lands squarely on that line.

One example runs through the whole conversation. Find a Stanford professor who works on the neuroscience of Alzheimer’s. The corpus has a thousand Stanford faculty profiles and a thousand Alzheimer’s researchers, and no single document states both facts. Naive RAG embeds the query and hopes for a co-located passage. If you are unlucky, you start from one side, pull a person, fetch more, check, and grind. Gutiérrez’s estimate: you may have to look at multiple thousands of documents. A human who simply knows that this professor is at Stanford and does Alzheimer’s research names them immediately.

Iterative retrieval is the current way around that: retrieve, reason, retrieve again, as many hops as the question needs. It is path following—walk the chain, even if the chain is long. The theoretically valuable split in the paper is a second class of problem, path finding. Path finding is set intersection. You need the one node that satisfies two conditions at once, and you get there by associative memory, not by temporarily flipping through documents. LLM reasoning chains are good at the first class and weak at the second, because you cannot retrofit associative memory at query time.

Labenz reads two GAIA questions that sound terrifying: one that triangulates axis labels across two arXiv papers, another that walks from the star of *Finding Nemo* to a USGS zip code for an invasive fish. Gutiérrez’s judgment is that both are still path following. Difficulty comes from hop count, not from a change in the nature of the task. True path-finding questions—pure intersection—do not even have a decent benchmark yet. GAIA is related, but it has no standard corpus, and pre-processing the whole internet is still a cost problem, not only a research one.

The theory underneath is hippocampal memory indexing. In the mid-twentieth century, patient H.M. had his hippocampus removed and could no longer form new long-term episodic or semantic memories, which is how the field learned the region was essential. The theory says the hippocampus does not store the memories. It stores pointers into neocortex, plus associations among those pointers. New knowledge iteratively builds that associative index: a node for the professor, a link to Stanford, a link to Alzheimer’s research, and a graph you can traverse.

Three components, two capabilities. Neocortex holds the detailed memories. Parahippocampal regions sit in the middle as a bidirectional bridge; they are overparameterized and sparse, and they do pattern separation—tearing a dense experience into concepts such as “Stanford professor” and “Alzheimer’s research.” The hippocampus, especially the hyperconnected CA3 subregion, does pattern completion: joining concepts that co-occurred in experience even when they are not semantically similar. Labenz hears the sparse autoencoder boom in the parahippocampal story. Gutiérrez’s version is a sparsity constraint on dense, real-time cortical representations. Hippocampus literature already has place cells, grandmother neurons, even a Halle Berry neuron, which suggests the stored codes end up cleaner and more monosemantic than the mess inside an LLM.

HippoRAG maps the parts almost one to one. The language model plays neocortex and pulls concepts out of language. Retrieval encoders play the parahippocampal bridge. A knowledge graph plays the hippocampus. Personalized PageRank is the associative-memory step. Offline, each document goes through the LLM for named-entity recognition, which only biases the next step: open information extraction with no rigid schema, producing entity–relation–entity triples. Encoders then add synonymy edges so “Alzheimer’s disease” and its abbreviation sit on the same connected graph. Online, the question is parsed into entities, mapped onto nodes, and used as seeds for random walks. Node probabilities become the document ranking.

The experiments use three benchmarks, about a thousand questions and ten thousand documents each. Head to head, HippoRAG competes with iterative retrieval; some sets a bit better, some a bit worse, with a ten-fold drop in online retrieval cost and time. Gutiérrez’s line is that any production system that needs fast user access cares about that gap immediately. Combine HippoRAG with naive iterative RAG and the average lift is 7 to 8 points. He calls it proto-self-correction: the model sees the graph ranking and then reasons with associative retrieval, which is stronger than either method alone. There is also a qualitative case that existing methods cannot touch—a medication that works by hitting a particular enzyme, the kind of intersection a doctor needs if the patient is already on something that interferes. HippoRAG did well on a few such tests; unlocking them fully, he says, still needs a smarter walk.

Indexing ran on GPT-3.5 for a boring reason: the job is extraction, not reasoning. Spot salient entities, and when two appear in a sentence, name the relation. Even Llama 3 at 8 billion parameters does a fairly good job; he would not expect Claude or GPT-4 to be much better out of the box, aside from formatting. Repeat the same experiments with those flagship models and you would spend around $500 on a limited slice. Claude is about twice as expensive on inputs and ten times on outputs. Indexing all of Wikipedia? Six million articles, first paragraph only, about $9,000 with GPT-3.5. A lab cannot afford that. A startup can. Labenz’s advice is to get the answer right, charge what you have to, and ride costs down. Founders, he thinks, obsess over cost while the real pain in AI products is being wrong, and hiring a human for these questions costs far more than a dollar. The path to a small model often runs through a large one; Gemini 1.5 Flash is reportedly distilled from a bigger sibling.

Three weaknesses. Graph traversal is naive. Personalized PageRank only sees connectivity: closer to the seeds, or linked to more of them, means higher probability. Turn the walk up to the maximum and you have original Google PageRank—popular nodes win—which is a disaster on a knowledge graph. Gutiérrez admits some of the benchmark wins come from few distractors; production scale needs a smarter walk, and that is the team’s first priority. Second is concept versus context. A MuSiQue question about who discovered that every element has a unique proton number has no named entity to extract; the model stuffs “proton” into the graph and the walk goes nowhere. Mapping entity-less queries onto graph entry points is the next attack. Third, HippoRAG only handles explicit knowledge. The corpus may record that Cinderella lives happily ever after without ever saying the ending is happy. That takes reflection.

One counterintuitive design choice: do not put document nodes in the graph. Labenz wanted raw passages on the graph so you could enter from document similarity. Gutiérrez says a document node links to every entity in the passage, probability smears, and Personalized PageRank can no longer do multi-hop. That matches the brain story. The hippocampus stores sparse, separated concepts. Full memories stay in neocortex.

Long context is the obvious rival. Labenz dumped his last 250 sent emails—about 250,000 tokens—into Gemini 1.5 Flash, asked for a character sketch, and got a remarkable two-page portrait in about 40 seconds for under 20 cents. Gutiérrez pours a little cold water. These models are good at synthesizing content that is directly relevant. Multi-hop reasoning inside a long prompt is still a challenge. He has a paper under review that uses long-context models as retrievers: they beat ordinary RAG, and you can see multi-hop traces in the attention patterns, but the ability drops as length grows. He suspects each point in that email sketch maps back to a passage; real knowledge consolidation may not have happened.

If you are willing to spend a quarter per query to be right, he would put long context to work as an entry finder, not as the multi-hop engine. Pull thousands or tens of thousands of facts off the graph, let Gemini pick the true entry points, then let it help guide the walk. Even a graph over the whole internet probably will not have more than the top 10,000 facts relevant to one question, and that needle-in-a-haystack job is what long-context models are good at. Do not ask them to replace graph multi-hop.

The end-state picture is an automatically generated semantic web. HippoRAG handles entities and associations. RAPTOR, MemWalker, and GraphRAG all add clustering, summaries, and more abstract ways into the graph—RAPTOR clusters similar passages and writes a higher-level summary; MemWalker builds tree memory over long documents. On top of that, something like the Stanford AI Town project’s periodic reflective memories: agents revisit observations and write higher-level memories, the way sleep post-processes the day. HippoRAG’s explicit-only limit is exactly what that layer would fix. FLARE, for the record, is a forward-looking retrieval loop that decides when to fetch again; useful, but a different trick from building structure in advance.

On tools, two names. ColBERT surprised him as a small model that generalizes: it mixes BM25-style old retrieval with semantic embeddings unusually well. Contextual AI’s GRIT is an instruction-tuned retriever; you can push the embedding in a direction, which helps when the corpus is narrow. Chunk small—two or three sentences. Longer paragraphs make the model drop triples and hallucinate relations across sentences. The opposite school is Long RAG, which refuses to chunk and concatenates semantically similar documents. The trade is local attention versus global consolidation. For HippoRAG as it stands, small chunks plus hierarchical summaries of long documents.

Labenz’s last question is which human capabilities still look unsolved. Gutiérrez points to a paper from MIT’s Nancy Kanwisher and Evelina Fedorenko: models are already strong at phonology, syntax, and local semantics, and weak at world knowledge, formal logic, and social reasoning. Continual learning is the world-knowledge hole HippoRAG is aiming at. The competency he cares about next is formal logic. Models lean System 1—fast, heuristic—and lack constrained System 2. Brains appear to solve that with modularity: a language circuit and a logic circuit doing different jobs.

Scale might grow those competencies, he says, but it is not guaranteed. His research bet is inductive bias through modularity: architecturally different parts of the system learning different things, which might fill in formal logic faster than stacking parameters. Labenz flags a related DeepMind paper, *Transformers Meet Neural Algorithmic Reasoners*, which cross-attends a standard transformer to more structured graph-like reasoners that are good at certain logical problems. Gutiérrez’s one-line close is that RAG’s next competition happens before retrieval starts. Whoever consolidates the corpus into something you can associate over gets answers that are both fast and right.