Sleep-Time Compute: Preparing AI Context Before a Question

What if an AI system could prepare useful inferences before you ask a question? A 2025 paper from researchers associated with Letta explores that idea as sleep-time compute. I previously covered MemGPT, a project from the same team concerned with giving agents more sophisticated memory.

Reasoning models use computation at inference time to work through problems, which can improve results on some tasks. The cost is that users may wait while the model generates reasoning tokens, and serving those tokens consumes computing resources. The amount of improvement, latency and expense depends on the model and task.

The paper asks why all of this work must happen after the query arrives. In a conventional setup, a model receives both the question and the relevant context at test time. If users repeatedly ask about the same documents, codebase or conversation, some analysis of that context might be reusable.

Code repositories, document question answering and conversational assistants are examples of potentially persistent context. Instead of processing only in response to each request, the proposed method prompts a model to make inferences about the context ahead of time and puts that representation into a later prompt.

The paper illustrates this with a juggler and 800 balls. One quarter are tennis balls, half of those are indigo, and one tenth of the indigo tennis balls are marked. Preprocessing can derive 200 tennis balls, 100 indigo tennis balls and 10 marked indigo tennis balls before either question is asked.

That work is not free: it shifts processing earlier, when it may be scheduled without the same response deadline, and can be reused for several questions. Berman describes latency-sensitive inference as potentially more expensive. The paper models a tenfold price difference for its cost analysis; actual prices and savings depend on deployment and usage.

The authors call this offline inference between interactions sleep-time compute. It does not require a model to continuously think around the clock. The technique prepares a natural-language representation that may help with later queries; its usefulness depends on what those queries turn out to be.

For example, a coding assistant might inspect a repository in advance, identify architecture and consider possible debugging approaches. The generated context can then be supplied when a user asks a question. This is a proposed application, and its usefulness needs evaluation in the actual workflow.

The paper reports that, in some settings on its adapted math benchmarks, sleep-time compute achieves comparable accuracy with roughly five times fewer test-time tokens. That comparison concerns tokens used when answering, not a fivefold reduction in all computation including the preprocessing. Query-time-only reasoning can outperform it at higher budgets.

The researchers also vary the preprocessing budget. They report improvements of up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME at similar test-time budgets. More preprocessing is not guaranteed to improve every configuration: in one experiment, five parallel generations generally beat ten.

The cost of preprocessing can be spread across several questions about the same context. In the paper’s linear cost model, test-time tokens are weighted ten times more expensive than sleep-time tokens. Under that assumption and ten queries per context, the reported average cost per query can fall by up to 2.5 times compared with the single-query baseline.

To create the stateful datasets, the researchers split existing math problems into context and question. Stateful GSM-Symbolic uses P1 and P2 variants; Stateful AIME combines questions from the 2024 and 2025 exams. The multi-query dataset adds synthetic questions about shared contexts, so results should be interpreted in that experimental setting.

The experiments use GPT-4o and GPT-4o-mini on GSM-Symbolic, and o1, o3-mini, Claude 3.7 Sonnet Extended Thinking and DeepSeek-R1 on AIME. For the non-reasoning models, the researchers vary test-time output by prompting for different levels of detail and checking. For reasoning models they vary the available reasoning budget.

The graphs compare accuracy against average test-time tokens per question. At lower budgets, preprocessing often improves the tradeoff, while at higher budgets the standard test-time baseline can regain an advantage. The paper reports limited gains for o1 on the AIME comparison, so the effect is not identical across models.

Another comparison uses parallel sampling: ask for several answers and choose among them. The paper compares against pass@k, which assumes access to an ideal verifier that can identify a correct answer among samples. On its evaluated tasks and test-time token budgets, the authors report that sleep-time compute outperformed that baseline.

To scale the sleep-time phase, the researchers generate several alternative context representations for non-reasoning models, or vary the reasoning effort used in preprocessing for o1 and o3-mini. This changes the accuracy and test-time tradeoff, but also increases the upfront work that must be accounted for.

Reusing a prepared context across multiple questions is the central amortization argument. The tenfold price weighting in the paper is a simplified assumption about latency-optimized inference, not a finding that every query-time GPU token always costs ten times more. Fewer related questions or different prices can change the result.

Question predictability matters. If the context describes the juggler’s balls, a question about the balls is more likely to benefit from prepared arithmetic than an unrelated question about the ocean. On GSM-Symbolic, the authors group questions by a model-based predictability score and find a wider accuracy gap for more predictable questions, though the harder P2 pattern is less smooth.

The paper concludes that sleep-time compute may be less useful when queries are difficult to predict or unrelated to the context. A future system might decide how much compute to spend before and after a query. Real workflows also involve changing context and multiple interactions, beyond the study’s simplified two-phase setup.