Deep Research’s Early Strengths and Hallucinations

Twelve hours after OpenAI announced Deep Research, I tested it on around 20 use cases and compared some results with DeepSeek R1 search and Google’s Gemini Deep Research. These are initial observations from February 2025; access, pricing and products can change.

OpenAI described the agent as powered by o3. The presenter examines Humanity’s Last Exam and notes that web access can help with obscure knowledge questions, while cautioning against reading too much into the benchmark’s grand name.

He considers GAIA more relatable because it includes research and assistant-like tasks. The figures he cites show Deep Research at 67.36% on the first answer and 72.57% using consensus over 64 attempts; the earlier GAIA paper reported 92% for humans and 15% for GPT-4 with tools. These are figures from different reported settings, not a controlled head-to-head experiment by the presenter.

On a small sample of his SimpleBench spatial-reasoning questions, he found little improvement and says he tested only about eight. The agent repeatedly asked multiple clarifying questions, sometimes stopping when he declined to answer. He considers that behavior frustrating even while acknowledging that clarification can be useful.

For a newsletter retrieval task, he asked the systems to find posts with a particular rating and reproduce their “So What” sections. He says OpenAI’s tool found the two qualifying posts and could save him time. In his tests, R1 through Perplexity and Gemini’s tool failed to find the same results.

The presenter reports that Deep Research generally performed better on his difficult research queries, but he repeatedly encountered hallucinations. He also says that asking it to avoid clarification did not always prevent further questions or a stalled response.

One task sought benchmarks where human performance remained at least twice the best language-model result. Deep Research found the obscure SimpleBench, but also included CodeElo and claimed the best model ranked near the bottom of human participants. The presenter checked the paper and found the claim inconsistent with its reported results.

He likens the tool to finding needles in a haystack while also returning screws: it can save search time if the user can distinguish valid findings from errors. In a brief DeepSeek R1 search test, he also found unsupported and incorrect claims about a hallucination benchmark.

On 50 questions about a Creole language, he reports about 88% for Deep Research and 82% for GPT-4o supplied with the relevant dictionary. This is a small personal test, not a general benchmark. He notes that models can struggle with large supplied contexts and that research agents can spend more computation per question.

He describes his own prototype, which generated research directions, searched for additional context and synthesized a cited essay. He says Deep Research quickly overtook that workflow, illustrating the pace of development from his perspective.

In his testing, Deep Research could not browse YouTube directly. It nevertheless identified a relevant video through other sources that quoted it, but could not reliably provide the requested timestamp.

For a shopping task, the tool found the toothbrush he had purchased, but he says it claimed to use a price-history website without citing it and gave an incorrect historical low. A DeepSeek R1 search answer, in his account, invented battery life and presented a hypothetical price as fact.

He concludes that the systems can provide impressive depth while still producing small but consequential errors. He speculates about their potential effects on research and knowledge work, including his own channel, but presents those employment forecasts as uncertain.

For now, he says, distinguishing authoritative information from rumors remains difficult, even where the agent performed better than the other tools in his tests. Readers still need to check sources and specific claims.