AI Scientist, Grok-2 and Agent Q: Matthew Berman’s August 2024 AI News Roundup

In this August 2024 roundup, I want to cover a remarkably busy week of AI releases. There was also plenty of hype: an anonymous account drove speculation about OpenAI's rumored Strawberry reasoning model, but many of its claims proved wrong. The memes were entertaining; they were not a reliable release announcement.

xAI announced early previews of Grok-2 and Grok-2 mini, identifying Grok-2 as the anonymous model previously tested as sus-column-r on the LMSYS arena. At the time I checked, only the mini option appeared available to me, despite xAI saying both were in beta on X. I found the model strong at reasoning, and its image generation uses Black Forest Labs' FLUX.1.

The image examples circulating online showed politicians, fictional characters, and other recognizable figures in improbable scenes. I was struck by how few restrictions appeared to apply to these examples. Those screenshots demonstrate what users were generating, not a comprehensive test of the service's safeguards or copyright rules.

I received early access to OpenAI's SearchGPT prototype and asked it about an upcoming UFC fight. It returned an answer alongside sources, with feedback controls and media. I prefer getting a sourced answer to sorting through ads and links, and have switched my default search to it. My prediction that SearchGPT and Perplexity would eventually take over search from Google is an opinion about the future, not an established market outcome.

MultiOn's Agent Q research combines guided search, self-critique and reinforcement learning to help web agents plan and recover from errors. I had not tried it, and noted that the project was closed source and appeared aimed at consumer use. Its reported OpenTable booking experiments show the Llama 3 agent improving from 18.6% zero-shot success to 81.7% after one day of autonomous data collection, and 95.4% with online search. These are results for that particular booking setup, not a general success rate across the web.

The techniques include Monte Carlo tree search and direct preference optimization. Some observers have connected similar ideas to rumors about OpenAI's Strawberry or Q*, but the identity behind the anonymous account and those connections were unverified. I am interested in trying Agent Q on actual tasks rather than treating the rumor as evidence.

Cosine reported that its Genie coding agent scored 30% on SWE-bench and 50% on SWE-bench Lite. I saw this as a substantial leap over the competing results in the chart, including one for Devin, though I was unsure why the chart listed different Devin figures. These are historical benchmark comparisons, and different testing and reporting methods matter.

Cosine described its training mix as 21% JavaScript, 21% Python and 14% TypeScript, with Ruby at 3%. Its examples also span feature development, bug fixes, refactoring, tests and documentation. Genie was designed to work as an agent; at the time, interested users could join a waitlist.

Sakana AI's AI Scientist was the story I found most exciting. Its proposed system generates machine-learning research ideas, writes and runs experimental code, visualizes results, drafts a paper and uses an automated reviewer for feedback. It builds on research templates and an existing experimental setup; producing a paper is not the same as demonstrating that its discoveries are correct or important.

I suggested that automated research in machine learning might eventually lead to recursive improvement and an intelligence explosion. That is my interpretation and prediction, not something the reported experiments establish. The researchers describe limitations and flawed outputs, and their automated review scores are not a substitute for independent human validation.

The researchers estimated roughly $15 per generated paper in model inference costs, which makes the experiment intriguing but is not the total cost of creating and validating a research program. The system can archive ideas and use prior work for later research. I had not yet read the entire paper and wanted to examine its details before drawing stronger conclusions.

OpenAI introduced SWE-bench Verified, a human-validated subset of the benchmark intended to evaluate whether systems can solve real software issues more reliably. The underlying examples come from resolved issues in 12 open-source Python repositories. The chart compares several agents and tools, but performance on this subset does not prove that an agent can reliably handle every software task.

Google's Gemini event included a demo that struggled in public. A presenter photographed a Sabrina Carpenter concert poster and asked Gemini to check a calendar for the San Francisco date. The first request failed, and the retry failed too; after switching to a backup phone, the presenter tried once more and the demo worked.

Google had also brought Gemini Live's conversational voice experience to market. Users could talk back and forth and interrupt it. I preferred the sound of OpenAI's GPT-4o voice, but credited Google for shipping its offering. That assessment reflects what was available and announced when this video was recorded.

Anthropic introduced prompt caching for Claude, which can reuse eligible input prefixes across requests and reduce the cost and latency of repeated context. Potential uses include long conversations, coding assistants, documents, detailed instructions and agent workflows. The examples shown included estimated cost reductions of 90% for chatting with a book, 86% for many-shot prompting and 53% for a multi-turn conversation; actual savings depend on the workload and cache usage.

Caching still requires engineering judgment about what context can be reused and when. I think the savings make it worth testing for applications at scale. Next, I discussed reports that Apple was considering an iPad-like home device on a robotic arm, potentially for smart-home control, video calls and security. The suggested 2026 or 2027 launch and 360-degree movement were rumors at the time, not an Apple product announcement.

Finally, Nous Research released Hermes 3, a set of models fine-tuned from Llama 3.1 in 8B, 70B and 405B sizes. The team described improvements in role-play, agentic tasks, function calling, multi-turn conversation and long-context coherence. I had not yet tested the models myself.

The displayed comparisons with the corresponding Llama 3.1 models showed Hermes winning on some benchmarks and losing on others, rather than sweeping every measure. Nous also emphasized user steerability and fewer restrictions. I liked the possibility of similar performance with more control, while acknowledging that the chart alone does not establish how it performs on everyone's tasks.