AI Model Rumors, Math Benchmarks and Med-Gemini Research in May 2024

In May 2024, I examine reports of an upcoming OpenAI model, a mysterious chatbot briefly available for testing, and two research papers on math benchmarks and medical AI. The model-release discussion is a snapshot of rumors and predictions at the time, not a current forecast.

A Politico report said some frontier AI companies had not provided the UK government with early access for safety testing after commitments at the Bletchley summit. The report also cited anonymous sources about an imminent OpenAI release. I inferred that a model was near, but an inference from anonymous reporting is not a confirmed launch schedule.

I predicted an iterative model before GPT-5 and cited Sam Altman’s comments about knowing a release date and deploying gradually. I also speculated about GPT-5 timing. These were my predictions in May 2024, not facts established by the evidence.

A mystery model called gpt2-chatbot briefly appeared on a model-comparison site. Some impressive examples circulated, but I tested only eight hand-crafted questions and found performance broadly similar to GPT-4 Turbo. That small test cannot establish the model’s identity or a robust ranking.

Other reported tests gave mixed results in translation and logic. At the time I considered several possible explanations, including a smaller model. The video description later adds that the chatbot was made by Microsoft’s Phi team; that update should be distinguished from the uncertainty expressed in the recording.

I discuss an OpenAI researcher’s argument that data strongly shapes model behavior, alongside the importance of compute and evaluation. My broader question is how much leading performance depends on data, scale and spending rather than a single secret architectural ingredient.

A clip of Mark Zuckerberg discusses GPU supply and the possibility that energy becomes a constraint before companies run out of capital. This is an industry forecast, not an established limit on every AI project.

A Scale AI paper revisits the GSM8K math benchmark by constructing 1,000 new problems of comparable difficulty. The purpose is to investigate whether models’ exposure to existing benchmark questions makes scores overstate generalization.

Some model families performed worse on the new questions, while GPT-4 and Claude performed similarly or better in the comparisons I describe. The authors also consider differences in generalization: exposure alone is not enough to explain every result.

Benchmark quality matters too. I point to errors in existing questions and to the possibility that training on exam-like material may improve benchmark scores without equally improving performance on everyday tasks. Similarity can matter even when a question is not an exact duplicate.

The paper’s fresh questions also show a small Phi model performing competitively with much larger systems in this specific test. I interpret this as evidence for the value of high-quality training data, while noting that one benchmark does not settle the full capability comparison.

I ask why advanced models that score well on graduate-level evaluations still miss some elementary arithmetic word problems. Possible explanations include flaws in the tests and limits to generalization; the video poses the question without resolving it.

The second paper introduces Google’s Med-Gemini research models. I describe their promising medical benchmark results, but benchmark performance is not the same as clinical validation, safe deployment or evidence that a model can replace a clinician.

One method uses the model’s confidence to decide when to generate a search query and incorporate retrieved information. The authors also explore using correct generated answers in a fine-tuning loop. A correct final answer does not guarantee that the reasoning behind it is sound.

Long context enables experiments on very large electronic health records. The paper also reports results on medical question-answering tests and identifies quality issues in some benchmark questions. I highlight summarization and referral letters as additional areas of research.

I compare the authors’ reported results with clinicians on particular questions and note their plans to incorporate other data. Such comparisons depend on the task, dataset and experimental setup; they cannot be generalized into a claim that the system is better than doctors in routine care.

The paper also explores multimodal assistance, including interpreting images and answering questions about surgical video. I emphasize that the authors had not deployed this for live surgery because of safety and ethical concerns. Demonstrations are not evidence of safe real-time use.

The authors suggest potential improvements, including restricting retrieval to authoritative medical sources. I close by asking how future, clinically validated systems might assist clinicians, while recognizing that this is a hypothetical question about future evidence and deployment decisions.