Sam Altman on Scaling AI Models: Matthew Berman Examines OpenAI’s 2023 Direction

In April 2023, I wanted to discuss Sam Altman's comments on the race to build ever-larger AI models. At an MIT event, the OpenAI CEO suggested that progress would increasingly have to come from other approaches, rather than simply making models bigger. That was a view about the direction of research, not a demonstration that scaling had stopped working.

Transformers helped power the new wave of large language models. My reading is that adding parameters alone can bring diminishing returns, and that the next major performance improvement may require new ideas, better data or different techniques. Those are possibilities, not a detailed roadmap that OpenAI announced at the event.

I compare GPT-3.5 with GPT-4 and say the latter achieved better performance with only a small increase in parameters. That numerical comparison is not established: OpenAI's [GPT-4 technical report](https://arxiv.org/abs/2303.08774) explicitly withholds model size and architecture details. The size increase cannot be verified from the report.

I turn to a [Wired article](https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/) quoting Altman on the end of an era of giant models and the possibility of improving them in other ways. The article also describes competitors including Anthropic, AI21, Cohere and Character.AI investing heavily in their own systems.

The article interprets Altman's remarks as suggesting that GPT-4 could mark a shift away from relying on ever-larger models and datasets. It does not identify a specific replacement research strategy. It also notes diminishing returns and the physical constraints on building data centers; neither establishes that further scaling is impossible.

Cohere co-founder Nick Frosst makes a similar distinction: progress on transformers need not depend solely on adding parameters. The article points to new architectures and additional tuning based on human feedback as promising directions. I think those could make systems more useful even without assuming a particular increase in model size.

I connect that discussion to a post from OpenAI co-founder Greg Brockman. At the time, OpenAI had said it was not training GPT-5, and Brockman's post stresses work on GPT-4 and safety rather than specifying when a successor would arrive. The idea that particular architecture changes were planned is my interpretation, not a confirmed announcement.

Brockman writes that OpenAI spent more than six months testing GPT-4 and making it safer, building on earlier alignment research. He says the company expected to increase precautions, even more proactively than some users might prefer. I see this as a signal that OpenAI intended to keep investing in the model it had just released.

Another part of Brockman's post calls for powerful training runs to be reported to governments, accompanied by increasingly sophisticated predictions of capabilities and impact, and subject to practices such as dangerous-capability testing. I welcome innovation, but I also think it is important to consider possible harms and safeguards.

My expectation at the time is that OpenAI will work on deploying and improving GPT-4, safety, data and human-feedback-based tuning while investigating new techniques. I also want higher ChatGPT message limits. This is a 2023 forecast, not a permanent statement about the company's plans: OpenAI later [introduced GPT-5 in August 2025](https://openai.com/gpt-5/).