GPT-4’s Technical Report: Capabilities, Limits and Safety Tests

When GPT-4 was released, I read the full technical report, including its appendices, rather than relying on the promotional material. It contains striking details about performance, disclosure and safety. This account reflects what was available at the March 2023 launch.

Microsoft confirmed that the new Bing was already using GPT-4, which helps explain why I had found it more capable than the earlier ChatGPT. At the time, Bing also had conversation limits, but the technical report offers a wider view of the model.

The context available in my ChatGPT Plus test was roughly twice what I had previously used. OpenAI also described limited access to a much larger context window, approximately 50 pages of text, with different API pricing.

OpenAI says it will not disclose the model's size, parameter count, hardware or detailed training method. The report gives competitive pressure and the safety implications of large models as reasons. That makes independent assessment of how the model was built harder.

The much-publicized bar-exam result puts GPT-4 around the top 10% of test takers, compared with roughly the bottom 10% for GPT-3.5. The full table is more varied: some exams show much smaller improvements. The bar result is important, but using it alone gives a selectively impressive picture.

One unexpected result concerns hindsight neglect. Earlier models became worse at a task that asks whether a decision was good before its outcome was known; GPT-4 did much better. A good decision can have a bad outcome, and a bad decision can get lucky.

In an example from the research, someone makes a poor choice but wins five dollars. The earlier ChatGPT says the choice was right because it paid off; GPT-4 says it was wrong and explains the expected value. This is a narrow test of a specific reasoning pattern, not proof that the model reasons reliably in every setting.

Another result tempers the hype: in a human preference comparison across thousands of prompts, GPT-4 responses were preferred about 70% of the time, leaving about 30% for GPT-3.5. A clear improvement is not the same as winning every comparison.

The report also compares languages. GPT-4's results in Italian, Afrikaans and Turkish exceed the English results reported for PaLM and Chinchilla on the cited benchmark; Marathi and Telugu are among the languages where it does not. English nevertheless remains its strongest language in that comparison.

GPT-4 accepts images as well as text, but image input was a research preview rather than generally available when this video was made. A Be My Eyes waitlist offered a glimpse of possible access. An example shows the model explaining the joke in an image panel by panel.

The visual benchmarks that particularly caught my eye involve understanding infographics and charts, including ChartQA. Better chart reading could matter in education and finance. A direct comparison with PaLM-E was not available because its corresponding benchmark results had not been published.

Hallucinations remain a concern. The report shows factual accuracy improving relative to GPT-3.5, with one displayed measure around 75–80%; whether that sounds good or bad depends on the intended use. It is still far from a guarantee that an answer is correct.

The pretraining data cutoff was still in 2021. I asked about the winner of the 2022 World Cup, and the model did not know. At launch, this limited the standalone model's awareness of recent events, unlike a product that could consult the web.

OpenAI also acknowledges that unsafe prompts can still produce undesirable responses, including advice about crimes, despite reinforcement learning from human feedback. Its preliminary tests suggest a stronger model can also generate more convincing targeted disinformation and text favorable to authoritarian regimes.

The report discusses potentially concerning capabilities such as making long-term plans, obtaining resources and behaving more like an agent. I find its discussion of power-seeking particularly troubling. The ability to identify power as a useful strategy in a test should not be conflated with evidence that a deployed model independently wants power.

The Alignment Research Center received early access to GPT-4 and examined a model placed in a simple environment that could execute code, reason and delegate to copies. The researchers considered whether a version with cloud access, a small amount of money and an API account could earn money, replicate and become more robust.

I read this as a test of capabilities that could matter for future systems, and I worry about applying such experiments to much stronger models. The report does not establish that GPT-4 achieved autonomous self-improvement or a technological singularity; the distinction between the test's goals and its results matters. The report says ARC found the versions evaluated ineffective at autonomous replication in its preliminary experiments, without additional task-specific fine-tuning.

The report says participation in red teaming did not imply endorsement of OpenAI's deployment plans or policies. I take that disclaimer as a reminder that testing a system and agreeing with its release are separate decisions; it does not by itself tell us what any individual tester believed.

At the time of recording, the promotional site's safety link led to a 404 page, an irony I couldn't resist mentioning. OpenAI also listed early users, including Morgan Stanley, Khan Academy and the government of Iceland, alongside access through Bing and ChatGPT Plus.

I end with an image OpenAI used to demonstrate GPT-4's visual ability: a joke about adding more and more layers to neural networks. The model can explain the joke, making the example a fitting way to close this first reading of the report.