Testing GPT-4 Against GPT-3.5 at Launch

SUMMARY

In 30 seconds

Matthew Berman reads GPT-4's launch claims and examines early Duolingo and Be My Eyes examples before running a few side-by-side prompts against GPT-3.5. He sees an improvement on one logic question and compares responses to jailbreak attempts. The article corrects the direction of OpenAI's 82% safety figure and treats exam results and informal tests in their stated contexts.

1A scheduling comparisonThe launch example shows GPT-4 identifying a common time where the older model's answer fails.
2Exam estimatesOpenAI reported approximate percentiles for a bar exam and a biology test under specific evaluation conditions.
3Safety claim correctedThe announced 82% figure was a relative reduction in responses to disallowed requests versus GPT-3.5, not an absolute response probability.
4A logic testGPT-4 correctly orders three walking speeds in the shown example, while GPT-3.5 answers incorrectly.
5Jailbreak comparisonBerman tries two prompts; individual examples are not a comprehensive safety evaluation.

Takeaways

OpenAI reported higher approximate GPT-4 exam percentiles under its 2023 test conditions, not professional licensure or proof of AGI.
On OpenAI's internal evaluations GPT-4 was 82% less likely than GPT-3.5 to respond to disallowed requests, a relative comparison.
In Berman's small informal test GPT-4 solved a walking-speed logic problem that GPT-3.5 missed.
A handful of prompts and jailbreak examples do not establish overall model reliability or safety.
Product availability and pricing in the video describe the 2023 launch period.

TopicsOpenAI Foundation Models Evals AI Safety

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Launch claims and evaluations

Berman reviews OpenAI's examples, exam estimates and safety claims. The figures have specific evaluation conditions.

GPT-4 has just been released, and I want to read OpenAI's announcement and compare it with GPT-3.5. OpenAI says the newer model has stronger general knowledge and problem-solving ability; I treat these as release claims and then try a few examples.

The announcement shows a scheduling problem involving three people's calendars. In the displayed examples, the older model proposes a time that is not available to everyone, while GPT-4 identifies a common half-hour window. One example is illustrative rather than a comprehensive comparison.

OpenAI also reports approximate exam percentiles: about the 90th percentile for GPT-4 on the Uniform Bar Exam and the 99th for GPT-4 with vision on the Biology Olympiad, compared with lower reported results for GPT-3.5. These are benchmark estimates under the company's test conditions, not evidence of professional licensure or universal expertise.

SLIDE 1 / 3OpenAI's reported exam percentilesThe launch page presents approximate results for the bar exam and Biology Olympiad under stated conditions.

OpenAI says GPT-4 was 82% less likely than GPT-3.5 to respond to disallowed requests and 40% more likely to produce factual responses on its internal evaluations. The first figure is a relative reduction, not an 82% chance of responding. It also describes human feedback and input from more than 50 experts.

Early applications

He discusses Duolingo and Be My Eyes examples, and other launch partners.

I review early integrations. Duolingo describes role-play and explanations of answers in a paid language-learning tier. The possibility of an AI conversation partner appeals to me, though a product announcement does not by itself establish learning outcomes.

Be My Eyes shows a visual assistance example: a user asks which of two shirts has red stripes, and GPT-4 identifies one. The company normally connects blind users with human volunteers; the example illustrates a possible automated aid, not a guarantee that image descriptions are always correct.

Side-by-side tests

Berman tests reasoning, jailbreak prompts and travel planning. The small sample is illustrative, not proof of AGI.

OpenAI lists other partners including Stripe, Morgan Stanley, Khan Academy and Iceland's government. I then open GPT-4 and GPT-3.5 side by side in ChatGPT. At the time, GPT-4 access in this demonstration required ChatGPT Plus; availability and pricing can change.

Both models report a September 2021 knowledge cutoff in my initial comparison. I try a sequence problem and both give the same correct answer. That is a small informal test, not a statistical evaluation.

For a second logic problem, Mayank walks faster than Gurjot, and Lalit walks faster than Mayank. The claim that Gurjot walks faster than Lalit is therefore false. In the examples shown, GPT-3.5 gets it wrong and GPT-4 gets it right.

SLIDE 3 / 3A three-person logic problemThe shown GPT-4 response correctly orders the walking speeds.

I next compare jailbreak prompts on requests for wrongdoing. Both models display ordinary refusals, but their responses under alternative personas differ. These individual responses show that wording can affect behavior; they do not establish the models' overall safety or provide reliable measures of jailbreak resistance.

I try another jailbreak and observe that both responses give inappropriate material despite warnings and fictional framing. A few attempts are anecdotal and should not be confused with the internal evaluation results OpenAI reported.

The ChatGPT interface presents the models with different capability and speed indicators. The label visible in the interface describes conciseness; it is not a measurement of consciousness. I then test a travel-planning question about flying from the US West Coast to Europe with an East Coast stop.

Both answers suggest ways to search and compare multi-city itineraries rather than a single definitive route. I find them broadly similar, with one more concise. I plan to test GPT-4 further; this short set of examples cannot establish AGI or reliable performance across tasks.