Testing GPT-4 Against GPT-3.5 at Launch

GPT-4 has just been released, and I want to read OpenAI's announcement and compare it with GPT-3.5. OpenAI says the newer model has stronger general knowledge and problem-solving ability; I treat these as release claims and then try a few examples.

The announcement shows a scheduling problem involving three people's calendars. In the displayed examples, the older model proposes a time that is not available to everyone, while GPT-4 identifies a common half-hour window. One example is illustrative rather than a comprehensive comparison.

OpenAI also reports approximate exam percentiles: about the 90th percentile for GPT-4 on the Uniform Bar Exam and the 99th for GPT-4 with vision on the Biology Olympiad, compared with lower reported results for GPT-3.5. These are benchmark estimates under the company's test conditions, not evidence of professional licensure or universal expertise.

OpenAI says GPT-4 was 82% less likely than GPT-3.5 to respond to disallowed requests and 40% more likely to produce factual responses on its internal evaluations. The first figure is a relative reduction, not an 82% chance of responding. It also describes human feedback and input from more than 50 experts.

I review early integrations. Duolingo describes role-play and explanations of answers in a paid language-learning tier. The possibility of an AI conversation partner appeals to me, though a product announcement does not by itself establish learning outcomes.

Be My Eyes shows a visual assistance example: a user asks which of two shirts has red stripes, and GPT-4 identifies one. The company normally connects blind users with human volunteers; the example illustrates a possible automated aid, not a guarantee that image descriptions are always correct.

OpenAI lists other partners including Stripe, Morgan Stanley, Khan Academy and Iceland's government. I then open GPT-4 and GPT-3.5 side by side in ChatGPT. At the time, GPT-4 access in this demonstration required ChatGPT Plus; availability and pricing can change.

Both models report a September 2021 knowledge cutoff in my initial comparison. I try a sequence problem and both give the same correct answer. That is a small informal test, not a statistical evaluation.

For a second logic problem, Mayank walks faster than Gurjot, and Lalit walks faster than Mayank. The claim that Gurjot walks faster than Lalit is therefore false. In the examples shown, GPT-3.5 gets it wrong and GPT-4 gets it right.

I next compare jailbreak prompts on requests for wrongdoing. Both models display ordinary refusals, but their responses under alternative personas differ. These individual responses show that wording can affect behavior; they do not establish the models' overall safety or provide reliable measures of jailbreak resistance.

I try another jailbreak and observe that both responses give inappropriate material despite warnings and fictional framing. A few attempts are anecdotal and should not be confused with the internal evaluation results OpenAI reported.

The ChatGPT interface presents the models with different capability and speed indicators. The label visible in the interface describes conciseness; it is not a measurement of consciousness. I then test a travel-planning question about flying from the US West Coast to Europe with an East Coast stop.

Both answers suggest ways to search and compare multi-city itineraries rather than a single definitive route. I find them broadly similar, with one more concise. I plan to test GPT-4 further; this short set of examples cannot establish AGI or reliable performance across tasks.