Testing ChatGPT o1 Pro Live: Puzzles, Code and the Cost of More Reasoning

Berman welcomes viewers and adjusts the audio before introducing OpenAI’s first day of announcements in December 2024. He has subscribed to ChatGPT Pro to test o1 pro mode with audience questions. OpenAI announced the plan at $200 per month, with access described as unlimited subject to abuse guardrails; the features and limits are historical.

He begins with a greeting to see whether the model can avoid lengthy deliberation for a simple request. The interface reports about seven seconds of thinking. Berman notices a progress bar and expandable summaries, but is dissatisfied when details are unavailable; these visible summaries do not reveal the model’s entire internal reasoning.

The first coding request is Snake in Python. Berman runs the generated code locally but cannot show that window, and reports that the game opens and immediately closes. Because the error and setup are not displayed, this is his report of one failed run rather than a reproducible benchmark.

A viewer suggests a timetable for 20 classes, 12 rooms and 13 teachers, with three days and 14 hours per class. Berman finds the wording difficult and waits while the progress bar moves slowly. He initially suspects a freeze, then acknowledges it may still be processing; the stream does not establish a final timetable or a cause for the delay.

For the married-person puzzle, Jack looks at Anne and Anne looks at George; Jack is married and George is not. The model answers yes: if Anne is married she looks at George, and if she is unmarried Jack looks at her. Berman accepts the two-case explanation.

A viewer asks for the 100,000th prime. The model displays 1,299,709, which is correct; the automatic captions misrender the number. Berman compares it with the viewer’s answer and treats the result as a success.

The host enters an abstract viewer prompt about an “infinite observer,” recursive feedback and neural networks. The model proposes recurrent or recursive processing, but Berman says he would need more time to understand and assess it. This philosophical language is not evidence of a model possessing an observer or consciousness.

Asked to prove P equals NP, the model says the question remains open and explains the distinction between efficiently checking and efficiently finding solutions. It does not produce a proof. The Clay Mathematics Institute continues to list P versus NP among the unsolved Millennium Prize Problems.

For “Ulam’s conjecture,” the model chooses the graph reconstruction conjecture and says it is unresolved. The on-screen summary title misleadingly says “disproven,” but the answer itself does not supply a disproof. The prompt is ambiguous because several questions are associated with Ulam; the chosen conjecture concerns reconstructing a graph from its vertex-deleted subgraphs.

A viewer proposes an ARC-style visual puzzle. Berman sends a screenshot containing example grids and a test grid, asking for block locations and colors. The model gives coordinates, and he requests a rendered HTML representation to make the output easier to compare.

He opens CodePen and works through how to render the generated result, first discovering that CSS alone is insufficient for his setup. Once the grid appears, he compares it with the original and concludes it does not complete the shapes correctly. He suggests that a clearer textual description or separate screenshots might help, but does not rerun a standardized evaluation.

Another viewer supplies a creative prompt about recursive universes. The model returns a short fictional description of civilizations recurring through loops. Berman likes the prompt and reads the answer without treating its speculative imagery as science.

Berman asks for the Mona Lisa in HTML and CSS. The first response attempts to place an external image in a frame rather than draw it, and the referenced image does not load in his test. He clarifies that the model must create a rendition without an external image.

The revised code produces crude pixel art. He sends a screenshot back and asks for a correction; after a reported two minutes and 47 seconds, he considers the result little improved. The outcome reflects that particular prompt, rendering environment and follow-up, not a comprehensive visual-coding evaluation.

A viewer questions whether a $200 plan will encourage expensive tiers across the industry. Berman says most of his own tasks do not require pro mode and wonders whether additional test-time compute might someday justify more expensive plans. His estimates of how many users need Pro and his hypothetical pricing are opinions, not measured results.

He asks for a Doom clone using only HTML. The model says the request is infeasible in that form; a follow-up involving JavaScript leads to a partial browser output that Berman cannot confidently evaluate because he is uncertain about his CodePen setup. He does not demonstrate a complete Doom clone.

For the marble puzzle, a glass containing a marble is turned upside down on a table and then lifted into a microwave. The model says the marble stays on the table. Berman accepts that result under the ordinary physical assumptions of the puzzle.

The killers puzzle asks how many killers remain after an entrant kills one of three people in a room. The model counts two surviving original killers and the entrant, for three. Berman accepts that interpretation but wishes it explicitly distinguished living killers from the dead person still in the room; the entrant’s prior status also matters.

He repeats the killers question with ordinary o1 and sees a similar answer more quickly. A viewer asks about Sora access, and Berman speculates about what a paid subscription might include. Those are launch-day predictions rather than a statement of later product availability.

A viewer supplies a Sudoku grid. After approximately four minutes, o1 pro says no valid solution exists. Berman asks the audience whether that is correct and does not independently resolve it during the stream.

The grid visible on screen does have a valid completion: 435871926 / 276349851 / 981256473 / 612598734 / 749132685 / 358764219 / 193427568 / 867915342 / 524683197. This completion preserves the displayed clues and satisfies the row, column and 3-by-3 constraints, so the model’s no-solution conclusion is incorrect for that grid.

Berman asks how many r’s are in “strawberry” and tries o1 pro, ordinary o1 and o1-mini. All three answer three in these runs; he likes the more explicit explanation from the mini model. This simple prompt does not determine overall relative performance.

He considers a viewer’s request to generate a browser run-and-jump game using HTML, CSS and JavaScript. The generated output runs as a very basic game in CodePen, with a score and game-over behavior. He considers it functional in the limited sense of the prompt, while acknowledging how simple it is.

A travel-time puzzle says Sam leaves home at 10:00, spends two minutes at a shop and returns home at 10:04. The model answers 10:03, assuming one minute each way, and Berman agrees. Equal travel times are not specified: without that extra assumption, the departure time is not uniquely determined.

Asked what may come in the remaining OpenAI announcement days, Berman emphasizes that he has no inside information. He speculates about agents, browser control, image generation and other products, while doubting an imminent Sora release. These were predictions made during the 2024 stream, not verified announcements.

He explains his preferred vision for agents: systems able to use authorized personal information and proactively perform online tasks. He contrasts browser-integrated controls with screenshot-and-coordinate approaches, which he says can fail. This is an aspiration and a description of limitations, not a demonstration of a working autonomous agent.

On whether a $200 plan is sustainable when cheaper models may catch up, Berman suggests the company is charging for scarce inference compute and higher-intensity usage. He expects competition and changing prices, but presents his business explanation as speculation rather than knowledge of OpenAI’s internal economics.

A viewer requests “hello world” in ASCII art. While o1 pro continues working, Berman asks GPT-4o the same thing and receives a much faster answer. The o1 pro response reports two minutes and 17 seconds; he considers the outputs broadly similar. One informal pair of runs does not establish the models’ typical latency or quality.

Near the end, Berman tells a casual user that the $200 plan is probably unnecessary for their needs and describes his own reliance on search-oriented tools and other subscriptions. He says Perplexity is not a sponsor and that he uses it voluntarily; no separate sponsor read appears in the stream.

He declines a Wordle request because it would require too much back-and-forth, then tries an audience prompt about subtle consciousness and virtual meditation. The model generates language about awareness and stillness, which Berman finds unclear. Such generated text is not evidence that the model has subjective experience.

Berman thanks viewers for their questions, discusses possible future streams and says he may try additional coding tools. He closes with invitations to follow the channel and newsletter.