The AI Scouting Report: 90 Minutes From GPT-4's Exam Scores to Whether Models Understand

On July 6, 2023, The Cognitive Revolution changed format. Host Nathan Labenz brought on no guest and talked for about 90 minutes by himself. The episode is part one of his AI Scouting Report for June 2023, the first installment of a series he plans to repeat. He is presenting as both a founder and an AI investor: he spent years as CEO of the AI video company Waymark, has followed papers, launches, and evals with unusual intensity, and also does AI advisory work for companies.

The success bar he sets sounds modest on purpose. He will consider the episode a huge success if listeners leave with "the equivalent of a high-school AP course level understanding of machine learning fundamentals." That is not a researcher’s syllabus. It is meant for founders, investors, and people changing careers who need to follow the field, ask better questions, and make judgments without living in arXiv.

He has curated graphs, illustrations, and figures that made an impression on him, and he wants them to land the same way for the audience. Doing that in 90 minutes is ambitious, he says, which is why this report is a YouTube series rather than an audio-only podcast. Short audio episodes will still go out as announcements, but the full experience needs the visuals. Part two, coming later, is promised to cover AI as it interfaces with the world.

There is a second number hiding in the same episode, and it is not from a benchmark. A remote executive assistant in the Philippines spent two years fine-tuning and validating more than 100 custom models for a U.S. company. Labenz treats that story as the practical twin of the lecture: one line is what the science actually is, the other is what the work looks like when someone has to run it every day.

The first half of 2023 was a firehose. GPT-4 landed on March 14, then model launches, funding rounds, layoffs, open-source drops, and congressional hearings arrived almost daily. Labenz’s view is that this cadence does not help people understand AI and often hurts. The scouting report is his attempt to zoom out from day-to-day news and hype and focus on what actually matters.

The metaphor is deliberate. A baseball scout does not judge a player by one highlight reel. The job is long-run data, underlying tools, growth curve, and remaining holes. Labenz wants the same posture toward AI: skip the single story, and try to assess real strength. The raw material is two years of obsessive reading, compressed into the concepts and trends he thinks you need in order to engage research, products, and public argument with more sophistication.

He lays out a four-part frame at the start. First, a headline summary of what general-purpose models can already do. Second, how modern systems are designed and trained. Third, the big capability unlocks. Fourth, what we know about how they work and what they truly understand. That list is also a self-test. If you can answer those four questions, you are already ahead of people who only follow headlines.

The AP-level target is a methodological choice, not just a humility tic. News goes stale in a day. The basics of pretraining, scaling, and alignment stay useful for years. If study time is limited, he wants it spent on the layer that does not churn. That is the core of the report: put attention on the parts of the field that will still be true when this week’s launch is forgotten.

Labenz anchors the capability summary in public exams rather than vibes. When GPT-4 launched in March 2023, it placed in the top 10 percent of all takers on the Uniform Bar Exam and scored a 5 on AP Biology. Those tests were written for humans, and the model was not being drilled on a leaked item bank so much as showing general skill. The previous generation, ChatGPT, had launched on November 30, 2022, and crossed 100 million monthly users in two months, then the fastest consumer-app ramp on record. Putting the scoreboard first is how he gives later arguments a shared set of facts.

Modern training, in his telling, starts with pretraining: feed the model a huge amount of internet text and ask it only to predict the next word. Phone keyboards do a shallow version of the same job; these models do it at internet scale and in much more depth. To get the next token right, a system has to pick up grammar, facts, logic, and even patterns of reasoning. The architecture underneath almost every major model is the Transformer, from Google’s 2017 paper *Attention Is All You Need*.

Then comes scale. OpenAI’s 2020 scaling-laws work argued that performance rises in a predictable way as parameters, data, and compute grow: double the investment, and capability steps up a rung. DeepMind’s 2022 Chinchilla paper added a crucial correction, that earlier large models had been underfed on data, and that tokens have to keep up with parameters for a given compute budget. Hovering over that engineering mood is Rich Sutton’s famous essay *The Bitter Lesson*: general methods that ride compute and data tend, over the long run, to beat clever human-designed shortcuts. GPT-3, at 175 billion parameters in 2020, was the milestone that made the new interaction mode obvious. Without changing weights, a few examples in the prompt were enough for the model to copy a new task, a trick now called in-context learning and the basis of almost every large-model product.

A pretrained model only continues text; it does not, by default, do what you asked. OpenAI’s 2022 InstructGPT paper described the fix that became famous as RLHF, reinforcement learning from human feedback: people rank several answers, and the model is trained toward the higher-ranked direction, like a room of teachers grading drafts. ChatGPT was that recipe turned into a product. On top of scale, Labenz flags the unlocks that did not look like smooth curves. Some skills are simply absent in small models and appear past a threshold, which is why people talk about emergence. Mixing code into the training data seems to help reasoning. Chain-of-thought prompting, asking the model to write the steps before the answer, was shown in a 2022 Google paper to lift math and logic accuracy by a lot.

The last fundamental Labenz takes up is the loudest argument of mid-2023: do these systems understand anything, or are they just fluent parrots? He does not pretend the field has a verdict. He puts evidence on the table. The skeptical pole is the 2021 "stochastic parrots" paper associated with Emily Bender, Timnit Gebru, and colleagues, which treats large language models as statistical stitchers of words, not understanders. The other pole is a set of experiments that are hard to square with pure memorization.

One of the strongest is Othello-GPT, from a 2023 Harvard-affiliated line of work. A model was trained only on sequences of Othello moves and never shown a board. When researchers probed its internal state, they found it had learned a board representation anyway: where the pieces were, whose turn it was, with structure to match. Follow-up work suggested that this world model emerged naturally during training. A system whose only job was to predict the next move had built a map of the game. If that internal board is real, the story that the model is merely parroting move lists gets much weaker.

Anthropic’s interpretability work supplies a different kind of object. Researchers isolated circuits such as induction heads: the model looks back for a fragment that appeared earlier and copies what came next, a mechanism that helps explain in-context learning. The research program is mechanistic interpretability, the attempt to find identifiable circuits inside the network rather than treating the whole thing as mush. "There is recognizable structure in there" stops being a metaphor once you can point at a circuit.

Labenz lays out both sides and does not bang a gavel. Putting Othello-GPT and induction heads on a fundamentals list still reveals a lean: models learn more structure than the parrot story allows, and understanding looks like a matter of degree. For people who actually use the systems, that is not academic. If the inside has structure, prompting is not mysticism. Clear context and examples are retrieval cues for machinery that already knows how to look back and copy a pattern. Writing them down is closer to giving the model a handle than to casting a spell.

After the lecture frame, Labenz switches to a much more concrete workplace. Athena is an executive-assistant company that pairs EAs in the Philippines with startup executives and other clients who have more opportunity than time, at $3,000 a month. That price, he says, is enough to hire management-level talent in the Philippines, on the client’s time zone, embedded in their life and team. He is not a distant host reading an ad. Two and a half years earlier, while still CEO of Waymark, he was an Athena client himself.

He matched with an assistant named Charisse. They started with the usual EA work: email, calendar, social media, ad hoc research. She was good, and projects he would not otherwise have reached actually got done. The interesting turn came when they moved into generative AI. The same qualities that made her a strong EA, he found, made her a natural AI operations lead: organization, attention to detail, a nose for outlier data, comfort with software, tolerance for repetition, the ability to balance speed and accuracy, plus the curiosity to ask why and the habit of being proactive.

Over two years, working on U.S. hours, Charisse built critical datasets, wrote large numbers of prompts, fine-tuned and validated more than 100 custom models, kept dozens of elaborate spreadsheets, and filled in wherever else she was needed. Fine-tuning here means continuing to train a base model on the company’s own data until you have a version that fits the job; validation means checking that those versions actually beat the generic model. None of that required a PhD. It required a stable method and a lot of repetition. Labenz says it is no exaggeration that she has become a core part of the Waymark team.

That experience is why he started advising Athena on AI, about six months before this episode. The work has two aims: give assistants a foundation, and build a task-automation practice inside the company. They have shipped an AI 101 class on prompting and savvy model shopping, added AI prompts into existing delegation playbooks, and started collecting automation templates. A small tell of how seriously they took the report: after his wife Amy, the first audience was 200 Athena EAs. One of Athena’s values is "always long-term," which is an awkward slogan in a market that changes daily. Their bet is to invest in the people, let them ride the wave, and share experiments with clients along the way.

Labenz keeps the success criterion low: AP-level literacy, not mastery. The sting is that in 2023 the gap between people who did that homework and people who did not was already widening by the month. The report is free on YouTube, and part two on how AI meets the real world is already queued. For a general listener, the useful residue is not a feeling of having kept up. It is a few actions that can start the next day.

The four-part frame doubles as a self-exam. Can you say, in plain language, what general models can currently do, how they are trained, how new skills unlock, and how far understanding claims should go? If those answers are crisp, you are already past most headline-only observers. If they are not, the right move is to walk the same four blocks again rather than to open another news tab. Ninety minutes of that beats a week of launch coverage.

The Charisse story is also a hiring filter. Look for the person nearby who notices details, asks why, and does not mind repeating a task until the log is clean. Those older professional habits are exactly what AI-era execution jobs reward. Behind more than 100 models was not a single clever prompt. It was a habit of recording what worked, checking it, and iterating. A prompt that actually helps should be saved, labeled with the job it is for, and treated like an asset. Six months later that file is often one of the most valuable documents a team has.

Savvy model shopping is the last practical cut. Copy, analysis, and code do not all want the same system, and brand loyalty is a weak substitute for trying the right tool on the right job. A blunt self-test is whether you can explain pretraining, RLHF, and emergence to a colleague in three sentences each. If you cannot, the briefing has not stuck yet, and the four-part list is still the homework.