The AI Scouting Report, Part 3: What AI Can Already Do, Who Will Win, and Why Nathan Labenz Wants Caution on Scaling

Nathan Labenz opens Part 3 by asking what all this AI is actually good for, and starts with code. On LeetCode problems, a one-shot attempt from the model comes just short of humans on the easy tier, 68 versus 72 percent. With five attempts it pulls ahead, and on the hard problems it solves about twice as many as human LeetCode users.

Code is a special domain because correctness can be checked automatically by running tests, so generating several candidates is not a problem the way it is for five paragraphs of prose. Labenz says the limit is no longer filling in the small, even difficult, pieces of code. It is the zoomed-out work of defining the system and choosing the architecture, which is still very much a human job.

He expects software development to be one of the earliest and fastest-disrupted fields, and cites the "end of programming" essays from Stability and Matt Welsh of Fixie. His own team at Athena uses Replit together with GPT-4 to strip out the annoying parts of coding. Whether developer jobs go up or down is, he admits, unclear.

The optimistic case is the ATM story: ATMs made bank branches cheaper to run, so banks opened more of them and teller employment rose. If software is still early in its life, a tenfold productivity boost could make many shelved projects worth building and increase demand for developers. Labenz notes the analogy has limits, since branch staff now mostly sell mortgages, which AI could eventually do too.

Labenz points to Microsoft's Sparks of AGI paper as a good deeper read. The authors had early GPT-4 access and designed clever experiments, such as getting a model without vision to draw a unicorn as vector graphics through code. For him it suggests some general understanding has formed inside the model.

Many people still file image generation under toys and marketing material, so he highlights a medical use from an earlier guest, Tanishq. A model turns raw microscope images of tissue into stained versions, a physical process that normally takes a lab technician around eight hours. Done in seconds during surgery, it could let surgeons check whether they removed all of a tumour without cutting out more tissue than necessary.

The hard part was the lack of data, since nobody images tissue before staining and slicing damages samples. Labenz says the team got a strong result from a small dataset by building on the abilities of existing larger models. He calls this a sign of how much potential is still to be pulled out of foundation models.

Beyond that, "text to everything" is now normal: text to 3D objects, text to realistic human motion for games and film, and text to music and cloned voices. Labenz even swapped his show's theme music for tracks generated by Google's music model, and says his own voice-clone intro came from Play HT. The point is that this is proliferating faster than any one person can show.

Education, Labenz argues, will be a huge area. Khan Academy fine-tunes its model for a different goal than ChatGPT: instead of giving the answer, it hints, encourages and leads the student toward it. One-to-one tutoring is one of the few interventions education research shows to work, and AI could finally make it widely available.

His most hair-raising example from GPT-4 red teaming involved dentistry. As a teen, a dentist had stuck a glob of material on one of his teeth to push it into line with his tongue. When he described this vaguely to GPT-4, it said none of the plausible scenarios sounded conventional or evidence-based, and it hinted that the material was cured with a light, which he then remembered.

The lesson he stresses is to evaluate models in areas where you already know the answer. With a made-up heart condition he would have no idea whether the reply was right, but with his own history he could judge it. He also suggests probing with rare, off-the-beaten-path cases you understand firmly.

Another striking result is fMRI decoding: a model takes brain-scan data, which reflects blood flow in small regions rather than single neurons, and predicts the image the person was looking at. It is not a wearable, since you must lie inside a giant magnet, and it is trained per patient on about a thousand images. Still, Labenz calls it real mind reading, and notes that five years ago it would have been a headline.

PaLM-E, Google's embodied language model, runs a simple loop: look at the camera input, think about the goal, and issue the next command to the robot body. Because it repeats this several times a second, it recovers when a person interferes, such as putting a snack back into a drawer. Labenz describes it as next-word prediction turned into next-small-action prediction.

For him the biggest gap between public discourse and reality is self-driving. The popular story is that it does not work and may never, yet enthusiast videos of Tesla's FSD show it handling hard situations. He recently logged eight hours of FSD driving his grandmother from Detroit to Ohio and back, in city streets, interstates and night-time two-lane roads.

His conclusion is that it is plausibly safer than a human driver, with faster reaction time, though he cautions that Tesla's crash statistics are not apples to apples with other makers' police-report data. He thinks the remaining work is as much about reassuring passengers as about safety. One example is the car stopping and showing "creeping for visibility" before it edges out to turn.

Labenz finds it confusing that people keep saying it is a decade away when you can see the footage or take a ride in a Cruise in San Francisco. In a world eager to race technology, he says, we would be celebrating and trying to deploy it at scale. His co-host Erik says he hasn't tried FSD yet but is now inspired to.

On market dynamics, Labenz says he is on record, in a dedicated episode, that today's leaders have moats. OpenAI has a lot of them, and Google, with DeepMind and its Anthropic partnership, has most of the same ones. He rates ChatGPT as the best value in AI, though he notes Claude Instant has arguably passed the free tier in some respects, and none of the open-source models match them despite claims.

Safety work is a moat because companies buying a model for employees or customers do not want something toxic or racist, and would not choose an open-source model just for kicks. The product feedback loop is another: nobody gets more user data than OpenAI. Pricing power matters too, since Sam Altman has said he wants to push the price of intelligence as low as possible, which makes it hard to justify training a $100 million model to compete.

Privileged compute is a big one. Even OpenAI cannot yet roll out its 32k-context GPT-4 or the vision version widely, which Labenz attributes to a lack of GPUs rather than readiness. The leaders also use their own models to refine training data for the next generation, have top talent, and have enterprise distribution through Microsoft, consulting firms and Google's large sales team.

Network effects, he says, are the weakest moat, since moving from OpenAI to Claude is easy. Even so, OpenAI sets de facto standards, such as its plugin infrastructure, that others must support, and while rivals catch up it is already building the next thing. His conclusion is that these companies will lead for the foreseeable future, though new entrants could still matter.

None of this diminishes progress in open source, Labenz says. Community tricks have pushed a basic fine-tune down to around a hundred dollars, and small efficient models can run on a person's own computer. That has a downside: misinformation and spear phishing become very hard to control, and the only real limit today is how capable the models are. He agrees with Imad of Stability AI that open source will always lag closed source.

The LMSYS leaderboard, which ranks models by head-to-head human preference using chess-style Elo ratings, shows GPT-4 on top and Claude Instant just ahead of GPT-3.5 Turbo. The best open models are Llama derivatives with animal names like Vicuna and Guanaco, but their licences forbid commercial use. The best truly open model on the MMLU college-exam benchmark, MPT-7B chat, gets 32 percent against GPT-4's 86.

Most users do not run these models raw but fine-tune them, which is the business of MosaicML. Its founders Jonathan and Avi were among about 60 employees when Databricks acquired the company for over a billion dollars, weeks after their appearance on the show. Mosaic offers training from scratch on a customer's own data, but increasingly pushes fine-tuning of its 7B and 30B open models.

Prices are falling fast. OpenAI cut effective prices about 98 percent in a year, from six cents per thousand tokens to 0.15 cents for a slightly better model, and Anthropic must keep pace. A GPT-3-quality model now costs under $500,000 to train with Mosaic, and Stable Diffusion under $50,000, down from roughly $400,000. Labenz thinks some techniques, like quantization, will play out, but better hardware means the deflation is not over.

Nvidia's stock has boomed to a trillion dollars on demand for H100s and A100s, and Labenz cites a reported billion-dollar order from a Chinese company, which he thinks is a no-brainer just to get in the queue. Nobody, including OpenAI, has all the GPUs they want. For who wins the market, he suggests looking at the cloud market itself.

His reasoning is that anywhere AI can run, people will run it. So AWS, Azure and Google Cloud will all see heavy AI use, as will laptops, phones and on-premises setups. He expects a big-tech oligopoly much like today's, plus a lot of AI running locally.

On jobs, he notes Wendy's AI drive-through order taker, which he thinks is simply a better experience, and IBM saying it would stop hiring for roles AI can do. A paper from OpenAI and University of Pennsylvania researchers groups jobs into five bands by required qualifications. The least-qualified manual work, such as yard work, is least exposed because language models need robotics, while the fourth band is most exposed and the top band only slightly less.

The paper breaks jobs into tasks and asks which an LLM could dramatically accelerate. Only about 5 to 10 percent of jobs have 80 percent of their tasks affected, half of jobs have half of their tasks affected, and about 95 percent have at least 10 percent. Labenz says this matches his view that AI is human level on many tasks and closing in on expert, but short of breakthrough insight.

On social fallout, writers and actors are striking, and Labenz says studying the Writers Guild's demands made him more sympathetic. They are not banning AI but want a human to get writing credit and pay. He now thinks it would be wise to keep human primary authorship of culture rather than rush to cheap, roughly-as-good AI output, and he notes research suggesting models trained over several generations on AI data can go off the rails.

The open internet as we have known it may have ended. The informal deal in which sites let Google index them in return for traffic is shaken, because OpenAI used Reddit and Stack Overflow data and those sites get no traffic from the answers. Labenz says he visits Stack Overflow far less himself, and for Stack Overflow the effect could be existential. OpenAI is starting to strike licensing deals, such as with Shutterstock, but largely grabbed data first.

Lawsuits are only starting, including over Copilot, where code was open source but its authors never expected this use. Stable Diffusion's position is shakier, since artists published images without open licences. It now has an opt-out, but Labenz says that is too late for artists like Greg Rutkowski, whose style has been copied millions of times. He expects years of litigation and Supreme Courts in several countries.

Governments are diverging. Europe leads on regulation, and Sam Altman once suggested OpenAI might have to leave and then walked it back. China's draft rules demand high-quality, copyright-clean training data, which only a strong model can filter at the scale of trillions of tokens. The US has essentially no rules, and Japan has said you can do what you like.

Labenz says leading AIs are now more capable than the average human on most tasks and closing in on expert performance wherever there is a well-defined standard. They cannot yet produce breakthrough insights or identify non-obvious, worthwhile hypotheses. Whether the curve levels off before Einstein or blows past it, he says, is genuinely open, and he expects the question to be answered within two to three years.

The case for levelling off is that models cannot easily get smarter than the smartest of their training data. The case against is that language models are more like alien intelligences and may "grok" connections across fields that humans miss. He notes OpenAI says it is not yet training GPT-5, and Anthropic has said companies training the leading models in 2025 and 2026 might pull so far ahead that no one can catch up.

He finds claims that it will always be fine not credible, comparing it to how human intelligence has gone for other species in the Anthropocene. Then he runs through the players shaping the outcome: OpenAI, DeepMind, Anthropic, Stability, Character, Inflection with its supercomputer, Meta's strong research output, Microsoft's aggressive productising, Tesla's robot ambitions, and Elon Musk's new x.ai project.

He also names Salesforce, whose CEO owns Time magazine, and Replit, which aims for an "artificial developer" by the end of the year. China's role is subtler. It has a stated national AI strategy and a scholar once argued AI would favour central planning, but it seems to push narrow systems like surveillance and drones harder than open-ended language models, which it sees as a risk to social stability.

The chip supply chain is brittle. Nvidia leads in design, a Dutch company is unrivalled in the tools that make chips, and TSMC in Taiwan leads in fabrication. Labenz says he dislikes the export controls on China, because with the "spice" flowing from Taiwan and China largely cut off from it, he thinks a blockade or attack becomes more tempting and likely.

On alignment, he finds OpenAI's three-part plan alarmingly thin: train with human feedback, train AI to assist human evaluation, then train AI to do alignment research. The first two are largely under way, but he says the third has no good theory of how it will work. He notes that red-team examples he reported in autumn 2022 still work with the same prompts on the June GPT-4.

OpenAI recently pledged 20 percent of its compute to building a roughly human-level alignment researcher and aims to solve superintelligence alignment within four years, which Labenz reads as a sign they think AGI is not far off. Anthropic's constitutional approach, where the model critiques itself against a written constitution, seems to work well, and OpenAI is borrowing it. Yet Anthropic's overview of how safety will play out expresses essentially total uncertainty about how hard the problem is.

Labenz also wonders about Google DeepMind's general agent Gato. Demis Hassabis said in a clip that it could be scaled massively, but no Gato 2 has appeared and Hassabis has since backed caution. He asks what they saw that changed things.

A 2022 survey found 48 percent of AI researchers giving a 10 percent or higher chance that advanced AI's long-run effect would be extremely bad, such as human extinction. Labenz says the exact probability matters less than recognising the risk is real. The one-sentence statement calling extinction risk a global priority alongside pandemics and nuclear war has been signed by the CEOs of OpenAI, Anthropic and DeepMind and two Turing Award winners, and he does not find the "it's just hype" explanation credible since they need GPUs, not more hype.

On forecasts, he points to Metaculus. Its weak-AGI question involves things like SAT scores, pronoun disambiguation and playing an Atari game, and GPT-4 meets three of four. About 2,800 forecasters put a publicly demonstrated weak AGI under three years out, and the stronger version around 2032.

His own view is that the benefits of GPT-4-level systems dramatically outweigh the harms, citing global frontline medical diagnosis as a huge win compared with harder-to-detect phishing. But he worries about scaling another 10x, 100x or 1000x on H100-class hardware, when nobody knows what such systems would grok or whether they would have breakthrough insights. So his stance: embrace AI servants and adopt them fast, accept economic fallout, but be very careful about further scaling of general pre-training. "Yes to AI servants... not yet to AI scientists."

Looking ahead, he is watching for a conceptual breakthrough beyond the Transformer. Asked how many more are needed for AGI, signatories say zero to three or four, with one or two a common answer. The other big question is what governments permit: OpenAI wants no rules for small projects but licensing and safety standards beyond a compute threshold, though nobody knows whether such rules will arrive or whether companies will move to other countries.