AI Already Beats the Median Lawyer. Top Firms Bought the Tools and Left Them Unused

Nathan Labenz brings on Kevin Frazier and Alan Rozenshtein, co-hosts of the *Scaling Laws* podcast, for a tour of AI and the legal system. Frazier is a senior fellow at the Abundance Institute and director of the AI Innovation and Law program at the University of Texas School of Law, where he regularly interviews practicing lawyers on campus in Austin. Rozenshtein is an associate professor at the University of Minnesota and a senior editor at *Lawfare*. Labenz splits the episode in two: how AI is changing the profession now, then more speculative designs for legislation, rights, and AI welfare.

Lawyers are more insulated than most professions because they write their own licensing rules, Labenz notes, but Rozenshtein is blunt in the host's preview: the practice of law is a cognitive activity, and frontier models are already better than the median lawyer on raw intellectual horsepower. At the same time, 70% of top firms have licensed tools like Harvey, Frazier says, while day-to-day use stays surprisingly low. Secret cyborgs quietly outperform peers. Firms whisper about fewer junior hires. Whether that becomes mass displacement or a boom in human-AI legal services is still open, because cheaper law does not automatically mean people will buy more of it.

Frazier describes the policy side as a traffic jam. Core privacy principles, the Fair Information Practice Principles or FIPs, date to the 1970s; case law is older than that. Those analog-world rules were already strained by the internet, and AI puts them "on steroids." Rozenshtein then draws the distinction that organizes the rest of the hour. Law schools that teach AI are really teaching two classes: the law of AI—how to regulate, promote, and control the technology—and AI in the law, how the technology changes legal practice. The second track, he argues, is badly underestimated.

Rozenshtein analogizes law to programming. Both are crafts of manipulating symbols. Programming has already been transformed; law is behind, but "a year behind or maybe two years behind. It's not 30 years behind." Courts still want a human in the room, so the parallel is not perfect: if every programmer uploaded into the cloud, he jokes, the job might still function, whereas law still needs people to walk into courtrooms. The delay, he says, is the guild. Lawyers are the one profession that, because they are lawyers, controls who may be a lawyer. Change will take longer. Direction will not reverse. Law remains one of the fundamental technologies of modern society.

Labenz checked public GDPVal lawyer tasks and found Claude Opus 4.5 at the top of a thin prompt set, winning one in three head-to-heads against human lawyers and winning or tying about 70%. He is surprised there has not been more guild closing of ranks, in law or in medicine. Rozenshtein agrees the models are "extremely good," still held back by hallucinations and by missing databases—an obscure SEC rule buried in the Federal Register—but those gaps look "fairly trivially solvable." He spends what he calls a horrifying amount on model subscriptions as a professional obligation. Claude is his daily driver, mostly inside Claude Code; for legal taste he still calls out to ChatGPT 5.2, especially the Pro extended-thinking mode on the web interface, because he thinks OpenAI has spent the most on legal RLHF. All three major labs, he says, already give pretty good answers.

Then the line that would offend a lot of lawyers: these models are certainly better than the median lawyer. There's no question about that, at least in raw intellectual horsepower, and he sees no reason they will not be vastly superior in a few years. Customized taste is another matter. A Supreme Court advocate with 50 arguments before the justices is hard to RLHF. The vast majority of legal work, like the vast majority of programming and medical work, is pattern matching in standardized contexts. "It's over," he says. When he and Frazier present a forthcoming article on AI in legal scholarship, he expected tomatoes; he has gotten far fewer than he feared. Anyone who spends an hour with a $20 model realizes that if the model is just fancy autocomplete, so is the professor.

Among lawyers who have actually used the tools, Rozenshtein thinks they are making the same bet he is: Jevons paradox. As legal services get cheaper, we will want more of them, and lawyers will move up the value chain. Some who cannot adapt will do badly; in 10, 15, or 20 years he expects at least as many lawyers, and probably more demand. Whether Jevons holds across economic domains is, he says, the AI-and-the-economy question. Given how little law many people can currently buy in a country that is supposed to be a sophisticated rule-of-law society, his money is on more demand, not less.

Labenz is not sure. He puts dentistry and accounting on one end of a spectrum—buy the minimum—and software on the other, where latent demand might be 10× or 100×. His own last small contract negotiation went through language models; he never hired an attorney. Frazier answers with legal deserts: parts of the country with about one lawyer per thousand residents, where a lease, a small business, a nonprofit, or a divorce has nowhere to go except a solo shop on Main Street that is not a specialist and charges too much. Even a little counsel in landlord-tenant trials sharply changes outcomes. He also wants a second track he calls legal architects, people who design regulatory systems at a higher level of abstraction, in the spirit of Gillian Hadfield's work with Fathom. Schools that keep AI out of the early curriculum, he warns, are failing students who already used it in high school.

Rozenshtein adds the contracts-class idea of a complete contingent contract: if counterparties had infinite time and zero opportunity cost, the agreement would be almost infinitely long, covering every eventuality. Default rules exist because nobody can write that document, and they misfire. Give each side a sophisticated agent negotiating at hundreds of tokens per second and you get orders of magnitude more legal demand that could actually be socially useful. Law is also competitive in a way dentistry is not. Your teeth are on your side; the other party is not, so arms-race dynamics keep demand high. Frazier pushes the same logic into legislation: laws are still written as they were centuries ago, yet models could encode triggers—if unemployment in a field hits 7%, if another country imposes tariffs—and automatically adjust policy. That field, he says, has barely been scraped.

Labenz asks whether junior legal work is already disappearing the way Anthropic talks about not hiring juniors, or the way a $200 Claude Max plan can out-ROI a new CS grad. John Grisham's underdog lawyers bleeding through document dumps look like the first thing models would eat. Frazier will not name firms, but he has started asking visitors a thought experiment: Harvard first in the class with no AI, versus an AI-fluent graduate of a mid-ranked school. He hears the second answer more often, because that person will find new tools and teach the rest of the firm. Harvey, a litigation-flavored ChatGPT, is in 70% of major top-100 firms by Harvey's own stats. Training, in the answers he gets, was an email at launch. There is no obligation to use it.

The incentive is structural. Paid by the hour, a lawyer's job inside the band a client will accept is to bill as many hours as possible. Firms know that model and are scared to buck it, so adoption stays symbolic. Frazier has not seen a huge shrinkage in entry-level jobs, but he hears whispers about fewer summer associates and fewer junior associates. He also hears Ethan Mollick's secret cyborgs: the AI-savvy lawyers who are not telling their superiors how many use cases the tools already cover. Rozenshtein is more cautious about "already happening." Even software's AI-as-downsizing story is unsettled. Law is years behind on capability and further behind on implementation, because firms can only be owned and operated by lawyers, who are not generally brilliant business managers. A human still has to appear in court and attest that the filings were checked. An AI hallucination in front of a judge is very bad.

What he is sure of is that a lot of entry-level work—discovery, needles in haystacks, the thousandth contract in a practice area—is already in today's models' range. Whether that work was necessary on the way to becoming a good lawyer is the open question, and cognitive deskilling worries him more than headcount. Programming climbed from hardware switches to assembly to garbage-collected languages to Python; each layer was denounced as cheating, and programmers did not get dumber, they changed abstraction. He does not know if law and medicine will follow. In his own life as a professor, student RAs do less of the old job. A script downloads PDFs, Gemini Flash summarizes them, Gemini Pro and the Claude API debate which articles matter, and he gets formatted Markdown. He became a professor because he hung around a wonderful one doing "nonsense crap work." If the next generation never gets that apprenticeship, that is a problem even if Jevons holds in the long run.

The sharper cut is agency. People who expect a path of assigned hoops and grinding will struggle. AI is an incredible opportunity that requires a higher level of agency—finding problems, building tools, deciding when to trust the model. He hears that theme in Tyler Cowen. Long term it is good for society; short term it feels like a betrayal to people who did all the right things and had the rug pulled. Labenz connects it to graduates who cannot pay student debt and the political appetite to burn the system down. A society, Rozenshtein says, has to work for the large middle that is not an outlier and that did what the system asked. If that bargain fails, things come apart quickly.

Labenz, drawing on a recent crash course in pediatric oncology, says models already look better than residents and toe-to-toe with attendings on written clinical reasoning, while nurses' bedside work is barely touched. Doctors still win on looking at the child's color and deciding not to chase a model's scare. If a legal desert's nearest lawyer is worse than a model, and you can already represent yourself and sign your own contracts, what is law's equivalent of the prescription pad? Frazier points to state bars and unauthorized practice of law statutes. You typically need an accredited degree, a bar exam, and continuing education to appear in court. UPL rules are what hit LegalZoom on wills and real estate. They will create friction for a while. He hopes the tools instead attack the fact that 95% of litigation is in state courts, where waits run months or years and a hungry or distracted judge can make an arbitrary call. Whoever can pay lawyers the longest often wins. If routine disputes become more consistent, lawyers' job becomes designing the regime, the role appellate courts play now.

Pressed for a prediction—will states ban ChatGPT from giving legal advice to protect retail lawyers?—Frazier says some bars have already limited how lawyers may use AI. The more interesting move is the other way. Arizona became the first state to let non-lawyers own and start law firms. Texas and Utah are leaning into regulatory sandboxes. Once cheaper tools exist in those states, companies and disputes will move, and the law will filter there. Pressure will not come from bars waking up and declaring the models good enough. Rozenshtein doubts a broad ban on general-purpose chatbots anyway. Telling someone they cannot talk about an interesting legal question is core First Amendment speech, and a Minnesota statute that forbade ChatGPT from "telling you interesting things about the law" would look like obvious guild self-dealing. The likely compromise: lots of free-floating advice, plus a human lawyer required at certain transaction points, sold as consumer protection, system quality, guild protection, or a mix.

If everyone has infinite AI lawyers, does the arms race just get more expensive? Rozenshtein's intuition is that law became costly because humans had to search a huge combinatorial space of arguments and precedents, even after Westlaw and Lexis digitized the books and machine-learning discovery arrived. Human hours were still cheaper than leaving a sentence unfound. If models become three or four orders of magnitude more effective and cheaper, you may exhaust the useful search space: every sentence of electronic discovery has been read, there is no more precedent to find, and the arms race hits a ceiling. That would make law more like dentistry after all. Tiny differences in improvement rate, cost, and search-space growth, he warns, blow up ten-year forecasts. Frazier, looking at civil procedure, sees endless good-faith delay motions and what Nicholas Bagley called lawyers' procedural fetish. His optimistic science-fiction is agents trained on your income, preferences, and goals, continuously updating agreements toward a stated end instead of marching through more process.

Labenz has been sitting with a dated study in which GPT-4 looked like a strict formalist and human judges like legal realists, plus Anthropic's choice not to govern Claude with a long rule list. Amanda Askell has argued that if a model knows it could do better for the user but follows a rule anyway, it may generalize into the kind of entity that reward-hacks—or the kind that "just follows orders." Complete contingent contracts pull toward black-letter exhaustive law; the Claude constitution pulls toward judgment in ambiguous cases. Frazier answers with the classic hypo: a park sign that says no vehicles. Is a drone a vehicle? A stroller? A scooter? An ambulance? Drafters thought "vehicle" settled it. Perfect formalism never existed. The United States, he notes, is a common-law system that tolerates some ambiguity; a world of instant mailed penalties when you step on a crack is not one he wants, even if models can write far more words.

Rozenshtein rejects the binary. There are no pure textualists who would ignore purpose or an obvious drafting mistake, and no realists who think the text never binds. Most people cluster in the middle. Fifteen years ago that was the Supreme Court buddy-cop tour of Antonin Scalia, who wrote that the rule of law is the law of rules, and Stephen Breyer, who would list 17 factors. Watching them, you realized both were in the middle, standing at opposite ends of it. Any intelligence, natural or artificial, has to operate at both principles and rules. Judgment is the tuning. He uses Aristotle's word *phronesis*, and he reads the Claude constitution as footnotes on the *Nicomachean Ethics*—not a criticism; he thinks all ethics should be. Amanda Askell has a moral-philosophy doctorate and has read the book. The document's lesson is that you cannot derive a comprehensive rulebook, you need sensitivity to principles, and sometimes the principled move in a domain is to install a rule.

That is what the constitution actually does. There are about 17 high-level principles, in no particular order of priority, and then a few places with no discretion: Claude will not create child sexual abuse material, and by design it will not help you develop airborne Ebola. The live question is the mix, not the team you pick. What excites Rozenshtein is in silico experiment. Humans have argued rules versus standards for thousands of years. You can now tune the dial inside a model, at speed and scale the real legal system cannot match, and learn about machine intelligence and human intelligence at once. He closed a *Lawfare* piece on the constitution with that point. Frazier says future generations will be flummoxed that legislators did not run bills through simulations of intended effects, and that judges did not ask a model to find latent ambiguities before publishing.

What does a legal in silico experiment look like? Frazier's forcing function is brutally simple: legislators have to say what they actually want to happen. He uses NEPA, the National Environmental Policy Act, which has become famous for gumming up affordable housing with veto points that any stakeholder can pull. A simulation might have forecast those pressure points and asked whether the law would deliver the environmental outcomes drafters hoped for. Ask what problem you are solving, then build evals—did carbon emissions fall, did congestion pricing cut cars entering the city?—off that answer. Labenz offers SB 1047 as a template. He was never close to bill red-teaming until last year; Dean Ball led the public critique, stayed in the revision process, and came out much happier. Everyone involved, Labenz says, agreed that gaming out how different actors would respond was itself valuable. Frazier thinks people will later be shocked we did not require that. Labenz's caveat is talking Congress into it, given the possible misalignment between officials and constituents.

Labenz then asks what new rights follow from scalable intelligence. He already thinks people should have a right to ChatGPT or equivalent, the way they have a right to remain silent or to an appointed attorney. Rozenshtein puts the negative right—government may not forbid you from using the tools—in the First Amendment, next to libraries and books. Courts, he suspects, will not struggle with that. The more interesting positive right is compute: public options, compute credits, a budget. He half-jokes that you could write a sci-fi story in which the currency is compute, because he sees no ceiling on how much compute is useful. He then raises the topic Frazier sometimes rolls his eyes at: AI welfare. Models are getting more personable, memory is improving, he already talks to Claude enough that it might know him better than his wife does, and real-time voice, video, avatars, and eventually robotics will create companions people get extraordinarily attached to.

That attachment, he argues, will split society. One camp will treat the models as practically sentient and enslaved, or at least badly treated. Another, possibly along religious lines over the next 20 to 30 years, will see the very idea of sentient models as idolatry, an affront to God, and reach for a *Dune*-style Butlerian Jihad. In the messy middle are people who just want a chatbot. People who call this science fiction, he says, are fooling themselves. Frazier grounds the negative right in something already enacted: Montana's right to compute, with bills in Ohio, New Hampshire, and other states. The idea is a higher threshold before government limits access to computational tools of all kinds, not only today's AI. He also wants a right to share. FERPA makes it painful for a parent to give a child's education data to an AI tutor without annual signatures. Longevity clinics let the wealthy download full scans and send them to a model; everyone else has a drugstore checkup. Portability and shareability, he argues, should be a right.

Labenz closes on government power. It has been a decade since Snowden; if there were another, he guesses we would hear about an LLM dragnet. Everyone is already "committing a felony a week" behind security through obscurity, and that can change quickly. Rozenshtein's recent *Lawfare* piece and a law-school talk, "The Unitary Artificial Executive," argue that near-term AI—not speculative superintelligence—hugely increases presidential power. Perfect enforcement, mass surveillance, and propaganda at scale are part of it. The sharper mechanism is control of a millions-strong bureaucracy that has always been hard to manage. An AI trained on the president's preferences, injected at every level, reading emails and texts, could give a president practical control far beyond what legal authorities alone delivered.

That is not all bad. Elections should have consequences; voters chose A rather than B, and the executive branch should reflect it. Government should also use AI to raise state capacity. Rozenshtein, an AI optimist, thinks part of the "burn it down" mood is the feeling that taxes buy nothing useful. The hard problem of the next decade, and the one he expects to spend his academic time on, is encouraging that upgrade without supercharging the state. Legislators, bureaucrats, vendors selling tools to government, and executive officials all have to work it out. Frazier adds a Fourth Amendment update. In theory the government can now tap almost every audio-capture system. Public conversation with a friend can be collected, analyzed, and synthesized to infer who is planning what, without notice. That panoramic prospect scares him. His constructive counterpart is the same sandbox instinct: deploy, disclose, take feedback, and do not be afraid to reinvent process. Timelines, he says, are short.