Twelve Hours After a GPT-4 Demo, He Turned a Decade-Old Legal Startup Around

Pablo Arredondo co-founded CaseText after a stretch as a patent lawyer at Kirkland & Ellis, and he is now VP of CoCounsel at Thomson Reuters, which acquired the company in 2023. Host Nathan Labenz asks him to start further back than the GPT-4 moment: how did lawyers actually find the law before computers, and what did the last decade of CaseText change? Law, Arredondo notes, had a citation graph long before the web had hyperlinks. Every opinion cites earlier opinions, often down to the page, and the American explosion of lawyers and litigation quickly produced more cases than anyone could read in a lifetime.

Two problems followed immediately: how do you find the right case, and how do you know the case in your hand is still good law? In the territory of Maine, before it was a state, a lawyer named Simon Greenleaf cited English authority because there was almost no American case law to use. The case had already been overruled. The judge humiliated him in open court. Greenleaf, Arredondo says with a little license, never wanted to feel that again, so he started a list of which later cases had killed which earlier ones. That list grew into the citator, the meta-resource lawyers still use; the famous brand was Shepard's.

Written case law itself was a mess until West Publishing, whose lineage now sits inside Thomson Reuters, put opinions into a uniform series of reporters. The wall of books behind a TV lawyer is that system. West also built a human taxonomy — tort, then injury, then animal attack, then dog — that helped people find cases and also trapped them. However the classifiers sliced the law was how you had to use it. In the late 1960s and early 1970s, digitization arrived, eventually as Lexis: millions of opinions searchable by keyword and Boolean proximity. Precision suffered because stray words came back; recall suffered more, because the case you needed had used different language. That, Arredondo says, was still the paradigm when he was practicing.

CaseText's first commercial product attacked a blind spot in the citators. Shepard's and its rivals only followed the direct citation path: cases that explicitly cited yours. Arredondo's analogy is a video-store clerk who, asked for films like *The Godfather*, names only *The Godfather Part II* and *Part III* and never *Goodfellas*. Spotify and Amazon already used soft co-occurrence — people who downloaded this also downloaded that. CaseText took a lawyer's brief, analyzed every cited case, and suggested opinions that were not in the brief but should have been read. At top firms where money was not the constraint, the reaction was dread: how did we miss this, we hire the best attorneys?

The company ended up in a strange bimodal market, which Arredondo compares to a Jack Daniel's ad that cuts from a Ritz wedding to a biker bar. CaseText sat in Manhattan skyscrapers on eight-figure antitrust fights and also in strip-mall solo shops that had been stuck on Google Scholar. Building a full research engine meant solving Greenleaf's 1800s problem: does this case overrule that one? Courts sometimes say "we overruled Jenkins." They also say they "regretfully consign Jenkins to the dust bin of oblivion," because judges may write however they like.

Around 2018 Google open-sourced BERT — paper, code, and method, Jacob Devlin and all. CaseText trained the same approach on U.S. common law, with humans still in the loop for triage. The first flicker Arredondo saw was the computer flagging an overruling that used none of the usual words. From there they built Parallel Search: a lawyer could paste a full sentence and retrieve analogous case law with no keyword overlap. A demo about someone fired for not wearing a mask at work surfaced hard-hat and workplace-safety opinions. For lawyers who had lived in what he theatrically calls the keyword prison, it felt seismic. GPT-3, in 2020, got ten minutes of wonder and then a no: not reliable enough, not nuanced enough, not something you could hand to an attorney.

Then came September 16, 2022. Arredondo and co-founder Jake Heller were shown a GPT-4 demo, and within twelve hours they had pivoted the entire company around it. ChatGPT, based on GPT-3.5, was still six weeks away; the public would wait months more for GPT-4. Sam Altman later said the lab split the releases because seeing chat and GPT-4 together would have been too much. CaseText got both barrels: a Slack channel where they could ask GPT-4 to build timelines, rewrite documents, try anything. The next seventy-two hours, Arredondo says, felt as if he would burn a hole in his brain from how many tasks they ran.

They also learned, immediately, that you cannot give lawyers a naked GPT-4 chatbot. He asked it about a case he had actually worked. The answer was so believable he second-guessed himself. When he pushed back, the model told him he could brag about his cases if he wanted, but it was right, and here was proof — a concocted URL that went nowhere. Labenz had an uncannily similar night: he asked whether an undergraduate chemistry professor had ever co-authored with him, the model said yes and named a paper, and he had to go look it up to re-ground himself. Later, lawyers would walk into court on hallucinated cites, and startups would put fake cases in marketing screenshots. CaseText got tricked in the lab first.

From the earliest days they treated retrieval-augmented generation as non-negotiable: a search engine retrieves real opinions, and GPT-4 must answer from what is in front of it, not freestyle. What convinced Arredondo the leap was real was not the bar-exam study he co-authored with Stanford colleagues, but law librarians. They live to tear tools apart. When they said the system was "solid" and seemed to handle nuance across areas of law, he believed it. CoCounsel — an homage to Copilot — debuted on MSNBC's *Morning Joe*. A legal-tech product launching on national television would once have been absurd. The old demo loop was one librarian saying "that's kind of neat" and maybe letting another see it in three weeks. Now entire partnership rooms assembled as if someone had been embezzling.

CoCounsel began as a grid of skill buttons — legal research, review documents — then moved to chat because people found that more natural, then had to recover some structure when chat made intent too ambiguous. The live product sits between Labenz's "copilot mode" and "delegation mode": you converse, spin off tasks that run in the background, and come back to a results tab. The skills span litigators, who live in case law, and transactional lawyers, who live in contracts and diligence. CaseText had spent ten years on the former. GPT-4 made the latter suddenly buildable.

Legal research runs RAG over real opinions. Deposition prep proposes topics, lets the lawyer edit them, then drafts questions. Timeline takes a messy corpus and builds a chronology — the kind of night work Arredondo remembers as a first-year associate. On the deal side, a lawyer can ask what a merger-agreement term says. Contract-policy compliance is more interesting: a Fortune 50 company feeds in its rules (Delaware law, certain IP terms) and every incoming draft is checked, with a suggested redline when a clause is not kosher. He analogizes the progression as spell check, then grammar check, then, with GPT-4, substance check. A "what's market" skill pulls SEC filings, scans them, and synthesizes a report.

The use case he most wants people to sit with came from another Fortune 500 client. Certain expert witnesses make a living testifying against the same company year after year. The client asked: if we give you this expert's prior reports and testimony from multiple litigations, can CoCounsel find contradictions we can use on cross? That, Arredondo says, is the kind of thing lawyers go to law school to do. The cliché is that AI will only automate menial work nobody wants. The models do help with tedium, but they also help with deeply substantive work — "finding inconsistency so you can go do the famous cross-examination that destroys the witness."

Long context windows have broken Arredondo's heart a little. They turn into a mirage: the model misses things in the middle and gets less accurate when you fill the window. His image is a boat advertised for a hundred people that sinks if you put more than ten on it. For legal work, where being right matters, those windows have not been there yet, though he sees progress. New OpenAI releases are not hot-swapped. The team retests because a model that is better at one thing can be worse at what lawyers actually use. Early access was worth it, he says, but it also meant doing things the hard way — weeks of begging GPT-4 for JSON before structured outputs and function calling existed.

Law is violent and often racist on the page, so they had safety filters removed on dedicated instances; a consumer chatbot's alignment is to steer clear of that material. They have not shipped a legally fine-tuned GPT-4 variant. The retrieval engine for case-law RAG uses a home-built embedding model trained on common law; GPT-4 does the heavy lifting. During alpha they got a tiny tile in GPT-4's reinforcement learning, with reference attorneys teaching the model how a document relies on a case. Users never see the most important work: after the pivot they stood up trust teams whose job is to wake up and find the system messing up — cartoon hallucinations, wrong answers, bad quotes, all of it.

The mix is as much automation as possible plus lawyers actually using the product. They launched with a small subset of skills that had passed; timeline was pulled even though users loved it, because testing said it was not there yet. After the Thomson Reuters merger they folded that philosophy into what he calls a master skills factory, pairing CaseText's velocity with Reuters's editorial gold standard. Model-on-model eval is a different calculus when GPT-4 would be policing GPT-4. They do use the model to re-rank its own document-review scores, which produces a more intuitive list, but he does not see a world yet where quality control is fully handed over.

Chunking, for Arredondo, is where domain expertise belongs. In a deposition transcript you do not split a question from its answer. Their in-house embeddings, as of late 2022 and early 2023 — a century in this field, he jokes — still outperformed the generic options they tried, and they re-evaluate constantly because user experience, not sunk cost, is the point. They are not throwing the latest vector databases at the problem, and they are not asking GPT-4 to traverse the citation graph they already maintain for traditional research. Firms, proud of eighty-year archives, used to ask whether CaseText could train "a GPT-4 on just our data," as if summary-judgment motions would be the missing ingredient after the whole internet. His answer: no. Point GPT-4 at the documents with RAG.

The pitch to build a firm-specific model, he says, had no evidence of a better outcome; it fit the ego of wanting a competitive edge. Firms have mostly come around. Inside the product, six database types look like different tools; Labenz is told, for listeners who deserve the unvarnished truth, that they may be identical and exist mainly to teach users what they can do. Security is the first question in every law-firm meeting, even if you are showing a time machine. Documents do go through GPT-4. Clients can have them deleted immediately or daily, the stack is SOC 2, and CaseText does not train models on customer data — a brief differentiator in a window when OpenAI's stance was still ambiguous, before enterprise reality settled the same way.

Thinking like a new assistant who can sneak into the office the night before day one is his metaphor for extra structure: cheat sheets, contact lists, a personal index. They have started parsing a docket before a skill runs so that "send a letter to IBM's counsel" has somewhere to land. GPT-4 first, then get cute, has been the default, and OpenAI's guidance matched it: build the best product, then optimize. Other models keep failing their reliability bar. Machine-learning colleagues are annoyed at how much of a GPT-4 maximalist he is. For most of the prior year the gap to second place was the difference between putting a system in lawyers' hands and not. Subtasks can go to cheaper models later; the main line does not move until something else actually works.

CoCounsel comes in around $200 per attorney per month, ten times a retail ChatGPT subscription. Labenz expected sticker shock. The profession's response, Arredondo says, has been better than they feared, with a split between huge multinationals and solos. If buyers compare it with a Westlaw seat or a document-management tool, it looks pricey. If they compare it with hiring a paralegal to do the same tasks, it looks like a bargain. Early electricity put street lamps only on Wall Street and Madison Avenue; high-throughput GPT-4 document review is still more practical for deep-pocketed firms than for an everyday attorney, but costs are falling and they keep designing prompts to burn fewer tokens.

Hallucinations of the grotesque kind are blocked by the RAG architecture plus links under every case and quote checks that verify a cited sentence actually appears in the opinion. A link cannot point at a fake case. The remaining errors are subtler: misreading a judge, missing user intent. Arredondo used to hate thumbs-up and thumbs-down on a professional search engine — a doctor should not ask how the prescription went. Generative AI converted him. There is real-time feedback, a weekly meeting that discusses only complaints, and systematic tests that never stop. Nobody is perfect, he notes, and you cannot say that about a human hire either.

They made an early decision not to sell to the public or to companies looking for a substitute for counsel. A memo from CoCounsel looks finished; a lawyer still sees exceptions, jurisdictional traps, and clauses that are enforceable in one state and dead in another. Providing help to people who cannot afford lawyers is, he thinks, one of the most important things large language models will do, and society should be judged by how well it solves that problem. He points to Stanford's Margaret Hagan as the person to study. CaseText did not want anyone to conclude "I have a lawyer, it is called CoCounsel." Unauthorized practice of law is a separate issue; the deeper reason was that they did not feel comfortable. Companies can use it on transcripts and files. They will not market it as a reason to skip an attorney.

GPT-4 still cannot hold the broader context of a matter well enough for legal strategy, and Arredondo does not think it writes persuasively enough for real briefs. He admits a romantic view of writing and cites Paul Graham: writing is thinking, and letting that atrophy is dangerous in a way spellcheck and GPS are not. Declarations and supporting documents can go to a model. Actual advocacy should not. When a judge floated first drafts of opinions, Arredondo called it dangerous. Law evolves because originality is written into it. A world where AI generates a brief, a judge's AI reads two briefs and emits an opinion, and humans only supervise, is one of the profession's deepest risks.

Richard Susskind, the English "OG" of AI and law, notes that when a new medicine appears, doctors talk about patients, not billable hours. Lawyers owe a zealous duty to the client; if that means less money at the end, that is fine relative to the duty. Partners who have a first-year associate do twelve hours of basic document review, bill the client $600 an hour, and pay the associate $100 will have to rework that paradigm. The contrast Arredondo likes is two year-end essays by Chief Justice Roberts. In 2016 Roberts almost bragged that the legal system is slow, and pointed out that the tortoise and the hare are carved into the Supreme Court building. Arredondo's retort is that life has more options than a turtle and a narcoleptic rabbit. In 2023 Roberts wrote about AI, quoted Rule 1 of the Federal Rules of Civil Procedure on just, speedy, and inexpensive resolution, said legal research will be unimaginable without these tools, and added that judges will still have jobs.

AI arbitration is the more ambitious sketch: both sides consent, submit information, and a background of RAG and prompting encodes the dispute-resolution rules, with a possible appeal to a human. His contracts professor, Larry Lessig, saw an early GPT-4 demo and, instead of applying the brakes, said he had been thinking about an AI judge for thirty-five years. Pre-LLM expert systems failed because law is too fuzzy; the new models reopen the question. On regulation of the bar, Arredondo is modest about fair use and sends listeners to Mark Lemley's Stanford work. His own advice to courts and state bars is that existing rules, properly applied, get you 99 percent of the way: a duty of technological competence in most states, candor to the court, and the ancient rule that you do not cite a case you have not read. New product-liability regimes can wait until those duties are enforced.

Thomson Reuters bought CaseText because it was already awake. Steve Hasker has said the company wants AI across every vertical via a build, buy, and partner approach, and the startup was brought in to join that work rather than sit in a silo. The personal thrill is content. Arredondo likens the old days to building with sticks and mud, and Reuters to the Roman Empire: reporters, dockets, law-review articles, Practical Law, Phoenician dyes and Greek marble. There are no excuses left. He closes by calling people who train large language models heroes of justice. Downstream of their work, his corner of the world is trying to make legal services faster, less expensive, more accessible, and higher quality. "Now," he says, "it's on us."