The Historian Who Taught GPT-3.5 to Beat GPT-4

Nathan Labenz invited Mark Humphries onto *The Cognitive Revolution* after an email. Humphries, a professor of history at Wilfrid Laurier University who writes on the fur trade and on Canada's First World War veterans, had tried Labenz's trick of fine-tuning GPT-3.5 on GPT-4's reasoning traces. It had unstuck his archival pipeline. What impressed Labenz more than the win was the trail of failures behind it. This was not a ChatGPT tourist. It was someone who had already become, in Labenz's view, one of the leading adopters of AI in the field.

Labenz's framing is blunt. AI's distribution network existed before the technology itself, so expert-quality applications are showing up months after the models became usable — unlike the Industrial Revolution, which took three generations to get from early steam engines to working locomotives. Humphries can now process about a thousand times more documents than before, and the searches themselves have changed in kind. Most historians will wait for products, he says, but they will not be able to ignore them. Canadian archive rules still block commercial APIs for much of the material, and Claude was unavailable in Canada until the week of taping. Labenz calls that an own goal: slowing a rare success story and blocking what is arguably the most safety-focused lab, while handing ammunition to people who want no oversight at all.

Humphries did not come to this as a typical humanities latecomer. As a kid he was into programming, followed Deep Blue versus Kasparov, and built a tic-tac-toe game in Visual Basic. His first machine-learning job was in 2007, translating German First World War histories with colleague John Maker on ABBYY FineReader, bootstrapping OCR onto an old typeface. It worked, not great. Then he ran a digitization project that scanned about 12 million pages — mixed handwriting, no standardization, a nightmare the models of a few years ago could not chew. ChatGPT, in fall 2022, did two things at once: the capabilities he had wanted for years, and a tutor that could teach him the coding that used to be prohibitively expensive to hire.

Before ChatGPT, digital history's first era was OCR on changing typefaces — long *s* characters that look like *f*s, Revolutionary-era newspapers, German blackletter. In the 2010s the field drifted toward visualization: word clouds from big data that, Humphries says, often collapsed into one sentence of an article after a huge amount of work. Transkribus was the first program that could accurately read early, flowy historical handwriting. In the winter its accuracy was about 96 to 97 percent, the old OCR bargain: three of every hundred characters wrong, and proper names the first to break. That spring he found he could dump the output into ChatGPT and let predictive text repair it, including names. By August, Transkribus had folded a language model into the pipeline so transcription and checking happen together. Historians have been snapping 10,000 JPEGs a trip for years. Those pictures were locked. This unlocks them, mostly copyright-free, and he expects the same pile to become training data.

Google Books and HathiTrust have digitized enormous amounts of bound print. That is not what most historians live on. They live on paper in boxes. For the First World War alone, documents stored in Ottawa, set end to end, run about a kilometer — three-quarters of a mile — and only tiny fractions are digitized. Extrapolate to hundreds of miles in repositories worldwide. The historian's problem has always been processing. You cannot read a mile of records in a lifetime, so you choose where to look, and that is why history is constantly rewritten. The theoretical offer now is to feed something close to all of it into a model. The same logic, he says, is already visible in law and in medicine: a radiologist can miss what a model sees in the scan because no person can process that volume. We are not there at archive scale yet. GPU capacity and price still sit in the way. The direction is not mysterious.

Labenz does the mental math on air. A kilometer is on the order of five million sheets; at a few hundred tokens a page you are in the one-to-two-billion-token range, just for Ottawa's First World War holdings. Transkribus appeared to cost more than ten cents a page, brutal on a grant if you multiply by millions. GPT-3.5 at two dollars per million tokens would theoretically process the same pile for a couple of thousand dollars. Not feasible today, Humphries says, but prices keep dropping — he expected an OpenAI cut around November 6 — and the old alternative was hiring a student at 20 to 30 dollars an hour to transcribe a couple dozen pages in a week or two. Ten cents a page already looks cheap against that. Vision APIs will make it cheaper still.

Head to head, GPT-4 Vision does not beat Transkribus on exact transcription. Drop in an 18th-century hand and it starts summarizing, rewriting, hallucinating in ways more familiar from 3.5 than from 4 on text. Humphries thinks it has seen too little old handwriting, and that it falls into a trough: it wants to *tell you about* the document rather than copy it. You can pad it — "I know you think you can't read this thing, but you can" — and it will try again. The twist is that historians used to assume you had to transcribe before you could use a page. On pension files, what he often wants is: I have this question; which of these 12 million pages matter? Maybe you only need the 2,500 that are useful. Vision already seems better at that than at a diplomatic transcription, which would short-circuit a whole digitization workflow.

The 12 million pages are not a toy dataset. Most of the pension files cannot go near a commercial API for privacy reasons, which is why Llama 2 and other local models are in the mix. The forms change year to year; some pages are handwritten notes, some are typed. A simple question — did this person receive a disability pension after the First World War? — is hard, because the award is taken away, restored, raised, vanished, restored again. A model can boil that into a case history the way a clinician boils a chart. On redacted files he can actually upload, GPT-4V does a decent job if you baby it. The real project lately is different.

With research partner Lianne Leddy he is trying to build an agent over fur-trade records: take a man who signed a contract in Montreal in 1780, then find him in thousands of pages of handwritten post journals and write a life history. Test case: a mid-level trader Humphries calls Ferdinand Wenzel, who refers to his Indigenous partner as "my girl" and his son as "my boy," in grammatically backwards English. Off-the-shelf embeddings — ada-002, Hugging Face sentence transformers — do not treat "my girl" as wife, so the most important documents never surface. HyDE, generating hypothetical documents to search against, fails in a different way. Ask GPT-4 for a choppy trader's diary and it writes literature: the trader pauses to reflect on the beauty of the morning. Real entries look like this: fine weather this morning; a tree fell on my man this afternoon; we traded five kegs of rum. Two sentences, three facts, no lyricism. No amount of prompting made the model write that way.

So the stack becomes a mixture of experts: a fine-tuned embedding model, a fine-tuned generator of search examples, a fine-tuned relevance judge. GPT-4 is decent at relevance and then stumbles on historical language, or flags racist language in the sources as off-limits and calls the page irrelevant. Keywording hits the same wall. Ask GPT-4 what *varang* is and it invents a definition; press it and it admits it does not know. The word is the wooden ribbing of a birchbark canoe. Nicole St-Onge and colleagues at the University of Ottawa built a voyageur-contracts database of 35,000 people. If the agent works, you could write mini-biographies for all of them and look for patterns no one had the hours to see. The limits now are context windows and the cost of recursive API calls. A year ago, Humphries notes, historians were not even having this conversation.

Labenz asks the obvious vendor question: OpenAI's API terms promise no training on your data and fast deletion. Why isn't that enough for Ottawa? Humphries's agreements with Veterans Affairs Canada were negotiated in 2018, when "don't email this" was the mental model and an API was not on the horizon. Canadian legislation is stricter still. Identifiable information held by the government on a person dead less than 20 years falls under the Privacy Act; you cannot transmit it, full stop, whether or not OpenAI keeps a copy. Researchers sign that in exchange for access. Hospitals will hit the same wall. Meanwhile most of his university colleagues are still at first contact with ChatGPT. There is a huge curve from "it can write a poem in the style of Bob Dylan" to "I will trust it with my data," and the curve gets longer as the models get more capable. Claude itself only landed in Canada that week, after launching in 90-odd other countries. Labenz has been running the show as an adoption accelerationist, trying to close that overhang.

Is any of this actually accelerating the history yet, or still in the promise phase? Humphries describes an iterative grind: days when the model will never do it, then a trick that works, then OpenAI ships an update that does in a week what he fought for three months. Fine-tuning on the OpenAI API, released at the end of August, moved problems he had filed as years away into the solvable column. Canada still did not have Claude; Labenz's producer Vivian had been unable to dump show transcripts into Claude 2 for topic outlines until that week. He planned to try the 100,000-token window that afternoon.

The misconception he keeps meeting is that fine-tuning pours in new knowledge. It is mostly behavior — though behavior, he adds, is a kind of knowledge, and the line is gray. A simple relevance lesson can be three versions of one page: "my girl" is relevant, "wife" is relevant, "my dog" is not. Keywords on a 650,000-word test corpus do not yield to that. Ask GPT-4 to assign keywords and you get about 11,000 overlapping labels. Give it a bank of 300 and it still sneaks off-list terms. Load the prompt with rules and it balloons to 2,000 tokens, which is expensive. That was the wall.

The unlock came from this show: Labenz talking about chain-of-thought for classification. Instead of showing "this document yields these keywords," Humphries prompted GPT-4 to break the text into thematic sections, write a description of each theme, emit keywords from those two artifacts, reread against the allowed list, then compile a final list. He ran that for about 400 examples and trained on the traces. Native GPT-3.5 on a 100-example test recovered maybe 50 to 60 percent of an ideal keyword set. GPT-4 hit about 70 to 72 percent against a human gold standard. Fine-tune 3.5 on raw GPT-4 outputs and you match GPT-4. Go through those 400 traces, correct them, add the missing slices of the distribution, and the fine-tuned 3.5 reaches about 85 percent. Students handed the same job land at 78 to 80 percent, with legitimate disagreement about which keyword fits. The smaller model, taught to think out loud the same way every time, is at the top of the human range and ahead of the larger one.

He took the same method back to relevance and saw about a 20-point jump — fatal on a coin-flip task, almost always right once it is 70 percent. Untuned GPT-4, given five bands from highly relevant to irrelevant, was too timid: of 500 semantic-search hits, about 75 percent landed in "likely relevant," and most of those were junk. A forced relevant/irrelevant split was cleaner, even if you lost a couple of true hits. Fine-tuning is what makes a five-band skim-the-top scheme usable again. An unexpected bonus: Indigenous names in the records are spelled phonetically and differently on every page. Asked for phonetic training examples, the model did not emit textbook transcription symbols. It wrote the names the way they sound. He still does not know how it learned that. Labenz's practical note for the Claude afternoon: a three-hour podcast dumped into the 100k window hallucinates at scale; about 90 minutes, or half the window, is reliable, and the 100k tier is roughly a sixth the price of 32k.

In the humanities, Humphries says, most colleagues are still stuck on plagiarism. Short take-home answers are now something ChatGPT does pretty well, which forces a rethink of what the course is for. Students starting now will almost certainly work with these tools. His classroom policy for two semesters has been permissive: use ChatGPT, you own the content. Hallucinated citations and an inability to talk about the paper fail on their own, AI or not. A student asked the obvious question: if it can do the assignment, why engage? His answer: if your job is typing the task into ChatGPT and handing over the output, that job will not last, "because you're a lot more expensive than ChatGPT."

The more useful failure mode is students bouncing off the tool because it does not finish the work in one shot. Used as an editor — make this paragraph cohere — it mostly leaves the ideas alone. It is a writing coach, or a friend who will mark up a draft. Students stuck in the C range can be pulled toward B and A with editing and explanation; a tired instructor can let the model keep restating a math concept until it lands. That is not yet systematic in universities. Institutions are in a holding pattern, often trying to avoid the issue. "ChatGPT-proof" assignments — longer papers, picky citations — are a holding action for people who have not logged in. The number of professors in his campus talks who have never actually tried the product still astonishes him. Labenz calls it a capabilities overhang: the tech is on the internet, and most people's understanding has not caught up. Closing that gap is one reason he makes the show.

The individual cost argument, both of them think, is a red herring. Twenty dollars a month is a textbook, or a Netflix subscription. The real bill is institutional. Buy Microsoft 365 Copilot at 30 dollars a month for 30,000 students and the university flinches, partly over equity between students who have a personal subscription and students who do not. Useful classroom products will need that institutional layer. Humphries expects the real shift when Copilot is simply in Office. Until then, spell-check is the analogy: grammar and spelling used to be 20 percent of an English grade; once the checker existed, you still needed them and you stopped scoring them the same way. What AI can do on an assignment, he says, is probably going to become the minimum. In the job market, someone who cannot reach GPT-3.5 on a task the job requires is unlikely to succeed at that job. Education's job is to get people above the model's baseline. It will not be possible for everybody, "just in the same way as not everybody gets an A today."

Khan Academy's Socratic tutor — it will not do the work, only lead you there — is a decent start. Labenz's colder snapshot of the present: GPT-4 is already better than the average person on most tasks, closing on expert performance for routine expert work, and still far from breakthrough insight. That raises the bar for getting hired, because most jobs are mostly routine. Humphries has nihilistic days when the model does something uncomfortably close to his own job. He also keeps repeating that the models are not autonomous. You cannot set them loose. You can expect people to summarize more documents, faster.

History's lesson, Humphries says, is that jobs tend to evolve rather than disappear. If you spend your day summarizing documents, you may keep the job and be expected to summarize more of them. Production norms go up. If you produce content for a living and it is not as good as GPT-4, that becomes a problem. Cover letters full of errors were already going to the bottom of the pile; once everyone can get a stylistically clean draft out of a model, the awkward ones sink further. People imagine a switch. He sees a gradient: tasks offloaded a bit at a time, fewer seats in some roles because people got more efficient, and new seats for people whose job is to "herd AI" — coordinate models and outputs. The auto industry started automating in the 1950s and 1960s and is still evolving. This revolution is faster. It is still not day and night. Engaging with the tools now is a way of future-proofing: know what a GPT-4 paragraph looks like before you turn one in.

Labenz agrees the monitor-the-machines pattern is well established and still worries about the chasm. "Unfortunately a lot of people cannot write at a GPT-4 level," and it is hard to see how they supervise it. One of the things he told the OpenAI team while red-teaming was: "For the vast majority of people this is superintelligence. It's just not superintelligence to you because you're really smart." For people who never sat in a university classroom, that is a very big gap, and he is not sure how many cross it. Humphries does not love sitting with that thought, and he does not dismiss it. Some of the obligation, he says, falls on companies like OpenAI — Sam Altman has talked about this — because history is ugly when technology displaces a lot of people and leaves them disaffected. This time the shock hits the knowledge economy first: people who write and analyze documents for a living. Previous revolutions were not like that. Society has not practiced for it.

You have to work both problems at once: adopt the tools, and keep people in incomes, living standards, and meaning. Asked for reading, he points to late-19th-century industrialization, when automation and devalued labor produced large impoverished classes and, eventually, the social programs of the early and mid-20th century. A more distant rhyme is enclosure: land turned into sheep runs because wool paid better than crops, peasants pushed into cities and into misery — less a machine than a new economic logic, subsistence agriculture giving way to manufacturing. The lesson he is willing to own as a historian is not a forecast. Large parts of the population become alienated from the work they knew. Governments that navigated it, in fits and starts, did so by building programs that bridged the gap. His daughter is seven. He often wonders what her world will look like in 15 years; it will not look like what he imagined when she was born. Congressional hearings about machines taking over have a place. The more immediate question, he thinks, is how we deal with the labor effects of a technology that is already here.