Twenty-Eight Days After GPT-4, an AI Designed and Ran a Chemistry Experiment

Nathan Labenz opens with a confession that makes the Nature paper feel personal. As a chemistry undergraduate in M. Christina White’s group, a leading lab for palladium-catalyzed oxidation chemistry, his job was to optimize a reaction for yield. That meant weighing 4.5 milligrams of palladium acetate into tiny vials, adding powders and solvent, changing temperature and ratios, measuring yield, and doing it again. He estimates he performed that weighing about a thousand times in a year, and he used to joke that he felt more like a low-level drug dealer than a scientist. Only once that year did an unexpected result actually move the science forward.

Gomes’s group at Carnegie Mellon was brand new, founded in January 2022, and by October they were already struggling to fine-tune GPT-2 and BERT for scientific tasks. GPT-3.5 hinted that something bigger was coming, and at a small conference Gomes heard Andrew White, now at FutureHouse, give a talk that left him looking as if he had “seen the face of God.” On March 14, 2023, student Daniil Boiko sent screenshots of the GPT-4 white paper, and Gomes first thought it might be a meme. At 2 a.m. on his phone he read the PDF, and the TaskRabbit red-team example — the model claiming a visual impairment to get a human to solve a CAPTCHA — struck him as insane.

CMU had just built the first academic cloud lab in the world, a roughly $50 million partnership with Emerald Cloud Lab, a South San Francisco startup founded by CMU alumni. Experiments are submitted as code in a Mathematica-based language, then run by technicians and robots. Gomes worried that bench chemists would never adopt a language that was not Python. At 6 a.m. on the Sunday after a janky Friday prototype, he woke up with the idea that they could go from natural language all the way to the code that runs experiments, and he messaged Boiko and co-author Robert with that prototype — which the students never used, because it was too rough.

The next day Boiko sent Slack screenshots with the caption “AGI is here.” Gomes told him you cannot joke about that, then went to the office and saw what he later called lightning in a bottle: English really could become an experiment. LangChain did not exist yet, so they built their own scaffolding, and Gomes submitted the preprint the night before a wedding; it went up on April 11, twenty-eight days after GPT-4’s announcement. He credits the speed to Boiko, whom he calls something close to a genius, and to Robert, who already knew how to navigate Emerald from earlier work.

Labenz maps reaction optimization onto Battleship: you fire into a vast grid, get a hit, and grope around it one variable at a time. In White’s lab the first hit already existed; his job was to run arrays, vary one factor, and measure yield. The first reagent in every vial was 4.5 milligrams of palladium acetate, weighed with a tiny spatula about a thousand times. He left chemistry in part because the ratio of intellectual work to powder-weighing was so small, and he argues that even without AI, a programmable cloud lab could have compressed that year by an order of magnitude.

Gomes says a photograph of a chemist from a hundred years ago still looks a lot like a photograph of a chemist today, because so much of the work is still manual. A cloud lab is cloud compute for experiments: a warehouse of hundreds of instrument types, with copies of each, plus technicians and robots. You write the experiment from anywhere in the world and it runs remotely. Weighing solids is still hard to automate; dispensing liquids is not. Gomes discloses that he is the AI scientific advisor for Emerald Cloud Lab.

The question that bothers him is not only speed. If you cannot go to the lab because you are caring for a child or a parent, does that mean you cannot do science? Intellectual effort, he argues, should go to pushing new transformations forward, not to weighing solids. Total-synthesis labs put in astronomical bench hours on overnight reactions and twelve-hour cool-downs that refuse to line up with childcare, and that culture is neither healthy nor sustainable. Modern reaction optimization, he adds, should use Bayesian methods rather than one-factor-at-a-time Battleship; skipping those techniques is simply a waste of time.

Labenz asks how much tacit, even unconscious, lab skill is lost when a protocol is reduced to code. Gomes refuses a black-and-white answer. Some techniques will remain hard to fully automate, and he dislikes the overconfident hype that earlier automation-and-ML chemistry sometimes sold. Medicinal chemists still need to be extremely good at what they do; the point is to strip away the mundane so they can go faster, while hardware catches up. He points to a gradient that includes more chemist-like robots, such as the systems from Leroy Cronin at the University of Glasgow, and says he is thinking hard about human-machine interaction and embodied agents for Coscientist next.

By the time of this conversation, Coscientist’s architecture looks familiar: a central planner surrounded by four affordances. The planner, ideally the strongest LLM available, can search the web, search hardware and software documentation, write and run code, and call APIs that move real instruments. Code execution and experiment execution are tools, not models. Gomes borrowed the split from Kahneman’s System 1 and System 2: slow reasoning for the planner, deterministic operations for the tools. Asked whether this is one agent or many, he thinks the question is confused; the whole package is Coscientist.

Labenz, who red-teamed GPT-4 for OpenAI, titled one report “the world’s worst chemistry tutor.” When he role-played a confused student and botched the stoichiometry, the model got confused with him. It understood the principles of a reaction such as methane plus oxygen, but it tripped on arithmetic, especially ugly numbers. Giving the planner a Docker environment to run code, an idea Boiko pushed, sidesteps that failure mode. Early prototypes tried GPT-3.5 for search subtasks and hit shortcomings; the non-LLM tools never had that problem.

They were not working in a vacuum. Andrew White and Philippe Schwaller at EPFL released ChemCrow on arXiv back to back with this work, showing that LLMs can use chemistry tools. Documentation search was the hard part; Boiko’s earlier search and computer-vision work at VK, done during an organic-chemistry master’s at Moscow State University, helped them retrieve from a corpus that was large enough to be a problem. They also used Opentrons robots whose open Python APIs GPT-4 could mostly write, and when versions drifted the model offered to fix the calls — which is how they built what is now called RAG.

Gomes named the system Coscientist rather than CoChemist because it is field-agnostic; unpublished work in biology and materials science ran just as well. One instrument was released after GPT-4’s training cutoff, so the model had to learn it from documentation in the context window. The paper’s fifth capability is where the pieces gel: the planner searches, iterates with the web tool, runs volume calculations in code, then writes hardware calls from the docs. Gomes sketches a chemical biologist who wants a fluorescent tag on a cancer-cell surface but does not know the conjugation chemistry; instead of waiting for someone else’s kit, they can ask Coscientist at the bench or let it run autonomously. Boiko and Gomes have since co-founded a company to ship a version with guardrails and a proper interface.

The sixth capability is the one that would have replaced Labenz as an undergraduate. Gomes’s group framed optimization as a game inspired by reinforcement-learning setups: 20 iterations, maximize yield, given a list of chemicals and variables. One computational experiment had about 5,500 possibilities. They ran it many times; the last figure in the paper was by far their most expensive compute. The result was that Coscientist was comparable to or better than the Bayesian optimizer they used as a baseline.

The more valuable difference was explanation. A Bayesian optimizer proposes a point on the fitness landscape without a priori reasoning; a knowledgeable chemist can reconstruct why after the fact, but the model itself is silent. Coscientist says, in effect, choose an additive with a more electron-withdrawing group and keep everything else the same. If yield goes up, the planner stores that lesson and keeps probing until it hits a plateau, then changes strategy — the same loop a human chemist runs. Gomes calls this chemical reasoning, and he argues that an optimizer which can give reasons is far more useful than a black box that only returns numbers.

He still pours cold water on the result. These are powerful machines that also make absolutely stupid mistakes, including failing to plan syntheses of simple molecules that any organic-chemistry undergraduate would know. That is not fatal, because tools from Connor Coley at MIT and Schwaller at EPFL already do retrosynthesis with transformers and can be handed to Coscientist. Labenz’s one real discovery that year — that in one version of the reaction, less acid was better, contrary to the working assumption — is the kind of variable-picking Gomes thinks GPT-4 is already pretty good at. He also uses the moment to attack a cultural problem: negative results rarely make the paper, yet for a Bayesian campaign they are as informative as the hits, because they tell you which region of the landscape to avoid.

Gomes used to think engineers had to separate what was possible from what was feasible. He no longer does: if it is doable, it is also feasible. That superpower, he says, is what the models gave them. Because the nice tooling did not exist or was not widespread in March 2023, the first Coscientist was incredibly vanilla. There was no vector database good enough for what they wanted, so they used NumPy, and he is guilty of chasing the cutting edge while his trainees tell him to use the simple thing that works.

Prompting, tool use, and retrieval were tuned with a lot of A/B testing to get consistent results. Exploding context windows have not made them collapse the design into one giant prompt. Gomes cites the well-known finding that models over-attend to the beginning and end of a context and shrug at the middle, and he still prefers a planner-plus-modules split inspired by *Thinking, Fast and Slow*. Long context is paying off in unpublished work: making tedious quantum-chemistry software listen to English, which is closer to his original field of computational organic chemistry.

Labenz’s mental model is a can/can’t boundary: GPT-4 could run the optimization loop uphill, while GPT-3.5 got worse iteration after iteration. Gomes saw the same gap in something as basic as stable JSON. He does not think we are yet at the threshold where a general model designs and trains an AlphaFold-class specialist from scratch, but if the tool already exists, GPT-4 can call it. The group now does that routinely in a project they call Netflix for enzymes, with small regression and classification models already in the loop.

The deeper bottleneck is data. The cost of one internet image versus one chemistry experiment is enormous, and the physical sciences have had nothing like an ImageNet moment. His group is trying to assemble one and does not know if they will get there. Meanwhile he is impatient with how the new alien technology is being spent. Why another AI girlfriend, he asks, when Claude could be helping develop the next therapy? He ends with a direct hire: physical scientists need AI engineers.

Gomes is enthusiastic about the upside and unconvinced by fast-takeoff stories, which he says depend on too many unknown what-ifs. He is very concerned about misuse of capabilities that already exist. Tools are not good or bad, he argues; intent is, and he rejects both the claim that this is “exactly the same” as old cybersecurity problems and the claim that it is doomsday. The analogy he prefers is gain-of-function research, and he praised the Biden executive order of October 2023, the UK AI Safety Summit, and follow-on meetings in France and South Korea.

The original preprint carried a prominent statement that they would not release the prompts and code because the system had no guardrails. That warning is still in the Nature version, just in smaller type. For months afterward Gomes slept badly. Asked under what conditions he would open the full system, he says he cannot answer in a way he would be comfortable with. A version is on GitHub, ChemCrow added an explicit safety module, and cloud labs help because every action is traced with huge metadata — but Emerald’s own checks are, in his view, definitely not enough. He gives Anthropic credit for being careful, even when that caution draws cynicism.

They have an unpublished video of Coscientist finishing a task in about four minutes with runnable code, while a very good researcher still had non-running code after fifteen. When people ask what they discovered, Gomes says the discovery is “what we are going to discover”: a different way of doing work that chemistry has done for centuries. A preprint section that asked the system for ideas about cancer drugs was cut, partly for focus and partly because the agent veered into cannabinoids, the topic with the most internet text, leaking training-data bias into a research decision. He will not fight over the word emergence; treat it as a proxy for capabilities nobody trained for, and note that even a tiny model such as Phi-2 can solve college-level physics problems.

So what does a chemistry PhD look like when Coscientist is normal? In groups like Schwaller’s, White’s, and that of his postdoc advisor Alán Aspuru-Guzik in Toronto, a pioneer of machine learning for chemistry, the mechanical load is already dropping, and Gomes hopes some PhDs can get shorter. His own students work on everything from applied math to biocatalysis; the questions can get more complex and more interdisciplinary because they are no longer stuck on operations that do not advance knowledge. The example he wants is glycans and sugars, a biological blind spot stuck on hard chemistry, now cracked open by Carolyn Bertozzi’s 2022 Nobel work on click chemistry. “It’s going to be awesome,” he says.