Ian Quigley on Testing Scientific Agents in Premeasured Worlds

The Latent Space Science hosts welcome Leash Bio cofounder Ian Quigley back after an earlier discussion of DNA-encoded libraries and high-throughput screening. They describe the new idea: turn the resulting measurements into a virtual sandbox where agents make medicinal chemistry decisions and receive rapid feedback.

The hosts ask how the initial idea became an environment for studying scientific reasoning. Quigley says the screens are large and sufficiently rich in alternative measured outcomes to give agents tasks under a limited experimental budget, then observe their choices across a known landscape.

The flagship task concerns structure–activity relationships. Starting from molecules that may bind a target protein, the agent tries to find changes that improve binding. Quigley stresses that this task does not yet cover the full set of drug-like properties, such as metabolism.

An agent receives a catalog, a target and a budget of 100 molecules. It must search for the best binder it can find. Different agents allocate that budget differently, write hypotheses about why molecules work or fail, and then test those hypotheses against premeasured results.

Because the measurements already exist, the team can replay the experiment many times; Quigley reports about 90,000 rollouts. He says the accompanying preprint separates performance across model families and levels, while the full measured landscape makes it possible to examine decisions as well as final scores.

An agent may stop at a molecule far below the best one in the environment. Quigley describes agents that give up, overlook results that contradict their hypotheses, or repeatedly test the same molecules despite a limited budget. The data also lets researchers investigate when agents explore versus exploit.

Quigley says higher-performing models tend to abandon unproductive molecular neighborhoods faster. Asked why, he says the team does not yet know: those models move around more generally, but they are particularly quick to leave poor areas. Lower-performing models can keep making small changes in an unpromising region.

The team can inspect the natural-language reasoning in each rollout. Quigley gives examples of agents discussing chemical rings or a nitrogen atom and says medicinal chemists who reviewed a limited number of trajectories found the proposed lines of thought plausible. Plausible narratives, however, are distinct from proving that a search decision was correct.

A host asks whether the task could become a game for human chemists. Quigley likes the idea but points out a constraint: in a premeasured environment, people cannot try every modification they might imagine. A human-facing version would need to make the measured options understandable, perhaps by reducing the set of building blocks.

Asked which frontier models lead, Quigley discusses the team’s current comparisons and says the rapid arrival of new models makes the results a moving target. He also says Leash tested smaller and open models, including DeepSeek V4 Flash. These are the team’s reported results on this task, not a universal ranking.

Quigley recalls that the previous summer the team began exploring supervised fine-tuning to improve models’ understanding of SMILES. At the time, he says, available frontier models struggled even to convert between different SMILES representations of the same molecule.

SMILES represents a molecular graph as a one-dimensional string. Because a graph traversal can start at different places, multiple strings can describe the same molecule. Tools such as RDKit can perform conversions, but Quigley says the team had wanted models to handle the representation more directly.

By around February, Quigley says, models began working with chemistry meaningfully enough to encourage this direction. The agents now inspect molecules and use feedback to choose subsequent candidates; when molecular information is removed, their decisions suffer. He corrects the language of learning: the model weights are not being changed during these runs.

A host asks whether Bayesian hyperparameter optimization could serve as a deterministic baseline. Quigley says the team has not used it and suspects the landscape is too spiky and high-dimensional for that approach. That is the team’s expectation, not an experimental comparison presented in the conversation.

Quigley describes the release as a preprint, blog post, measured environments and many agent transcripts. He says the purpose is to let researchers examine scientific decisions, mistakes and use of physical feedback, and test whether a new training recipe improves those decisions at scale.

He calls the design a lab in the loop with amortized latency: measure a world once, then run many trajectories against it. This is close to fast feedback for a rollout, but it is not literally a live lab with zero latency. A conventional live experiment takes time and may not reveal the alternatives an agent did not choose.

The tradeoff is freedom. A live experiment can allow a much broader set of choices at each step; replay is limited to what was measured in advance. Quigley says the team has screened roughly a thousand proteins against chemical libraries and is working to grow the number of available environments.

He says the team is producing roughly two new environments a day and hopes eventually to reach 20, while describing that figure as a plan rather than a result. Fast, scalable feedback is attractive for agent development, but gathering physical measurements remains difficult.

Quigley says the team does not yet know whether reinforcement learning progress in these worlds will transfer to new biology. The participants discuss the broader problem of spending a limited experimental budget: when to keep exploring a promising area, when to leave and how a decision skill might generalize.

Quigley adds that the team is incorporating more structural tool calls. The hosts close by pointing listeners to the forthcoming resources and thanking him for the follow-up conversation.