Training Data, Agent Evaluation and Multilingual Models

At YC Paper Club, Francois Chaubard introduces an evening about training data and evaluation. He argues that the data behind an AI system deserves as much attention as its architecture.

He recalls asking candidates what they would do after a classifier reaches 85% F1. His preferred answer is to inspect false positives and false negatives, group the errors and identify the largest practical failure modes.

For retail stock detection, fog on refrigerator glass or a person blocking the view may matter more than adding layers. Production inputs can differ from training data, so teams must understand where the model encounters unfamiliar cases.

Chaubard argues that expert data and reinforcement-learning environments need careful product design. Subjective preferences and real-world edge cases cannot be reduced to a generic collection of files.

An interface change illustrates why data maintenance continues: old traces of a software workflow may no longer represent the interface an agent must use. He sees this continuing need as a business opportunity, while making forecasts and market claims as his own assessment.

Vincent Sunn Chen of Snorkel frames expert supervision as a bottleneck: specialists know what good work looks like, but their time is limited. Turning that knowledge into usable training and evaluation data is a research problem.

Earlier data workflows often centered on individual labels or prompt-response pairs. Agent evaluations can instead require tasks, rubrics, tools and environments, each taking substantial effort to build and maintain.

Chen describes weaknesses of manual-only labeling: each example costs expert time, labels may be noisy, and changing the task can require substantial relabeling. Specialized medical expertise makes scaling especially difficult.

Snorkel’s data-programming approach expresses expert rules as labeling functions. These programmatic signals can be reused, changed and audited, but they can overlap or be wrong.

A weak-supervision model estimates the reliability of those imperfect signals and combines them into probabilistic labels. Chen compares the challenge to inferring answers from students of different skill levels when no answer key is available.

As coding agents work on longer and more complex tasks, Chen says evaluation must evolve as well. A benchmark that becomes easy does not mean all relevant software work has been solved.

He introduces Senior SWE-bench, built with collaborators from the original SWE-bench team. Its goal is to represent higher-level engineering work such as interpreting logs, implementing user stories and making architectural choices.

The tasks use relatively high-level instructions rather than prescribing every implementation detail. The evaluation also attempts to capture code quality and whether a change fits the practices of a repository.

Chen describes a verifier tradeoff. Fixed tests execute reliably but may reject valid alternative implementations; a language-model judge can adapt to different solutions but may accept plausible code that does not work.

Their proposed middle ground is a validation agent. An expert supplies user stories and requirements; the agent uses the candidate patch and specification to write deterministic tests against relevant code paths.

The team calibrates this process and checks for issues such as incomplete tests or reward hacking. Chen says its “tasteful pass” metric combines correctness with qualities such as fit to the codebase and avoiding unnecessary bloat.

Chen closes by calling for more research and benchmarks as agents gain autonomy and outputs become harder to verify. His company also offers grants for benchmark work.

Volo Kuleshov of Inception Labs presents diffusion language models. Rather than producing each token strictly left to right, the models he describes start with a noisy sequence and refine multiple positions over several steps.

Kuleshov argues that parallel token generation can improve throughput, and cites speeds above 1,000 tokens per second for his company’s models. Such comparisons depend on the hardware, workload and measurement conditions.

He points to real-time voice agents, where a pipeline may include speech recognition, a language model and speech synthesis. Faster generation can reduce one source of latency or allow a different quality and compute tradeoff.

Kuleshov says widely used customer-support benchmarks may underrepresent real business policies, tools and complexity. He also warns that high benchmark scores can coexist with much lower performance on production logs.

His team’s Tau Forge system generates reinforcement-learning environments, drawing on real interactions when available. For a banking scenario, it may synthesize policies, accounts, a database and tools that an agent can use.

The system builds tasks from scenarios and user personas, then adjusts difficulty. Kuleshov says tasks that are always solved or impossibly hard supply less useful learning signal, so the team filters and modifies them.

He reports that training on generated tasks improved his team’s model on a separate set of business environments. He describes a split across businesses to reduce overfitting; the result is a company-reported experiment, not proof of performance in every deployment.

Shayne Longpre presents research on multilingual pre-training and interactions among language datasets. He says the same methods can also be applied to other data sources or domains.

Using Thai as an example, Longpre says one common web-derived corpus contains far fewer Thai than English tokens. A model trained only on the scarce language may repeatedly see the same material and overfit.

A mixture can combine target-language data with other high-quality, useful sources. Which languages help is an empirical question: shared topics, data quality and representation can matter as well as linguistic family.

The researchers compare learning curves for monolingual and mixed-language training to estimate transfer. A matrix summarizes which source languages help or interfere with performance on a target language under their experimental conditions.

Longpre says the transfer matrix changes with model size: larger models in their experiments showed more synergy, while smaller ones experienced more competition. These findings are conditional on the tested models and data.

Transfer is not necessarily symmetric: if language A helps B, B need not help A to the same degree. In their measurements, shared script was somewhat more predictive than language family, although both mattered.

The team fits scaling and mixing laws that account for target-language data, useful transfer languages and diminishing returns from repetition. The objective is to estimate loss and select data mixtures and model sizes under a compute budget.

Longpre closes with practical questions: when to fine-tune an existing multilingual model versus pre-train a new one, and how much model capacity and data are needed to add languages without losing performance.