Ashwin Paranjape: Testing financial agents against earnings consensus

Ashwin Paranjape introduces his work at Samaya AI, from retrieval and long-context systems to a financial-agent harness and benchmark. He asks whether agents that perform increasingly complex technical tasks can improve financial forecasting.

A colleague with investment experience cautioned against starting with stock-price predictions: prices are noisy, drivers are hard to isolate and an apparently correct prediction may be based on flawed reasoning.

The team chose quarterly earnings instead. Companies report revenue, margins, operating income and earnings per share, while analyst estimates are pooled into consensus; results above or below that consensus are described as beats or misses.

Paranjape sees three advantages for evaluation: a quantitative outcome, a human expert baseline and management commentary that may help explain surprises. He asks whether an agent can make a useful forecast before the release.

The team built an environment around a research harness. An agent receives a task and historical information, researches the company and produces a forecast that can later be compared with the reported result.

A point-in-time gate limits retrieved information to the selected prediction date, such as five trading sessions before earnings. Samaya modified document retrieval and structured-data tools to support that cutoff.

Unrestricted web search is difficult to constrain retrospectively, so the team instead used additional timestamped news feeds. The harness also provides expert guidance on using context and approaching the prediction.

Paranjape reports results for more than 450 earnings releases and seven frontier models. The period begins after the newest tested model’s knowledge cutoff, an attempt to reduce the risk that reported outcomes were already in its training data.

The first metric is prediction error normalized by each company’s historical volatility. The comparison includes raw analyst consensus and a bias-corrected consensus intended to account for the tendency of many reported results to exceed estimates.

In the revenue comparison, Paranjape says GPT-6 Astra significantly outperformed even the bias-corrected baseline. Results vary across revenue, gross margin, operating income and adjusted earnings per share; not every difference is statistically significant.

The team also measures hit rate: whether a forecast correctly calls a result above or below consensus. For below-consensus outcomes, it compares models with a baseline based on the company’s majority outcome in the previous eight quarters.

A third metric, surprise correlation, measures whether predicted surprises line up with actual surprises in direction and magnitude. Paranjape says the leading models outperform the baseline on that measure, while their differences from one another are not statistically significant.

For an example of a good forecast, he describes a semiconductor supplier to data centers. The agent found timely information from a smaller supplier, incorporated it into an income-statement forecast and moved above consensus toward the eventual result.

A missed example concerns Ross Stores. The agent found information about a peer’s tariff refund but did not incorporate it into the gross-margin forecast; its estimate remained near consensus while the reported result was much higher.

Paranjape says the models’ strongest departures above or below consensus were particularly interesting when actual surprises were large. The examples illustrate possible reasoning patterns, but a few cases alone do not establish a trading strategy.

The team next tests data freshness. A model using only its stored parametric knowledge performs poorly on the task, despite pretraining that may have included earlier documents.

Using data from about 11 weeks before the release, rather than the latest available information, reduces performance by roughly 12 to 15 points in the comparison he describes. He compares the size of that gap with the gap between two model generations.

Expert guidance led agents to spend more time and tokens on research. Paranjape reports a statistically significant improvement in prediction error for GPT-6 Astra under the guided setup.

The team also withheld analyst consensus, including redacting mentions from retrieved sources. The best model’s result did not differ significantly with and without consensus; Paranjape suggests it may reconstruct aspects of that view from other information.

Samaya is testing the forecasting capability with customers and studying whether internal data can help. Future research includes examining model reasoning, combining it with human analysis and learning from prior mistakes.

Paranjape points to the company’s public research and FrontierFinance benchmark for further details. The reported study predicts company earnings metrics; it does not establish that the agent can reliably pick stocks or earn risk-adjusted trading returns.