Hassan and Omar El Mghari: When an AI financial forecast is right for the wrong reasons

Hassan El Mghari introduces a joint project with his father Omar: can a language model build a financial forecast for Costco using only public information available before a selected date? They focus on the effect of a membership-fee increase.

Hassan leads developer experience at Together AI and builds open-source AI applications. Omar brings experience in auditing, finance leadership and internal audit; he serves as the domain expert reviewing the engineering work.

Hassan previews the result: the forecasts were close to the reported outcome, with an average error of about 4.4%. Omar’s review revealed that the close result did not mean the underlying assumptions were sound.

Costco sells large volumes of goods at relatively low margins, making membership fees important to operating profit. The speakers use the 2024 increase in standard and executive membership prices as the event to forecast.

The team chooses a cutoff before the announced change and gathers the information available up to that point. They ask how a fee increase would affect the financial model over the following two years.

Newer models already answered a check about the 2024 increase, suggesting that they had encountered the event. The team therefore selected older open models that did not answer that check; the check reduces, but cannot prove the absence of, knowledge leakage.

They collect 214 public filings and turn them into roughly 3,000 timestamped facts. Facts after the cutoff are filtered out before the model builds the forecast.

The language model generates structured assumptions and formulas rather than performing all the arithmetic in a single response. The resulting model is then evaluated separately, making its inputs and calculations easier to inspect.

For example, the model represents the recognition of annual membership fees over the membership period. It also estimates unknown quantities such as executive-tier share and the proportion of members in affected geographies.

The resulting spreadsheet includes assumptions, 14 generated formulas and numbers from filings. The team tests properties such as avoiding circular dependencies and whether a fee increase raises revenue.

They run 16 independent attempts and return validation errors for up to two correction attempts. Five generated models pass the checks; Hassan says the 16 model runs cost about 19 cents.

Hassan asks the resulting financial model to estimate the impact of the fee increase. One output projects approximately $560 million in additional profit over two years; the calculation runs quickly once the formulas are built.

The speakers compare the forecasts with reported results and say the average of the five builds was about 4.4% off. Hassan initially viewed that as encouraging, but a close numerical result does not by itself validate the model.

Omar starts his review with the assumptions. He notices that the executive-tier share moves the result substantially, while the US-and-Canada share appears to have no effect even though it should be relevant to the fee change.

The model used an executive-tier assumption of 15% and cited a statement about increased penetration without a supporting numeric value. Omar says the relevant filings already disclosed a figure of about 45%, which the model missed.

The team identifies two errors that partly offset each other. The model applies the increase to too broad a geographic membership base, overstating the affected population, while using too low an executive-tier share.

Omar describes a diagnostic technique: correct one assumption at a time and observe whether the result gets worse. In his example, replacing the 15% assumption with 45% exposes a much larger error rather than an improvement.

He compares the engineering checks with three lines of defense in auditing: management controls, internal audit and independent external review. In this demonstration, he supplies professional judgment as an additional layer.

Hassan says some of Omar’s accounting checks can be converted into programmatic validation. The broader lesson is to ask a subject-matter expert to examine assumptions, inputs and outputs before treating a close forecast as reliable.

The speakers conclude that domain checks and human review complement automated tests. Their single retrospective case illustrates a failure mode, not evidence that this workflow will reliably forecast other companies or investment returns.