Jonathan Ross on Groq, Inference Economics and the AI Infrastructure Race

Groq founder and CEO Jonathan Ross argues that the coming demand for AI inference will require a different approach to chips and infrastructure. In this February 2025 interview, he tells Harry Stebbings about model scaling, Groq’s architecture, infrastructure financing and AI competition. Groq sells inference hardware and services, so its CEO has a direct commercial interest in the comparisons he makes.

Ross describes scaling laws as relationships between model size, training data and performance. He says data quality matters as well as quantity, and that training approaches can waste effort by presenting simple and difficult examples without an effective progression.

His proposed feedback loop uses a model to produce synthetic examples, filters poor outputs and trains a better model on the selected data. He expects iterations of that process to improve training efficiency. This is his explanation and prediction, not proof that synthetic data is always higher quality or free of error.

He uses sorting algorithms and multiplication to illustrate that some problems require intermediate steps. More training can make a model more intuitive on familiar examples, he argues, while test-time computation supplies additional reasoning steps for harder tasks.

Ross expects better training and runtime reasoning to work together. Asked about constraints, he says compute, data and algorithms all matter: more compute can sometimes compensate for a shortage of improvement elsewhere, but it is not the only factor.

He attributes part of DeepSeek’s progress to algorithmic improvements that make generating and checking training data easier. This is his simplified account of the work, not a full technical evaluation of the model or its training costs.

Ross says that at Google, serving models ultimately used ten to twenty times as much compute as training them. That is an account of his experience, not a universal ratio for every model, deployment or time period. He also disagrees with the market reaction that sent Nvidia shares lower after DeepSeek news.

He advises founders to position for future capability changes rather than assume today’s limitations will last. Medical and legal applications, for example, face challenges from hallucinations; a reduction in error could change what is possible. Groq itself spent years betting that scaled inference would become important.

There was doubt during those years, Ross says, but he remained motivated by a desire to preserve human agency as AI becomes more powerful. At one point the company nearly ran out of money and asked employees to exchange some salary for equity through a program it called Groq Bonds.

He recalls roughly 80% of staff participating and about half moving to the statutory minimum salary. When the first part of a later $300 million round arrived, he says, the company had less cash left than the program had saved. He presents the episode as an example of communicating vulnerability to a team.

Ross compares inference capacity to employees performing work, while noting that models can be deployed consistently once trained. His analogy suggests that companies may substitute some computing expenditure for hiring; it does not establish that a chip or model is equivalent to a human worker.

He says Groq grew from roughly 640 chips in production at the start of 2024 to more than 40,000 at year-end, and aimed for over two million in 2025. The latter is a target reported in the interview, not an achieved deployment.

Asked about supply, Ross says Groq’s next scale goal would require substantial foundry capacity. He describes high-bandwidth memory and advanced packaging as important constraints for GPU production, naming SK hynix, Samsung and Micron as memory suppliers.

His account of Groq’s architecture is that model parameters are held on chips and computation moves through a pipeline across many chips, rather than repeatedly reading external memory. He contrasts that with a GPU setup that may use fewer devices and external high-bandwidth memory.

Ross claims this approach uses about one third as much energy per generated token for the comparison he has in mind, despite a larger system footprint. He attributes the difference to the energy cost of moving data over longer, wider connections. The interview does not supply a benchmark, model, batch size or hardware configuration needed to generalize the claim.

He argues that data-center batching can be more energy efficient per task than small edge devices, even when an individual edge device draws less power. The result depends on workload and utilization; the interview gives an illustrative analogy rather than a comprehensive measurement.

Ross sees GPUs as strong training hardware and says more affordable inference could increase demand for training as well. Groq has also experimented with combining its chips and GPUs for parts of a model, which he says could improve the economics of existing GPU deployments.

He reports a Saudi deployment reaching first production tokens in 51 days from contract. He attributes some of the speed to a simpler interconnect and predictable execution. This is a company-reported example, not a standard deployment time for all customers.

Ross argues that chip marketing often emphasizes specifications that do not directly answer a customer’s question. He prefers tokens per dollar and tokens per watt, alongside performance, to isolated figures such as theoretical operations per second.

He questions the way a 30-times GPU performance claim was presented and says Groq replied that it was still faster. Neither claim can be assessed from the interview alone: model, workload, latency, throughput, power, price and measurement conditions all affect the result.

He describes Nvidia as especially strong in training and Groq as focused on low-cost, fast inference. Although both can serve inference customers, he says some customers buy both. This framing reflects Groq’s competitive positioning, not an independent conclusion that either company owns a market.

Ross claims Groq inference costs more than five times less in his comparison, and cites roughly one-third the energy per token. He offers a three-year cost illustration in which operating expenses and capital expenditure contribute differently. The necessary hardware, utilization and pricing assumptions are not provided, so the numbers should not be treated as a universal benchmark.

He says Groq’s limited deployment scale, rather than only cost or speed, constrained its market share. He believes avoiding external high-bandwidth memory makes scaling easier. That is a company executive’s supply-chain view, not proof that there are no other capacity limits.

Ross argues Nvidia could continue selling GPUs for training while Groq addresses high-volume inference. He says some Groq deals have about a 20% upfront margin and additional potential later; that figure relates to the deal structure he describes, not a comparable company-wide margin.

Under that model, he says, partners supply capital for deployment and receive a return and a share of proceeds, after which the sharing changes. Groq’s ability to expand depends on deployment capacity as well as capital. Actual returns depend on the terms and performance of each deal.

Ross worries that the same power demand can be counted multiple times as hyperscalers approach different data-center developers. He predicts some near-term projects may not materialize, followed by a more serious power constraint in a few years. His estimates of capacity and timing are forecasts, not a verified inventory.

He distinguishes a proposed data-center site from a working facility: reliable power, generators, water and other equipment take planning and time. He says buyers can encounter projects promoted as ready before those requirements are secured.

The financing mismatch spans different useful lives. Ross illustrates model training over months, chips over several years, data centers over roughly a decade or more, and power plants over longer periods. Each participant wants commitments that match its own investment horizon.

He says Groq partnered with Aramco and a Saudi entity able to fund long-term infrastructure. He clarifies that a reported $1.5 billion deal was revenue in his description, not a $1.5 billion fundraising round. The amount and timing are his characterization of the agreement, not independently verified cash received or recognized revenue.

In his account, the partner supplies capital to deploy chips in data centers, while Groq repays it from the resulting business and shares upside. He cites an earlier deployment of about 19,000 chips in 51 days. Specific commercial terms and ultimate returns are not established in the interview.

Ross claims Groq had positive contribution margin on open models at the time and contrasts that with competitors subsidizing prices. Contribution margin is not the same as company-wide net profit, and the interview does not provide enough detail to verify the comparison.

He says Groq also works with proprietary model providers, including a voice-model demonstration with PlayAI. He wants prices to fall while maintaining margins, betting that lower costs will drive more use.

His stated ambition is to provide at least half of the world’s AI inference compute by the end of 2027. He calls scale more important than near-term profit. This is an aggressive corporate goal, not evidence of future market share.

Stebbings asks about large announced capital spending by Meta, Google and Microsoft. Ross notes that such figures include chips and systems as well as physical data centers. He believes AI creates a substantial opportunity, though the returns on individual projects remain uncertain.

He expects a highly uneven distribution of value and worries about a single entity becoming dominant. His account of startup development moves from solving an unmet problem, to competing through marketing, to building durable advantages and responding to the next disruption.

Ross expects many individual AI investments to lose money but believes aggregate returns could exceed aggregate investment. He warns that speculative capital can flow into products with only superficial AI branding. Both the losses and overall returns he describes are predictions.

Using Keynes’s beauty-contest analogy, he argues that investors sometimes bet on what others will fund rather than on actual customer value. With several well-funded rivals in AI, he expects product quality to matter, but says abundant financing can also split scarce talent among competitors.

Groq does not aim to win every salary bidding war, Ross says. It tries to attract people who want to work on the mission and value potential equity. That is his company’s hiring philosophy, not a guarantee of employee loyalty or returns.

He says Groq decided not to train its own foundation models because providers entrust the company with their weights and do not want a direct competitor. It may fine-tune for specific cases. He also says Groq does not persistently store user queries for training, although applications built on top can have their own data practices.

Ross says the company tries to automate work so headcount grows more slowly than customers or deployments. At the time, he reports roughly 300 employees building chips, networking, software and cloud infrastructure. He values small teams and talent density.

He calls each tripling of a metric a “problem unit”: scaling chips, customers and employees simultaneously creates different sets of problems for limited management bandwidth. His goal is to design systems that can grow without requiring proportional increases in staff.

On China, Ross credits DeepSeek with innovations but alleges it distilled an OpenAI model. The interview does not establish that allegation. His comments about access to chips, competitive strategy and Chinese model restrictions are his views and speculation, not verified findings.

He argues that China may offset some hardware efficiency limits with scale and power, while export expansion could face different energy constraints. He identifies openness and censorship as questions that could affect model development; the interview does not establish how those choices will play out.

Discussing Europe, Ross sees strong talent but worries that it leaves for other markets. He proposes dense entrepreneurial communities, easier movement between employers and incentives for risk-taking. These are policy proposals, not an assessment of their legal or economic effects.

He says the aim is to preserve human agency in an age of AI abundance. Drawing on his family’s financial ups and downs, he worries that a life without material pressure could still leave people searching for purpose and fulfillment.

Ross says he looks for employees who treat an unrealized improvement as a loss, not merely an optional gain. In a world where AI makes prototypes easier to create, he expects people will need to polish products to stand out.

He favors delegation over telling every employee how to perform a task. To align the team, Groq uses a challenge coin bearing its goal of 25 million tokens per second. The coin represents an internal target, not a verified public performance result.

If failure were impossible, he says, he would order far more chips, but real expansion requires care because every increase in scale brings new problems. He expects Nvidia may grow over the next decade but cannot give a reliable multiple.

Ross speculates that AI could help produce a sudden advance in slowing aging within ten years, explicitly conditional on such an advance being possible. He does not establish that it is possible or predict a particular treatment. Stebbings shares his own hope for medical progress.

After seven years without product-market fit, Ross says finding demand changed the emotional experience of running Groq. He describes founder happiness as a shift from living for a hoped-for future to seeing results in the present.

He predicts influential AI companies will emerge around reducing hallucinations, breaking down agentic tasks, invention and delegated decisions. He sees these as possible stages and opportunities, not solved capabilities or named winners.

Hallucination risk matters differently across applications, he argues: a user may check sources for a low-stakes search, while medical diagnosis and law require much greater reliability. He says early teams can still position themselves for future improvements, but the risk depends on the use case.

He says the Saudi deal could produce more revenue over time if capacity is deployed and sold, and discusses potential future hardware capacity. These are conditional projections, not booked future sales or profits.

Ross ends with the hope that natural-language tools will make software creation accessible to far more people. He imagines people who cannot write code building applications by speaking; that is a vision of potential entrepreneurship, not evidence that access alone removes all barriers.