Jonathan Ross on Groq's Inference Chips, AI Economics and Human Agency

Matthew Berman welcomes Jonathan Ross, founder and CEO of inference-chip company Groq, for a conversation about hardware, business and AI's future. Ross previously worked on Google's Tensor Processing Unit. As Groq's CEO, he has a direct commercial interest in the performance and adoption claims he discusses.

Ross says he enjoyed Google and worked on the TPU and at Google X, but came to see a large company's approval process as a constraint. Inside a company, a project may need many people to agree; outside, he says, one investor can fund an ambitious idea. His advice to prospective founders is to consider whether they feel constrained by their current environment.

Berman asks whether Ross left with the Groq architecture already in mind. Ross says he did not even know he would build another chip: after the TPU paper appeared, investors wanted to discuss AI hardware. When he suggested making the software easier to use, an investor offered funding. Ross says the team spent roughly its first six months on the compiler before beginning chip design.

Berman asks why Groq chips have relatively little memory per chip and whether the need for many chips makes deployment expensive. Ross says the relevant measure is not chip count alone but how efficiently the system uses its resources. He describes Groq's system as a large pipeline intended to reduce repeated access to external memory.

Ross compares the system to a factory assembly line: spreading work across a large installation can be expensive to build, yet each job spends only a short time at each stage. He argues that repeated movement of model parameters from external memory can slow other hardware. This is his architectural explanation, not an independently measured comparison of every GPU workload.

Asked whether a company should buy hardware or use a service, Ross recommends starting with GroqCloud's API and discussing on-premises hardware only at very large scale. He says Groq had around 70,000 developers and 17,000–18,000 API keys at that time; these are company-reported adoption figures, not independently audited usage.

Ross says Groq did not plan to rent individual chips and expected eventually to let customers upload models for Groq to run. He argues that automated scheduling can improve utilization, while claiming that GPUs are often only about 25% utilized and that Groq hardware uses about one-tenth the power. Those comparisons are not accompanied by workload, system-boundary or measurement details and should not be treated as universal results.

Berman asks what will constrain AI over the next five to ten years. Ross calls compute the new oil: unlike merely distributing stored information, he argues, generative systems must compute a response to each request. He estimates that deploying a model can cost 10–20 times as much as training it. That ratio depends on usage, model and time horizon; the interview supplies no general supporting calculation.

Asked where entrepreneurs might capture value, Ross says physical infrastructure can be easier to monetize than a model given away freely, but starting a chip company is especially hard because hardware requires a complete software ecosystem and long lead times. He sees potential in less glamorous infrastructure work, while cautioning that competition has grown.

Ross compares AI businesses to the early printing industry: demand for materials may be more predictable, while some publishers can become valuable despite greater uncertainty. Model and application companies may have a high upside but wide variance, he says, and founders should work in an area they understand and enjoy. These are his judgments, not forecasts guaranteed to hold.

Berman is especially interested in agents and asks what fast inference makes possible beyond the speed a person can read. Ross replies that people scan complete pages and quickly refine a query, so latency affects an interactive experience even when output exceeds reading speed. He also cites claimed conversion improvements from a 100-millisecond reduction, without identifying the source or conditions.

Ross says Groq's chat user growth rose from 8,000 to 400,000 new users in a week and repeats the developer and API-key figures. He attributes interest partly to speed and cites interactive speech, search and an 'infinite Wikipedia' that generates a page when a link is clicked. These are his company's figures and examples; generated articles are not necessarily factual.

Berman asks how model builders should optimize for Groq and how the company chooses models. Ross refers to the hardware lottery: prevailing hardware shapes which model architectures receive attention and optimization. He claims a 5–10-fold performance advantage over NVIDIA GPUs in some comparisons, but does not supply models, precision, hardware configurations or test methodology.

Ross says Groq's compiler automates much of the work, while low-latency architectures such as recurrent networks, quantized arithmetic and fast interconnects may offer opportunities for further optimization. He says the company intends to curate a smaller set of interesting models rather than host a repository as broad as Hugging Face.

Asked about hopes and fears, Ross hopes generative AI will encourage curiosity and help people understand one another with more nuance than the incentives of television or social media often allow. He imagines children growing up with tools that invite questions and multiple perspectives. This is a hopeful prediction, not an established social outcome.

Ross likens language models to a telescope that reveals the scale of intelligence, and hopes people will see that scale as something to explore rather than merely fear. He says Galileo invented the telescope, but that historical detail is incorrect: Galileo improved and used an existing invention for astronomy.

Berman asks who controls model weights and bias if a generation relies on AI for information. Ross says Groq's mission is to preserve human agency: his concern is that convenient assistance could gradually become decision-making, from ordering milk to influencing a political choice. He says Groq wants to select models that help users understand options rather than make choices for them, while acknowledging that curation will remain a challenge.