Inside Groq’s LPU: Compilers, Chip Networks and Fast Inference

Matthew Berman interviews Andrew Ling, Groq’s VP of compiler software, and Igor Arsovski, its chief architect and fellow. They discuss how Groq designs its language processing units and software for AI inference. Groq sponsored the video, and the technical and comparative claims below are the guests’ explanations, not independent measurements.

Ling studied computer architecture, machine learning, compilers and FPGAs at the University of Toronto, then spent more than a decade building compilers at semiconductor companies. As his work shifted toward machine learning, colleagues and industry contacts introduced him to Groq.

Arsovski also attended the University of Toronto and worked on custom networking chips at IBM Microelectronics and GlobalFoundries. He met Groq founder Jonathan Ross when the company was very small, later worked at Marvell and on Google TPU physical design, and then joined Groq.

Berman asks how conventional inference hardware works. Arsovski describes a high-end GPU with a large compute die and nearby stacks of high-bandwidth memory (HBM). The memory holds much more data than on-chip storage, but accessing it has different bandwidth and latency characteristics.

Asked about process-node labels, Arsovski says nanometer names historically referred to tiny feature dimensions but have become less direct measures of a specific physical size. He distinguishes GPU memory capacity from the bandwidth and placement of the memory technology.

The first Groq LPU shown uses a 14 nm process, older than the GPU process in the comparison. Arsovski says that choice reflected when the chip was developed and a desire for a North American supply chain: fabrication was in Malta, New York, and packaging in Bromont, Canada.

Berman asks what the guests mean by deterministic execution. Arsovski explains that a processor can receive data quickly from a nearby cache or wait longer for external memory; timing variation can complicate coordination when many cores and chips depend on one another. This discussion concerns execution timing, not a guarantee that model outputs are always identical.

Ling says programming and scheduling variable hardware can require conservative assumptions and extensive human optimization. He describes specialized mathematical libraries and hand-tuned kernels as ways industry extracts performance, rather than claiming all GPU software is written by hand.

According to Ling, Ross began by considering how to decompose machine-learning workloads and then designed hardware to execute those operations. Arsovski says Groq’s hardware and software teams work closely together across the stack, with compiler engineers involved in hardware decisions and vice versa.

Arsovski describes the LPU as a regular chip without the external HBM stacks shown alongside the comparison GPU. He says software can plan which functional units and memories are active at particular times. The guests attribute performance to the whole design, including the compiler and system, rather than the manufacturing node alone.

Ling says predictable behavior allows the compiler to schedule a larger problem across many chips as though they were one coordinated substrate. The guests argue that a new company could start with a different architecture, while established vendors had substantial investments in existing hardware and software ecosystems.

The conversation moves from a single chip to the network connecting processors. Arsovski says conventional routing can introduce congestion and variable delays, making it difficult to schedule a multi-chip workload. Berman restates the problem as independent processors needing to communicate while each can finish at a different time.

Arsovski says Groq’s chips combine acceleration and switching, communicating directly within and across groups instead of relying on separate top-of-rack switches in the described system. The software schedules both computation and communication, so the network and chips can be treated as a coordinated whole.

Ling explains that the compiler and silicon are a platform: Groq uses some common compiler infrastructure but builds a scheduling approach specific to its architecture. If execution time is predictable, the compiler can plan when to send data across chip links.

Arsovski compares scheduled data transfers to cars given exact routes and departure times, avoiding intersections where they must wait. The analogy illustrates the intended reduction in congestion and better link utilization; actual performance depends on the workload and the system.

Asked about applications beyond language models, Ling mentions workloads with substantial memory bandwidth needs, including recurrent models, graph neural networks and possible drug-discovery applications. These are areas the team sees as promising rather than independently verified superiority claims.

Berman asks whether this architecture could eventually appear in consumer devices. Arsovski says the regular structure could be tiled into a smaller design, a chiplet or a standalone chip. The guests present this as a design possibility, not an announced consumer product.

A phone demonstration produces a quick spoken response, but Berman points out that the phone is contacting a server. The guests distinguish that remote inference from running a model fully on the device. They suggest smaller, specialized or distilled models could be more practical locally than the largest server-scale models.

Ling says bringing a new model to Groq may require adapting its PyTorch description to remove vendor-specific primitives before passing it through Groq’s compiler and software stack. The team also tests proprietary customer models. The supported-model list in this 2024 conversation is historical.

Arsovski gives a high-level overview of lithography: layers are patterned on silicon using masks, photoresist and etching. He describes advanced fabrication techniques and says Groq’s first chip used the foundry’s normal logic process, rather than requiring a unique fabrication process.

The regular design, Arsovski argues, can simplify scaling and make dense structures easier to arrange. He says relatively little of Groq’s die is devoted to instruction control compared with the illustrative CPU and GPU designs he discusses. Those percentages are company-side estimates and should not be generalized to all chips.

Berman asks how Groq became widely noticed after years of work. Ling credits the decision to make the technology accessible; Arsovski says open-weight Llama models helped the company demonstrate the hardware’s capabilities. The team describes the excitement of seeing a long-running project reach a wider audience.

Berman is interested in using low-latency inference with agents and coding. Ling suggests that faster calls can leave time for multiple drafts, evaluation and revision before a response is shown. He says Groq had proofs of concept but had not exposed such a commercial feature at the time; extra iterations do not guarantee a better or more factual answer.

Arsovski adds that developers can already experiment with successive prompts and evaluate whether the result improves. Berman thanks the guests and closes the interview.