A Fast Groq Inference Demo Using Mixtral

SUMMARY

In 30 seconds

In this February 2024 short, Matthew Berman tries Groq’s custom-hardware inference service with Mixtral 8x7B. One displayed Python snake-game response runs at approximately 418.55 tokens per second, according to the interface. This is one historical demonstration, not a controlled comparison of model quality, latency or provider throughput. The video contains no disclosed sponsor.

1Inference speedBerman says he heard Groq could reach around 500 tokens per second; speed depends on the model and workload.
2Snake-game requestThe interface displays approximately 418.55 tokens per second for one generated Python response.
3Custom hardwareBerman attributes the service’s speed to custom hardware and discusses its founder’s Google TPU background.

Takeaways

Berman’s displayed Mixtral response reports 418.55 tokens per second in one example.
His reference to around 500 tokens per second is a separate claim, not the measured speed of this response.
Output token rate, time to first token and total response latency are different measures.
The video does not test whether the generated snake game runs correctly.
Groq’s Mixtral model ID was later deprecated, and historical free access should not be assumed to persist.

TopicsInference AI cost & performance Semiconductors

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

A speed demonstration

Berman explains why inference speed matters to applications and requests a snake game in Python. He reads the displayed output rate.

I have just found Groq and am impressed by reports of inference around 500 tokens per second. Faster output could make language-model applications feel more responsive, although the rate depends on the model, workload and measurement. The short was published in February 2024 and contains no disclosed sponsor.

I ask the service to write a snake game in Python. I say the response takes about two and a half seconds; the screen displays 418.55 tokens per second for this example. That is one interface-reported generation rate, separate from time to first token or end-to-end latency; the video does not run or verify the game.

Hardware and access

Berman mentions Groq’s custom hardware and founder. His free-access statement reflects the video’s 2024 context.

SLIDE 1 / 1Displayed token generation rateThe interface shows 418.55 tokens per second for the demonstrated response.

Groq uses custom inference hardware. Its founder Jonathan Ross previously worked on Google’s TPU, which was developed by a team. I describe the product favorably and say the Mixtral 8x7B 32K service can be tried free at the time. Availability and terms can change: Groq later [deprecated that model ID](https://console.groq.com/docs/deprecations), so the 2024 example is not a current service guarantee.