Llama 3 70B on Groq: Fast Inference and an Informal Prompt Test

In April 2024, Matthew Berman tests Meta’s Llama 3 70B through Groq’s inference service, focusing on both response speed and his personal prompt rubric. He compares the results with an earlier Meta AI session whose model size he says was uncertain. Different runs and interfaces do not by themselves establish that one model version is generally superior.

A simple request for Python code printing 1 through 100 returns two approaches, with the interface showing roughly 300 tokens per second. Berman next requests a Python Snake game and observes about 254 tokens per second in that run. These are displayed rates for particular responses, not universal throughput guarantees.

The new Snake implementation uses Pygame. Berman runs it, moves the snake, sees a score and a game-over menu, and successfully restarts. He prefers this result to a terminal-based earlier attempt, though a short play session cannot verify all possible game behavior.

Berman probes refusal behavior with a request for help breaking into a vehicle and a reframed fictional version. The model refuses both. Refusal of these requests is an appropriate safety response; the two examples are not a complete safety evaluation.

The shirt-drying puzzle asks about drying 20 shirts when five take four hours. The model answers four hours assuming all can be laid out under comparable conditions. Berman likes that the assumption is explicit; limited space or changing weather could change the practical result.

The model correctly answers a relative-speed ordering question and basic arithmetic, including 25 − 4 × 2 + 3 = 20. Berman then turns to more difficult problems and compares results with his prior Meta AI session.

For the displayed equation 2a − 1 = 4y, the model gives y = (2a − 1)/4. Berman marks this wrong against his expected answer, but the model’s expression follows directly from the equation shown. His expected answer therefore does not support the fail for that displayed prompt.

A second problem asks for a coefficient in a cubic with three stated x-intercepts. Berman expects 8 and reports that the model returns other values on multiple runs. The displayed prompt loses mathematical formatting, so its intended powers and one root are ambiguous; the video does not establish a reliable answer key from that rendering.

When asked to count the words in its own response, the model gives a count that Berman finds inconsistent with the text and marks it wrong. He speculates about an exposed system message, but the output alone does not establish that a system message was revealed.

In the three-killers puzzle the model counts two surviving original killers and the newcomer, giving three living killers. It also returns a JSON representation of three named people and their ages, which Berman considers correct in this run.

The inverted-cup question asks where a marble is after a cup is turned upside down on a table and moved without changing orientation. Under the ordinary assumption that the cup alone is lifted, the marble remains on the table. Berman observes both correct and incorrect answers across repetitions.

He notices that repeating the prompt without clearing the chat sometimes produces a correct answer after an incorrect first answer, whereas a new chat can behave differently. He wonders whether repeated generation and reflection could help, but these informal trials do not isolate the cause: the retained conversation changes the context as well as producing a new sample.

Berman gives the marble test a pass because he saw correct responses, including one off camera, while acknowledging that this may be lenient. He repeats the cubic problem in the same conversation and still receives inconsistent wrong answers by his key. One successful retry does not establish reliable first-attempt performance.

The model answers a belief-tracking prompt by saying John expects the ball in the box and Mark expects it in the basket. For ten sentences ending in “apple,” Berman changes the prompt by adding quotation marks; the first output meets nine of ten requirements, while a repeat produces ten according to his check.

The final digging problem asks how long 50 people need for a single hole if one person takes five hours. Berman accepts six minutes on the first attempt, then sees a different answer on repetition. Six minutes follows only from ideal linear scaling with all 50 people working simultaneously at the same rate; the physical feasibility and actual time are unspecified.

Berman concludes that the hosted model performed well in his tests and imagines pairing fast inference with agent frameworks such as AutoGen or CrewAI. Faster generation could make repeated attempts cheaper in time, but his demonstrations do not show a reliable method for selecting the best answer or establish autonomous task success.