Llama 4 Reactions: Benchmarks, Long Context and Early Tests

Meta has released Llama 4 Scout and Maverick, and the reaction is coming in. In this April 2025 update, I look at independent benchmarks, pricing, comments from the industry, the long context window, and a few early examples of what these models can and cannot do.

Meta also discussed a much larger model, Behemoth, as a teacher for the smaller releases. Behemoth was a preview, not an available set of weights. Scout and Maverick were released under Meta's Llama 4 Community License, so their weights can be used subject to that license rather than without restrictions.

The models use a mixture-of-experts architecture: only part of the network is active for each token. Maverick has 402 billion total parameters, with 17 billion active, while Scout has 109 billion total and 17 billion active. That distinction helps explain why a large total parameter count does not translate directly into the computation of a dense model of the same size.

Artificial Analysis put Scout at 36 and Maverick at 49 on the intelligence index shown here. Maverick was near GPT-4o at 50 and Claude 3.7 Sonnet without extended thinking at 48, with DeepSeek V3 at 53. These are scores in that particular aggregate and its testing conditions, not a guarantee that the models are interchangeable on every task.

For an open-weight release to approach prominent proprietary models on this measure is a big deal. People can choose their hosting setup and build on available weights. I am especially interested in how much capability the new models deliver for their inference cost, although a benchmark ranking is only a starting point for deciding which model fits an application.

The chart compares intelligence with price and speed across a crowded field. Scout is the less capable but cheaper option; Maverick trades more resources for a higher index score. The model, provider, hardware, and workload all matter when turning such charts into a practical deployment decision.

The mixture-of-experts design makes inference more efficient than a dense model activating all its parameters for every token. But inactive experts still affect storage and deployment, and real throughput also depends on memory bandwidth and the serving system. The active parameter count alone does not determine how fast a model will run.

The historical pricing shown for Scout was approximately $0.15 per million input tokens and $0.40 per million output tokens; Maverick was around $0.24 and $0.77 respectively. These were snapshot figures from April 2025, not promises of current prices, and rates differ by provider and model configuration.

The video includes a paid sponsorship from Box, which presents its enterprise content platform and AI features for working with business documents. The sponsor describes tools for finding, extracting, and using information across company content; those are Box's product claims rather than an independent test in this video.

Major industry figures quickly commented on the launch. Microsoft highlighted availability through Azure AI Foundry, and Google CEO Sundar Pichai congratulated Meta. Other reactions from Dell, Hugging Face, David Sacks, and Reid Hoffman show the level of attention the release received across infrastructure, policy, and the model ecosystem.

These reactions do not independently establish the models' quality, but they do show how quickly an open-weight family can become part of a wider cloud and developer ecosystem. Different companies have their own reasons to welcome a new model: hosting demand, user choice, and more competition all play a part.

Scout's advertised context window reaches 10 million tokens. Meta says its training used shorter contexts and describes methods for extending the usable window; the advertised maximum should not be mistaken for proof that every detail is reliably retrieved or reasoned over at that length. The 10-million figure applies to Scout, not automatically to every Llama 4 model.

A huge context window raises the question of whether we still need retrieval-augmented generation. I can imagine placing a large corpus directly in the prompt instead of building a retrieval pipeline. But retrieval can still reduce the amount of material processed, select relevant evidence, and bring in fresh information; the best choice depends on quality, latency, and cost.

I also look at examples posted by people trying to bypass the model's safety controls. An individual jailbreak demonstration can expose a weakness, but it is not a comprehensive safety evaluation, and the way a model is hosted or configured matters. The broader question is how developers will assess and mitigate harmful behavior when weights are widely available.

There is a tension between letting users run a model on their own systems and maintaining safeguards in deployed products. Neither a permissive example nor a single refusal tells us how a model behaves across all requests. That needs systematic testing, not just screenshots from social media.

Groq also offered a hosted version and a way to prompt the model to think through a problem. Prompting a model to produce reasoning is a useful interface experiment, but a prompt or a visible reasoning trace is not evidence that the underlying model was trained as a dedicated reasoning model.

One early example criticizes an experimental Maverick chat variant for giving an overly long response to a straightforward question about a quotation. The screenshot also shows a factual mistake: it attributes the line to Aliens when the prompt refers to Castlevania: Symphony of the Night. This example is from an experimental chat-tuned variant, so it should not automatically be treated as a test of the released checkpoint.

That example illustrates why a model's style matters as well as its aggregate score. A quick factual question often calls for a direct, accurate answer, not a long detour. We need more than one anecdote to judge a model overall, but examples like this can reveal a failure mode worth testing.

Exo Labs showed a cluster of four Mac Studios as a way to distribute inference. Pooling machines is an interesting approach for people who want to work with larger open-weight models locally, though networking, available memory, quantization, and software support will determine what actually runs and how quickly.

The performance figures discussed for the cluster were estimates associated with experimental parallelization, not measurements established by this video. Estimates for Behemoth in particular cannot establish actual local performance: Meta had previewed that model but had not released its weights. A real benchmark requires the available model, the configuration, and a measured workload.

A critic questions whether the ten-million-token claim reflects useful performance rather than just an accepted input length. That is an important distinction. A context limit tells you how much text a system can accept; it does not by itself tell you whether the model will find a small fact or reason accurately across a very long document.

Finally, I look at a coding prompt for a ball bouncing inside a rotating hexagon. The displayed attempt does not produce the intended working simulation, an example of the gap that can appear between a headline benchmark and an everyday programming task. I also note that results may vary with prompting and later attempts.

My overall reaction is that the release is an exciting step for accessible models, particularly given the price and the range of deployment options. The reactions and early demos are interesting, but we should judge context, safety, and coding ability through repeatable tests as well as hands-on use.