Meta Llama 3 Launch: Benchmarks, a Snake Test and the Meta AI Assistant

Llama 3 day is here. Meta has just announced the third generation of its Llama family, and I want to look at the announcement, what is new and how the models perform. I plan to follow up with separate coding, testing and fine-tuning videos. This review reflects the April 18, 2024 launch.

I see the earlier Llama models as important to the growth of locally run language models. Meta initially offers Llama 3 in 8-billion- and 70-billion-parameter sizes, both pretrained and instruction-tuned. I notice the absence of a middle-sized model and guess one may arrive, but that is speculation, not part of the initial release.

Meta also offers a Meta AI chat interface. The announcement discusses applications such as agents, and I find that encouraging: an agent can involve more than a single prompt. I will try a quick example here and leave the full testing rubric for another video.

I ask Meta AI to write the game Snake in Python. It responds quickly with code using the curses library, which appears to target a terminal. I copy and run it; a window and score appear, and when the snake runs into itself, the game ends. I regard this as a pass, though I do not know whether the interface used the 8B or 70B model.

Meta says Llama 3 improves language understanding, instruction following, coding and multistep tasks, and that its post-training reduces false refusals. Those are the company’s claims. I am especially interested in whether multistep behavior can help with agents and expect to try it with CrewAI; the Snake demo alone does not establish those broader capabilities.

Meta says it pretrained the models on more than 15 trillion tokens using two custom-built 24,000-GPU clusters. The company says the data set was seven times larger than Llama 2’s and contained four times as much code. The initial models use an 8,192-token context; I compare this with larger advertised contexts from other 2024 models and hope future fine-tunes extend it.

The release charts compare Llama 3 8B with Gemma 7B and Mistral 7B Instruct on tests including MMLU, GPQA, HumanEval, GSM8K and MATH. I am excited by the reported improvements, particularly in code and math. These are Meta’s reported scores for specific models and test settings, not proof that one model wins every task or remains the best later.

For the 70B model, Meta’s chart includes Gemini 1.5 Pro and Claude 3 Sonnet. I notice that Sonnet, rather than Claude 3 Opus, is the Claude comparison. The chart reports a strong HumanEval code-generation result for Llama 3 70B; I wonder whether the coding improvement will also help function calling and agents, but that connection needs its own tests.

The reported 70B results vary by benchmark, including a math score below Gemini 1.5 Pro in the comparison. That is a useful reminder that a model can lead on one measurement and trail on another, and benchmark configurations and chosen competitors matter.

Meta also updates its responsible-use guidance and introduces Llama Guard 2, Code Shield and CyberSec Eval 2. I look at a diagram that places model and system-level safeguards alongside transparency and evaluation. These tools help developers assess and manage risks, but they do not make an application automatically safe.

The cyber evaluation covers areas including insecure coding practices, attacker helpfulness, code-interpreter abuse, offensive cybersecurity and susceptibility to prompt injection. I am particularly interested in the last category because jailbreaks have affected both proprietary and accessible models. I leave the associated paper for a possible later discussion.

Meta AI is Meta’s assistant interface, which at the time is offered free in the settings I discuss. Meta calls it a leading free assistant; that is a company claim, not an independently established ranking. I think Meta’s accessible model releases could increase competitive pressure and lower prices for developers and users, although that is my prediction.

I expect value to accrue at different parts of the AI stack: hardware, infrastructure such as orchestration and observability, and applications built on or enhanced with AI. I am more skeptical of durable value in the model layer as models become commoditized. This is a view of how the market may develop, not a settled outcome.

Meta describes using its assistant in feeds, chats and search to obtain information without leaving its apps. I would like it to incorporate relevant context from a user’s activities and conversations rather than always acting as a stateless engine. That is a proposed direction, not a claim that the assistant already has unrestricted access to all such information.

I try the image feature by asking for a robotic llama. I like the result, though I think it is not as high quality as the images I typically get from DALL-E. I notice an “Imagine” watermark on the displayed image. This is one subjective example, not a comprehensive image-quality or watermarking test.

Meta says it is rolling out Meta AI across Facebook, Instagram, WhatsApp and Messenger and expanding English-language availability to countries including Australia, Canada, Ghana, Jamaica, Malawi, New Zealand, Nigeria, Pakistan, Singapore, South Africa, Uganda, Zambia and Zimbabwe. Availability and features described here are those announced in April 2024.

The examples include asking for a restaurant with sunset views and vegan options, finding concerts for a weekend, using the assistant inside Messenger, and searching for a recipe video. Meta also shows it in the feed. I see this as evidence of the company’s investment in AI, though a demonstration alone does not establish that every answer will be correct.

Another example creates an image of a bird and then animates it into a shareable GIF. I still prefer the image quality of DALL-E or Midjourney in this quick impression, but the feature is free in the context I am discussing. Quality comparisons and product terms can change.

I open the Llama 3 GitHub repository and model card and express uncertainty about whether the original weights are available. Meta’s repository says trained weights and a tokenizer can be downloaded after accepting the Llama 3 Community License. Its terms include an acceptable-use policy and an additional license requirement for qualifying services exceeding 700 million monthly active users; availability does not make use unrestricted.

I return to the training-token figure and the 8B and 70B model sizes, then review Meta’s comparisons with Llama 2. The charts show improvements on the listed benchmarks, within their specified settings. I am excited to test the models more thoroughly in a separate video rather than treating the launch charts or one Snake prompt as a complete evaluation.