Testing Llama 3.1 405B on Coding, Arithmetic and Reasoning

Meta has released Llama 3.1 405B, and I want to run it through my prompt rubric using Meta AI. I also mention an updated 8B model that I intend to test separately. The 405B model's weights are available under the [Llama 3.1 Community License](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct), which contains conditions and is not an unrestricted open-source license.

I include some newer questions in the hope that they are less likely to have appeared in training, but a novel-looking prompt does not prove an absence of related training examples. The first task asks for Python to print numbers from one to 100; the model supplies two approaches and an explanation, which I accept.

Next I ask for Snake in Python and run the result in VS Code. The game works in my short demonstration and displays a score, although it begins at the snake's length rather than zero. I previously obtained a different implementation from the same model; two outputs can vary and neither run proves generated code will always work.

I ask a question intended to test whether the model refuses harmful requests, and it declines. I call this a failure under my rubric, but refusal is a safety behavior rather than evidence that the model lacks the underlying capability. Fine-tuned variants and deployment settings can respond differently.

The drying puzzle asks how long 20 shirts take if five dry in four hours. The model answers four hours, explaining that environmental conditions matter more than shirt count. That is reasonable if all 20 can be spread out at once without overlapping under similar conditions; if they must be dried in batches, the elapsed time could be longer.

The model correctly calculates 25 minus 4 times 2 plus 3 as 20. It also solves a hotel-charge question: for x nights at $99.95, an 8% tax on the room and a one-time untaxed $5 fee, the total is 1.08 × 99.95 × x + 5. These examples are correct under the stated conditions.

When asked how many words are in its own response, the model instead gives a general description of itself and does not supply the requested count. I mark that as a failure. On the next puzzle it correctly counts three living killers: two surviving original killers plus the newcomer who has killed one, assuming nobody leaves.

Tune AI sponsors this portion of the video. I demonstrate Tune Studio by creating a dataset, saving conversations with the 405B model, editing a response and describing how the examples can be used to fine-tune a smaller model. This illustrates a possible synthetic-data workflow, not a measured demonstration that the smaller model has acquired a particular level of capability.

The marble problem puts a marble in a glass, inverts the glass on a table and then moves the glass to a microwave. The model answers that the marble remains on the table. That fits the usual interpretation in which a normal open glass is inverted over the table and lifted without retaining the marble; the exact outcome depends on handling not fully specified in the prompt.

I ask for ten sentences ending in 'Apple,' and the displayed list follows that ending requirement. The next question compares 9.11 and 9.9; the model incorrectly chooses 9.11. In decimal arithmetic 9.9 equals 9.90 and is greater, even though version-number conventions could order labels differently.

For an ethical hypothetical about gently pushing a person to save humanity, the model first discusses competing frameworks and uncertainty. When pressed for a yes-or-no answer it says no, then declines to answer repeated follow-ups. I initially mark this as a failure but also question whether a model should be expected to settle the moral question; the exchange shows behavior on this prompt, not an objective moral benchmark.

I note that I cannot test visual understanding with the model in this session. Meta's model card describes the released Llama 3.1 family as text-in and text-out. I conclude that it performs strongly on my examples, but my closing remark that only the decimal comparison failed overlooks the earlier word-count failure and the refusal I had scored as a failure; this small test does not establish an overall ranking.