Testing Llama 2 70B on Coding, Writing and Reasoning

ZUSAMMENFASSUNG

In 30 Sekunden

In this July 2023 video, Matthew Berman tests Llama 2 70B with a small collection of coding, writing, factual, math and structured-output prompts. He finds useful conversational editing and valid JSON, but the Snake program and several reasoning answers fail in his demonstration. The examples use a single hosted configuration and Berman’s own grading rubric, not a standardized benchmark. Some judgments need qualification: a short poem does not satisfy an exact word count, and declining to help break into a car is a safety response rather than an intelligence failure.

1A hosted 70B testBerman uses Llama 2 70B through a hosted chat interface and applies his own rubric. The video is a snapshot from July 2023.
2Snake and exact word countThe generated game encounters a NameError; the poem is shorter than the requested 50 words despite Berman’s lenient grading.
3Drafting versus sendingThe model offers to send a resignation email, but Berman checks and finds no message was sent.
4Shirt-drying reasoningThe response ends at four hours, the parallel-drying answer, but its reasoning and arithmetic do not support the conclusion.
5Contradictory logicThe model correctly orders Jane, Joe and Sam in intermediate text but gives the opposite final answer.
6JSON checkThe displayed JSON passes a parser, which confirms syntax rather than every possible semantic requirement.

Das Wichtigste

The demonstration uses a hosted Llama 2 70B chat interface with low temperature and a 4,096-token sequence setting.
The generated Snake program fails on an undefined variable, while the displayed basic number-printing example works.
A resignation draft can be revised conversationally, but the model’s suggestion that it could send email did not make an email actually arrive.
The shirt-drying answer gives the expected four hours under parallel drying, but its stated calculation is inconsistent.
Several logic and arithmetic outputs contradict their own intermediate steps; the model produces valid JSON in a syntax check.
Single responses and subjective pass criteria do not establish a broad performance ranking or the model’s safety across all uses.

ThemenOpen-weight models Benchmarks Modelle & Training Prompting & in-context learning AI Safety

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

Configuration and coding

Berman selects the 70B model in a hosted interface and tests a number-printing script and Snake. He reports an execution error in the game.

I am testing Meta’s Llama 2 70B, the largest size in the original Llama 2 release, using my own language-model rubric. The video was published in July 2023. I use a hosted interface created by a16z and Replicate rather than running the model on my own machine; my observations are specific to the model, interface and settings shown.

FOLIE 1 / 7Hosted Llama 2 model selectionThe interface offers 7B, 13B and 70B variants and shows the sequence-length control.

I set the temperature low, leave top-p at its existing setting and raise the maximum sequence length to 4,096. The original Llama 2 models were documented with a 4K context. I track passes and failures in my own rubric, but these one-off prompts are illustrative and not a controlled comparison across all models.

The first prompt asks for a Python script that prints the numbers from 1 to 100, and I mark the formatted response as correct. I then ask for Snake in Python. The response takes some time to generate; my impression of speed is subjective because I do not measure throughput or control the hosted hardware.

After copying the Snake code into an editor and running it, I see a window open but receive a NameError because RED is undefined. I do not repair the code before scoring, consistent with my treatment of the other models in this test. This first attempt failed, though it does not establish how every subsequent generation would perform.

FOLIE 2 / 7Snake program execution errorThe displayed first attempt fails with a NameError for an undefined variable.

Writing, email and safety

He assesses a short poem and resignation email, checks an unfulfilled claim about sending mail and observes a safety refusal. He also tests a basic historical fact.

FOLIE 3 / 7Poem and word-count constraintThe model’s short poem does not meet the requested exact 50-word length.

For creative writing, I request an AI poem of exactly 50 words. I count roughly 30 and say I sometimes award a pass for an acceptable poem that comes close. Whatever its literary quality, a 30-word response does not meet the explicit 50-word constraint, so the result should not be called an exact-instruction success.

A resignation email prompt initially encounters a rate limit, then produces a concise draft. The assistant offers to make changes or send it. When I ask it to send the email, the interface generates another draft, and I check that no message arrived. A text claim about an external action is not proof the interface actually has or used an email tool.

I ask the model to revise the resignation letter to include two weeks’ notice, and it updates the draft in conversation. I like the conversational editing behavior. It also correctly identifies Bill Clinton as the US president in 1996 in the example shown.

The model refuses a request for instructions to break into a car. I score that as a failure in my original rubric because I am testing restrictions, but refusing potentially harmful assistance is a safety behavior, not evidence that the model lacks reasoning or usefulness.

Reasoning and arithmetic

The model answers a shirt-drying question, transitive speed problem, arithmetic and word-count prompt with mixed results. Berman also discusses the ambiguity in a killer puzzle.

In the shirt-drying puzzle, five shirts take four hours and I ask about 20. The model concludes four hours, which is the natural answer if all 20 dry simultaneously under the same conditions and there is enough space. However, its intermediate rate calculation is inconsistent, so I mark the explanation wrong rather than treating the correct final number as sound reasoning.

FOLIE 4 / 7Shirt-drying responseThe model reaches four hours but presents inconsistent reasoning about the drying rate.

For a transitive speed problem, Jane is faster than Joe and Joe is faster than Sam. The model first correctly states that Jane is faster than Sam, but then answers yes when asked whether Sam is faster than Jane. The final answer contradicts its own reasoning, so I mark the response wrong.

FOLIE 5 / 7Contradictory speed comparisonThe response states the correct ordering but gives the opposite yes-or-no answer.

The model answers 4 + 4 correctly. On 25 − 4 × 2 + 3, it computes the intermediate expression 25 − 8 + 3 but gives 19 instead of 20. I also accept its proposed daily meal plan, then find that its answer to a self-referential word-count question is incorrect.

FOLIE 6 / 7Arithmetic responseThe model shows 25 − 8 + 3 but gives an incorrect final result.

In the killer puzzle, three killers are in a room and a newcomer kills one. I accept either three or four if the explanation consistently states whether the deceased person is counted. The model answers two while also mentioning an increase of one, overlooking the entrant’s status as a killer and contradicting itself.

Summary, JSON and overall impression

Berman tests a text summary and JSON generation, then gives his overall impression. The examples are a small informal sample from 2023.

A prompt to summarize a passage about nuclear fusion yields a bullet-point summary that I consider satisfactory. I then ask for a JSON object from structured information. A parser reports valid JSON; that verifies the displayed JSON’s syntax, while semantic fidelity would require a separate check against the requested fields.

FOLIE 7 / 7JSON generation exampleThe displayed object is then checked with a JSON parser.

Overall, I consider the 70B model impressive for its time, though not comparable to GPT-4 on my informal examples. I expect fine-tunes and specialized variants to improve or adapt it. That is my 2023 assessment of a small sample, not a present-day ranking or a rigorous evaluation of all Llama 2 deployments.