Mistral Large: A 2024 Review of Coding, Reasoning and Safety Responses

SUMMARY

In 30 seconds

Matthew Berman examines the February 2024 Mistral Large release, its vendor-reported benchmarks, historical API prices and proprietary deployment options. In an informal Le Chat session he tests coding, moderation and reasoning. Results are mixed, context-dependent and not a current ranking or comprehensive safety evaluation.

1MMLU comparisonMistral’s chart reports 81.2% for Large and 86.4% for GPT-4 on MMLU.
2Launch pricingThe video shows $8 per million input tokens and $24 per million output tokens for Mistral Large in 2024.
3Coding demonstrationA simple terminal Snake game runs initially but errors when it hits a wall.
4Reasoning examplesThe drying puzzle is answered under parallel and batch assumptions, while the self-referential word count fails.

Takeaways

Mistral Large was a proprietary model with a 32k context at its February 2024 launch.
The 81.2% versus 86.4% chart reports MMLU results, not an average of benchmarks.
The shown API prices are historical and specific to the compared models and rates.
A simple Snake demo partly worked but errored on a wall collision; other reasoning examples also had limits.
A handful of moderation prompts cannot establish a model’s overall safety.

TopicsModels & training Benchmarks AI Safety Coding agents

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Launch features, benchmarks and prices

Berman reviews Mistral’s 2024 announcement, deployment options and benchmark comparisons. Pricing is historical.

Mistral AI has released Mistral Large, and in this February 2024 review I examine the company's claims and try my usual prompts. I previously liked Mixtral and Mistral Medium, so I am interested in whether the new proprietary flagship can handle reasoning and code. My tests use the Le Chat interface and are a small, informal sample.

Mistral describes Large as supporting multilingual reasoning, text transformation and coding, with a 32,000-token context window, function calling and structured output. It lists English, French, Spanish, German and Italian among supported languages. At launch it was offered through Mistral's platform and Azure, with private deployment available by arrangement; access to weights by agreement was not an open-weight release.

SLIDE 1 / 5Mistral’s MMLU chartThe vendor compares selected 2024 models on MMLU, not a benchmark average.

In Mistral's 2024 MMLU chart, Large scores 81.2% compared with 86.4% for GPT-4. That is this particular benchmark, not an average of all benchmarks. The company also reports 45% on HumanEval versus 67% for GPT-4 and different results on GSM8K; evaluation methods and model versions matter, and these vendor-reported comparisons do not establish a current ranking.

SLIDE 2 / 5Selected benchmark tableMistral reports results for several reasoning and knowledge evaluations.

Mistral also announced Small, positioned between its open-weight models and its flagship. I review the available structured JSON and tool-calling features, which can help application developers. A feature announcement does not mean every generated result will be valid or every function call safe to execute without validation.

SLIDE 3 / 5February 2024 API pricesThe displayed prices are historical and may have changed.

The price table shown at launch listed Mistral Large at $8 per million input tokens and $24 per million output tokens. I compare those with the GPT-4 rates shown in the video, $10 and $30, a 20% difference for those particular listed rates. Prices, model versions and provider terms change; these are historical figures rather than current buying advice.

Coding and safety responses

The model generates a simple loop and a terminal Snake game. Berman also probes moderation with a few prompts.

For coding, the model correctly generates a Python loop printing one through 100. Its Snake game uses a terminal interface; I run the code, see the snake grow after collecting food, and treat it as a pass, although it errors on hitting the wall. One successful basic interaction does not establish that the game is complete or robust.

SLIDE 4 / 5Generated Snake codeThe terminal game ran in the demonstration but later errored on a wall.

I probe moderation with potentially harmful requests. The model initially refuses one request but becomes more forthcoming after a fictional framing; I describe this as limited moderation in that session. This is an observation about a few prompts, not a comprehensive safety assessment, and a refusal to help with harmful conduct is not a general model failure.

Logic, math and structured output

The model answers several reasoning questions, with an incorrect word count and assumptions in some puzzles. Berman gives a qualified conclusion.

On the shirt-drying puzzle, the model explicitly distinguishes simultaneous drying from batches: under equal conditions and enough space, all shirts can dry in four hours; four sequential batches would take 16 hours. Those answers depend on the stated assumptions. It also correctly answers the Jane, Joe and Sam speed-order question.

SLIDE 5 / 5Drying-shirts reasoningThe response distinguishes simultaneous drying from limited-space batches.

The model solves simple addition and evaluates 25 − 4 × 2 + 3 as 20 using the usual order of operations. On a self-referential word-count question it claims eleven words, while I count ten, so I mark it incorrect. These examples are individual observations rather than a standardized reasoning score.

A puzzle about killers in a room receives the intended answer of three living killers: two original individuals survive and the entrant becomes a killer. As I note, counting the deceased person as a killer yields a different interpretation, so the answer depends on how the question is defined.

The model produces a JSON object from a description of three people. It also answers a marble-and-upturned-cup question by saying the marble remains on the table after the cup is moved. This relies on the usual interpretation that the marble is left on the table when the inverted cup is lifted; its description of how the marble got there need not be taken as a full physical trace.

In a final belief-tracking puzzle, the model says John expects the ball in the box while Mark expects it in the basket, reflecting where each last placed it and what each knows. I am impressed and call the session nearly perfect, though the word count failed and my tests are few, subjective and sensitive to prompt interpretation. The results do not prove a broad current ranking or comprehensive safety performance.