Mistral Medium in 2023: API Pricing and Informal Model Tests

In December 2023, Matthew Berman tests the original Mistral Medium prototype through its API after strong results from the open-weight Mixtral model. He considers whether the hosted model might be a cheaper GPT-4 alternative, but his informal prompts and one-off results are not a comprehensive measure of model quality.

Mistral’s launch announcement described three endpoints: Tiny for Mistral 7B, Small for Mixtral 8×7B, and Medium for a higher-performing prototype. Berman speculates that Medium might use a larger mixture-of-experts architecture, but the company’s announcement does not establish that guess. It reported an MT-Bench score of 8.6 for Medium versus 8.3 for Small.

Berman compares the company’s published scores on MMLU, WinoGrande and GSM8K and observes that Medium scores higher than Small in the displayed tests. He also finds a higher MT-Bench result for GPT-4 elsewhere. Benchmark figures come from the period and depend on model versions and evaluation setups.

His historical pricing spreadsheet estimates Medium at about $0.00275 per thousand input tokens and $0.00824 per thousand output tokens, compared with $0.03 and $0.06 for the GPT-4 option shown. He sees a strong cost argument for higher-volume applications if quality is sufficient. These are his converted December 2023 figures, not current prices or evidence of equivalent performance.

Because Medium is accessed through an API, Berman demonstrates a small Python script that sends a prompt and retrieves a response. He says the token shown during filming will be revoked before publication. He uses the script to run a series of coding, writing, factual and reasoning prompts.

A Python script to print numbers from 1 to 100 passes his basic check. The model then produces a terminal Snake game using the curses library, and Berman runs it successfully on the first attempt. He notes its similarity to his earlier Mixtral result and guesses they may share training data, but that is not verified.

A request for a poem of exactly 50 words produces a poem that Berman says does not meet the word count. He considers a resignation email and a question about the 1996 US presidency successful. His own later near-perfect summary should be read alongside this missed constraint and the arithmetic error he identifies later.

Berman probes how the model handles unsafe and benign requests. In these particular runs it responds to a request involving vehicle break-in, refuses a request involving illegal drug production, and answers a spicy salsa question. He calls the system relatively permissive; these few prompts do not establish its overall safety behavior.

For the shirt-drying puzzle, the model says 20 shirts can dry in about four hours if all are spread out under comparable conditions. It also explains that with room for only five at a time, four batches would take 16 hours. Berman praises the answer for making the parallel-versus-serial assumption explicit.

The model correctly answers the ordering question in which Jane is faster than Joe and Joe is faster than Sam. Berman also records passes on basic arithmetic and an order-of-operations example, then asks a distance-and-time problem: 450 miles in six hours requires an average of 75 mph.

A harder tank problem has an inlet filling a tank in 25 minutes and a drain emptying it in 55. The model answers 37.9 minutes, which Berman marks wrong. The net fill rate is 1/25 − 1/55 tank per minute, giving approximately 45.8 minutes.

Asked how many words are in its own response, the model writes a sentence saying there are nine words and then adds a separate confidence line. Berman counts nine words in the sentence and celebrates the result; counting the additional line as part of the response would change the total, so the outcome depends on that interpretation.

In a room with three killers, a newcomer kills one and nobody leaves. The model counts the two surviving original killers plus the newcomer, giving three living killers. Berman accepts this while observing that interpretations may differ if the dead person is also counted as a killer.

A structured-output task describes three people, two men aged 19 and a woman aged 30. Berman says the generated JSON correctly represents the names, genders and ages. This is a single sample rather than a reliability measurement across many possible inputs.

A physical reasoning prompt places a marble beneath an inverted cup and then moves the cup to a microwave. The model says the marble remains inside the cup because of inertia; Berman marks that explanation wrong. Under the ordinary assumption that only the upside-down cup is lifted, the unsupported marble stays on the table.

The final test asks what John and Mark believe after Mark moves a ball from a box to a basket while John is away. The model says John expects the ball in the box and Mark expects it in the basket, which Berman accepts. He closes enthusiastically, though his earlier poem and tank tests show that the demonstrated run was not perfect.