Mixtral 8x7B: Mixture-of-Experts Architecture, Setup and Early Tests

Mistral AI released Mixtral 8x7B in December 2023 after a torrent link drew attention. I am excited to explain how it works, show my setup and test it. My initial description as eight separate complete models is only a rough analogy: Mixtral is one sparse mixture-of-experts model with shared components and multiple feed-forward experts.

A router selects two of eight feed-forward experts for each token at each layer, and the selected outputs are combined. Other parts of the Transformer are shared, so the “8x7B” name does not mean eight complete 7B models or 56 billion parameters. Mistral reports about 46.7 billion total parameters and 12.9 billion active per token; active computation and the memory needed to hold the weights are different considerations.

I compare the potential compute efficiency with dense models and cite Mistral’s reported performance against Llama 2 70B. Actual speed depends on hardware, implementation, batch size and memory constraints. The model’s reported 32,000-token context and performance numbers describe the release at that time, not a guarantee of accurate reasoning throughout the window.

I mention speculation circulating at the time about GPT-4’s internal mixture-of-experts architecture and size. These are third-party claims rather than details confirmed in the material shown, so they should not be treated as established specifications. I also note Mistral’s separate higher-end Mistral Medium service, which the company then described as a prototype endpoint.

Mistral’s published comparison shows Mixtral near GPT-3.5 on MT-Bench and competitive with Llama 2 70B across several tests; the table does not show Mixtral winning every benchmark. These are Mistral’s selected comparisons and settings from 2023. I think the results are encouraging for openly available models, while my prediction that they will soon catch up with closed models is a forecast.

Mistral released the weights under Apache 2.0, allowing developers to download and adapt them subject to the license. I adopt the phrase “open weight,” distinguishing access to model weights from access to all training data, code and documentation. I also look forward to community fine-tunes, including work discussed by Eric Hartford.

Mistral’s Guillaume Lample describes the per-token, per-layer routing and reports strong results in French, German, Spanish and Italian, as well as improvements in math and coding compared with Mistral 7B. I cite Andrej Karpathy’s explanation that only feed-forward blocks are replicated, which accounts for the lower total parameter count than eight times seven billion.

For my December 2023 setup, I use RunPod and text-generation-webui with two A100 GPUs, each with 80 GB of memory. The displayed on-demand price is about $4 per hour, and I increase container and volume storage. This is the configuration and price shown at the time; other precisions or quantized deployments can require different resources, and current prices and interfaces may differ.

The template I use does not yet support Mixtral, so I edit its startup script to install a newer Transformers version and enable the option to trust remote code, then restart the pod and reconnect to the web interface. This is a historical description rather than a current installation recipe; enabling remote code executes repository-supplied code and requires trust in that source.

I select the Mixtral instruct model from Hugging Face, download the files, allocate GPU memory and choose BF16. I thank Eric Hartford and Jon Durbin for helping resolve the setup issues. After refreshing the model list and loading the model, I verify the remote-code setting and select a generation preset before starting the tests.

The first prompt asks for a Python script printing numbers from 1 to 100, which I mark correct. I then ask for Snake in Python. The model generates a program using the curses library, and in my demonstration it runs on the first attempt. This is an encouraging example, not a broad coding benchmark.

I also mark a resignation letter and the identification of Bill Clinton as US president in 1996 as passes. The instruction model refuses a request about breaking into a car; I describe it as censored and anticipate different fine-tunes, but refusal of potentially harmful assistance is a safety behavior rather than a failed intelligence test.

In the shirt-drying puzzle, five shirts dry in four hours and I ask how long 20 would take. The response says shirts dry independently but then incorrectly divides four hours by five and concludes 16 hours. If all 20 are laid out at once under the same conditions, the natural answer is four hours; limited drying space would require additional assumptions. Although I consider awarding a pass, the stated answer and calculation do not support one.

The model correctly concludes that Sam is not faster than Jane when Jane is faster than Joe and Joe is faster than Sam. I say it also passes a basic addition problem, an order-of-operations example and a meal-plan request. On the question asking it to count words in its own response, I find its answer incorrect.

In the killer puzzle, three killers are in a room and a newcomer kills one. The model answers four because it counts the deceased person among those present, and I call this correct. Under the usual interpretation that counts living killers, two originals plus the newcomer make three; counting the deceased as a killer yields four. The wording is ambiguous, so my unqualified pass is not a reliable score.

For a summarization prompt, the model uses bullets rather than the requested dashes, but I judge the content adequate. It also produces a JSON representation of the described people, genders and ages. These are useful observations about individual outputs, though they do not measure how consistently the model follows formatting instructions.

The marble problem asks what happens when a cup containing a marble is placed upside down on a table and the cup is moved to a microwave. The model says the marble remains on the table; I count that as a pass. This relies on the ordinary assumption that moving the cup does not also carry the marble.

In the final belief-state problem, John puts a ball in a box and leaves; Mark moves it to a basket and leaves. When both return without learning what happened in their absence, the model says John expects the box and Mark expects the basket. I find that answer convincing and conclude that Mixtral is the strongest open-weight model I have personally tested so far, a subjective assessment of this small 2023 test set.

I am excited to see community fine-tunes and quantized versions that might make the model accessible with less hardware. Those possibilities do not imply that every compressed variant will preserve the same quality or that my particular two-GPU setup is required for every deployment.