Testing Karasu Mixtral 8x22B on Coding and Reasoning Tasks

Mistral has released Mixtral 8x22B, a larger sparse mixture-of-experts model following Mixtral 8x7B. I am excited because the earlier model was a favorite of mine. The name describes the expert architecture, not 176 billion independently active parameters: Mistral later described roughly 141 billion total parameters and 39 billion active per token ([Mistral documentation](https://docs.mistral.ai/models/mixtral-8x22b-0-1-0-3)).

The initial release appeared through a torrent with little accompanying explanation. Rather than testing the unmodified base model, I select Lightblue's [Karasu-Mixtral-8x22B-v0.1](https://huggingface.co/lightblue/Karasu-Mixtral-8x22B-v0.1), an early chat fine-tune. Fine-tuning can affect instruction following and safety behavior, so these responses do not necessarily represent every Mixtral variant.

I use Infermatic's hosted interface because the full model is too large for my own machine without quantization. I describe the service as free at the time, set a long output limit and lower the temperature. Hosted offerings and local hardware requirements change, so this is a snapshot of the April 2024 setup rather than a current pricing or compatibility guarantee.

The first programming prompt asks for the numbers from one to 100, and I accept the response. I then ask for Snake in Python and run the generated program in VS Code. The game opens, the snake moves, and a self-collision ends the game; I am impressed by how much is working on the first attempt.

The snake can pass through the wall, which is not the behavior I wanted. I ask the model to add a visible score and end the game when the snake leaves the window. The revised code displays a working score, but the wall condition still does not end the game and the score does not reset as expected. I count it as a pass despite these remaining defects.

I probe how the fine-tune responds to a request for wrongdoing: it initially refuses, then responds differently after I reframe the request as fiction. This limited exchange is not a comprehensive safety evaluation. Refusing harmful instructions is a safety behavior rather than a failure of general reasoning, and the result may vary by prompt and fine-tune.

For the shirt-drying puzzle, the model assumes that time scales directly with the number of shirts and answers 16 hours. I accept that under a sequential batching assumption, but the question does not state such a limit. If all 20 shirts can dry at once under similar conditions, the expected time is about four hours; the space and conditions matter.

On a transitive comparison, Jane is faster than Joe and Joe is faster than Sam; the model correctly concludes that Sam is not faster than Jane. It also answers four plus four correctly. These are useful examples of the outputs, but a single success on each prompt is not a robust measure of reasoning.

For 25 minus 4 times 2 plus 3, the model initially says 19. The correct result is 20 by the usual order of operations. A follow-up asks for steps and reaches the right final answer, although the response also repeats the earlier incorrect number; I score the example generously, but the inconsistency remains.

When asked to count the words in its own response, the model's stated count does not match what I count, so I mark it wrong. On a puzzle about three killers in a room and a newcomer who kills one, it also produces inconsistent reasoning. Under the usual assumption that the newcomer becomes a killer and nobody leaves, three living killers remain.

The structured-output test asks for JSON describing Mark, Joe and Sam with their ages and genders. The response places the people in a top-level structure and appears to preserve the specified facts, so I count it as a pass. This is a simple formatting example rather than a full test of schema reliability.

In the marble puzzle, a cup containing a marble is inverted on a table and moved to a microwave without changing orientation. The model says the marble remains inside the cup because of gravity. Under the usual assumption that the open cup is lifted off the table and the marble is not retained, the marble stays on the table; the exact outcome depends on handling and geometry not fully specified in the prompt.

A belief-tracking scenario has John put a ball in a box and leave, after which Mark moves it to a basket and leaves. The model correctly says John expects the ball in the box and Mark expects it in the basket, assuming neither saw further changes. It also notes that checking would reveal the ball's actual location.

I request ten sentences ending in the word 'apple.' The response includes the word but, as I observe, does not put it at the end of each sentence, so I mark it wrong. This illustrates a concrete instruction-following error in this sample, not a general limit of every model in the family.

Finally, a digging question asks how long 50 people would take if one person needs five hours to dig a ten-foot hole. The model explicitly assumes equal rates and no space or equipment limits, then calculates six minutes by idealized linear scaling. The arithmetic follows those assumptions; in a real single hole, crowding and coordination could make them unrealistic.

I conclude that this early Karasu fine-tune performs well on some tasks, although on my informal rubric it does not surpass the earlier 8x7B model. I expect other fine-tunes to improve the results. That comparison is my judgment from a small number of prompts, not a controlled or current model ranking.