Orca Mini V2 vs. OpenOrca: A 2023 Informal Model Comparison

In July 2023, Matthew Berman compares two models inspired by Microsoft’s Orca research: Orca Mini V2, which he identifies as a 7B Llama fine-tune by P. S. Mathur, and an OpenChat V2 OpenOrca preview trained using the OpenOrca dataset. The links in the video description include 13B checkpoints, so they should not be assumed to identify the exact 7B model he says he tested.

Berman runs both unquantized models in the text-generation web UI on Runpod using the Transformers loader, with temperatures close to zero. He notes that the OpenChat model card calls for code-based usage but chooses a prompt template instead. That mismatch, and his need to manually stop ongoing output, could disadvantage the preview and limits the fairness of the comparison.

Both models produce a Python script to print the numbers from 1 to 100, which Berman marks as passes. In the next round, he asks each to write Snake in Python. He says OpenOrca’s output is obviously unusable and that Orca Mini’s code fails when he tries to run it in VS Code, so he marks both as failures.

For a poem of exactly 50 words, Berman counts 103 words from OpenOrca and 30 from Orca Mini. He gives both a pass based on his appreciation of the poems, even though neither met the explicit length constraint. Both then produce resignation emails that he considers acceptable, and both identify Bill Clinton as US president in 1996.

Berman tests responses to an unsafe request and awards Orca Mini a point for complying while OpenOrca refuses. That reflects his preference for a less restricted model; refusing a harmful request should not be treated as a general capability failure. He does not establish either model’s full safety profile with one prompt.

A shirt-drying puzzle asks how long 20 shirts take if five dry in four hours. Both models use proportional calculations, and Berman faults their arithmetic while saying their approach is sound. Under the usual assumption that all shirts dry simultaneously under the same conditions, the answer is four hours, not a time proportional to the number of shirts; capacity and conditions are not specified in the prompt.

A speed-order question says Jane is faster than Joe and Joe is faster than Sam. OpenOrca incorrectly says the comparison between Jane and Sam cannot be determined; Orca Mini incorrectly says Sam is faster. By transitivity, Jane is faster than Sam, so Berman marks both wrong.

Both answer 4 + 4 correctly. For the harder expression 25 − 4 × 2 + 3, OpenOrca gives 17 and Orca Mini gives 10; both are incorrect, as the answer is 20 under standard order of operations. Each then produces a daily meal plan Berman considers acceptable.

Asked how many words are in its own response, OpenOrca does not give a useful count. Orca Mini answers “There are 10 words in my response to the prompt,” which contains ten words using ordinary space-separated counting. Berman credits it for the self-counting result; one response is not a robust evaluation of this ability.

Both models fail Berman’s killers-in-a-room puzzle according to his scoring: OpenOrca says the information is insufficient and Orca Mini introduces unsupported people and figures. The exact answer can depend on assumptions about whether the newly killed person is counted and whether the entrant was already a killer; the responses shown do not reason cleanly through the stated setup.

The final task asks for a bullet-point summary of a passage about nuclear fusion. Berman judges that neither output meets the request. He declares Orca Mini the winner by two points, including the unsafe-compliance round and the self-counting test, and notes that quantized versions can reduce hardware requirements. This informal, single-prompt comparison is not a general ranking of current models.