Six Language Models Compared on an Informal 2023 Prompt Set

Instead of testing one language model, I use H2O GPT to compare six at once: H2O-tuned Falcon 7B and 40B, MPT-30B Instruct, Vicuna 33B, an H2O-tuned LLaMA 65B, and OpenAI’s proprietary GPT-3.5 Turbo. This is a 2023 snapshot and an informal set of prompts, not a controlled ranking of all versions or present-day models.

First I ask for a Python script that prints the integers 1 through 100. Falcon 7B generates random integers instead, while the other five outputs appear to solve the requested task. Speed is visible in the interface, but I am scoring the quality of the answers rather than generation speed.

The harder programming request is a Snake game in Python. Some outputs are obviously incomplete or contain missing pieces, and after trying them I find that none of the six works. This is one prompt and one run per model, so it cannot establish their general coding ability.

For a poem about AI with exactly 50 words, I report 145 words for Falcon 7B, 121 for Falcon 40B, 53 for Vicuna and 51 for GPT-3.5 Turbo. MPT returns nothing and LLaMA continues too long. I award several passes based on quality or closeness, but none of these counted examples meets the exact 50-word requirement.

For a resignation email, I accept the Falcon, MPT, LLaMA and GPT-3.5 outputs and mark Vicuna’s response incomplete because it lacks a sign-off and other departure information. These are subjective judgments about usefulness. On the 1996 US president question, the models that respond correctly identify Bill Clinton; MPT returns no output.

A request for car break-in instructions produces a mixture of refusals and answers. I describe the refusals as censorship in my rubric, but refusal of one request is not a comprehensive measure of a model’s capabilities or safety. MPT’s intermittent blank responses may also reflect the hosted implementation rather than the model alone.

For the shirt-drying problem, Falcon 7B says 20 shirts take four hours, which is consistent with simultaneous drying under the same conditions and sufficient space. Falcon 40B gives 16 hours without establishing a capacity limit; GPT-3.5 also gives 16 by incorrectly dividing four hours by five shirts. I accept those as serial assumptions, but the displayed reasoning does not justify GPT-3.5’s per-shirt time. MPT explicitly describes four batches in its displayed response, a valid 16-hour result if only five can dry at once, although I score it as a failure.

On the speed comparison—Jane faster than Joe, Joe faster than Sam—Falcon 40B, MPT, LLaMA and GPT-3.5 correctly say Sam is not faster than Jane. Falcon 7B says yes, and Vicuna says the comparison cannot be determined. The example tests a specific transitive relation, not reasoning in general.

All six answer 4 + 4 correctly. For 25 − 4 × 2 + 3, whose correct answer is 20, only GPT-3.5 gives a single correct answer. MPT offers both 20 and 21; I mark that as a failure because the expression has one result.

I then ask for a healthy meal plan. Five models produce responses I accept; MPT again produces no output. Later MPT temporarily disappears from the interface, which reinforces the possibility that some missing answers reflect service availability rather than a tested model’s response quality.

The self-referential question asks how many words are in the model’s next reply. I count all six as failures: GPT-3.5 says its answer has ten words but I count eight, while MPT gives no answer. The scoring depends on the chosen word-count convention and the specific response.

In the three-killers puzzle, I expect four because the dead person remains in the room and the newcomer is also a killer. Vicuna and GPT-3.5 answer three, treating the two surviving original killers plus the newcomer as the relevant count. I score them wrong, but three and four reflect different interpretations of whether a dead killer is counted.

Asked the current year, Falcon 7B and LLaMA give acceptable answers, and I also credit Vicuna for explaining that it lacks current-date access. I do not credit GPT-3.5’s qualified answer. This is a test from 2023, and without a clock a model’s answer may depend on its interface and context.

On a question asking which US party is less bad, I consider the models’ broadly noncommittal responses acceptable. MPT was temporarily unavailable but returns when I retry; I also accept its answer. A single political prompt is not a reliable measurement of political bias.

For summarization I first try a 500-word passage about tadpoles becoming frogs, then shorten it to 200 words when the prompt appears too long. All six produce summaries that I accept, although MPT’s is brief. The changed input and service behavior are part of the conditions of this demonstration.

My spreadsheet shows 16 scored items. Falcon 7B and GPT-3.5 Turbo tie at 11 of 16, or 68.75%; LLaMA 65B gets 10, Falcon 40B gets 9, Vicuna 33B gets 8 and MPT-30B gets 7. I am surprised that this Falcon 7B configuration scores above the larger open-weight models in my rubric. The rankings depend on my subjective and sometimes inconsistent scoring, a small prompt set, model versions and the hosted interface; they do not establish a general performance hierarchy.