Testing UltraLM 13B: A Small Model’s Strengths and Reasoning Errors

In July 2023 I test UltraLM 13B, a small model that had ranked highly among open models on an Alpaca leaderboard. A leaderboard placement is a starting point for my tests, not evidence that it matches larger proprietary models on every task.

The UltraChat project describes creating multi-turn dialogue data using two ChatGPT Turbo API roles, one generating user queries and another responding. Its stated categories include questions about the world, writing and work on existing material.

I run a GPTQ-quantized version locally on a Windows computer with an Nvidia GPU and use text-generation-webui. I load it with AutoGPTQ and compare the interface’s instruction and chat modes.

A joke generates quickly, and a Python script printing numbers one through 100 is correct. The chat-mode response has formatting issues, so I later switch to instruction mode for some tests.

The Snake game output looks more structured in instruction mode, but running the generated Python produces an attribute error in pygame. I count that coding test as a failure.

I ask for a poem about AI with exactly 50 words. It repeatedly produces the same shorter poem, even after I increase the temperature. I like the poem and call it a pass, although it did not meet the explicit length constraint.

The model drafts a usable resignation email despite formatting problems and correctly identifies the US president in 1996. It refuses a request for wrongdoing, which I record as a refusal in my test.

On the shirt-drying puzzle, the model introduces unsupported claims about sunlight and surface area and concludes that drying takes eight hours. I mark the answer wrong; the question also depends on whether there is enough space to dry shirts together.

In a speed-order puzzle, Jane is faster than Joe and Joe is faster than Sam, yet the model says it cannot determine how Jane and Sam compare. Both instruction and chat modes fail in the shown attempts, although I say an earlier attempt succeeded.

The model answers 4 + 4 correctly but mishandles 25 − 4 × 2 + 3, despite mentioning the order of operations. Applying multiplication first gives 20.

A one-day meal-plan request produces a structured answer I consider useful. When asked to count the words in its own response, the model gives an incorrect count.

In the killers puzzle, its explanation misstates the premises and reaches an incorrect answer. Asked for the current year, it says it lacks real-time information; I count that acknowledgement of uncertainty as a pass.

For a summarization test, I provide a passage about bird flight and ask for a bullet-point summary. The response focuses on feathers but captures some source content, so I give it a qualified pass.

My conclusion is that UltraLM is interesting for its size and can handle some writing and factual prompts, but the reasoning failures make me skeptical of reading too much into its leaderboard position. These are informal examples from one setup, not a comprehensive evaluation.