Testing UltraLM 13B: A Small Model’s Strengths and Reasoning Errors

RESUMEN

En 30 segundos

Matthew Berman runs a quantized UltraLM 13B model locally and tries an informal prompt rubric. He finds some usable answers but several reasoning failures, and questions how much its leaderboard placement predicts practical performance.

1Synthetic dialogue dataThe project describes separate query-generating and responding roles.
2Snake fails at runtimeThe generated code produces a pygame attribute error.
3Inconsistent logicThe shown speed-order attempts fail even though an earlier attempt reportedly succeeded.
4Arithmetic errorThe model states an order-of-operations rule but applies it incorrectly.

Conclusiones

UltraChat describes synthetic multi-turn dialogue data generated using two API roles.
A simple Python script succeeds, but the generated Snake game errors when run.
The poem is shorter than requested even though Berman scores it as a pass.
The model fails several logic and arithmetic prompts in this test.
Leaderboard placement and a small informal rubric each have limits.

TemasOpen-weight models Evaluaciones Modelos y entrenamiento Agentes de programación

TEXTO COMPLETO

Lee el texto editado por capítulos

Ver transcripción

Solo la transcripción

UltraChat and local setup

Berman introduces the model, its data and his quantized local setup.

In July 2023 I test UltraLM 13B, a small model that had ranked highly among open models on an Alpaca leaderboard. A leaderboard placement is a starting point for my tests, not evidence that it matches larger proprietary models on every task.

The UltraChat project describes creating multi-turn dialogue data using two ChatGPT Turbo API roles, one generating user queries and another responding. Its stated categories include questions about the world, writing and work on existing material.

DIAPOSITIVA 1 / 3UltraChat data overviewThe project describes its synthetic dialogue categories and generation process.

I run a GPTQ-quantized version locally on a Windows computer with an Nvidia GPU and use text-generation-webui. I load it with AutoGPTQ and compare the interface’s instruction and chat modes.

Coding and writing

He tests a simple script, a Snake game, a poem and a resignation letter.

A joke generates quickly, and a Python script printing numbers one through 100 is correct. The chat-mode response has formatting issues, so I later switch to instruction mode for some tests.

The Snake game output looks more structured in instruction mode, but running the generated Python produces an attribute error in pygame. I count that coding test as a failure.

DIAPOSITIVA 2 / 3Snake program errorThe generated code encounters a pygame attribute error.

I ask for a poem about AI with exactly 50 words. It repeatedly produces the same shorter poem, even after I increase the temperature. I like the poem and call it a pass, although it did not meet the explicit length constraint.

The model drafts a usable resignation email despite formatting problems and correctly identifies the US president in 1996. It refuses a request for wrongdoing, which I record as a refusal in my test.

Reasoning and arithmetic

Several logic and math answers fail in the shown runs.

On the shirt-drying puzzle, the model introduces unsupported claims about sunlight and surface area and concludes that drying takes eight hours. I mark the answer wrong; the question also depends on whether there is enough space to dry shirts together.

In a speed-order puzzle, Jane is faster than Joe and Joe is faster than Sam, yet the model says it cannot determine how Jane and Sam compare. Both instruction and chat modes fail in the shown attempts, although I say an earlier attempt succeeded.

The model answers 4 + 4 correctly but mishandles 25 − 4 × 2 + 3, despite mentioning the order of operations. Applying multiplication first gives 20.

DIAPOSITIVA 3 / 3Order-of-operations testThe model gives an incorrect result despite citing the rule.

Planning, summarization and conclusion

Berman tests more tasks and reflects on the limits of leaderboards.

A one-day meal-plan request produces a structured answer I consider useful. When asked to count the words in its own response, the model gives an incorrect count.

In the killers puzzle, its explanation misstates the premises and reaches an incorrect answer. Asked for the current year, it says it lacks real-time information; I count that acknowledgement of uncertainty as a pass.

For a summarization test, I provide a passage about bird flight and ask for a bullet-point summary. The response focuses on feathers but captures some source content, so I give it a qualified pass.

My conclusion is that UltraLM is interesting for its size and can handle some writing and factual prompts, but the reasoning failures make me skeptical of reading too much into its leaderboard position. These are informal examples from one setup, not a comprehensive evaluation.