Testing Dolphin 2.2 Yi 34B on Coding and Reasoning Tasks

RÉSUMÉ

En 30 secondes

Matthew Berman introduces 01.AI’s Yi model family and tests Eric Hartford’s Dolphin 2.2 Yi 34B using a GPTQ quantization. The model handles some simple writing and structured-output tasks but fails his Snake, arithmetic and spatial reasoning examples; his small, subjective prompt set is not a controlled model benchmark.

1Yi and Dolphin01.AI released Yi models in 6B and 34B sizes; Berman tests a quantized Dolphin fine-tune of the 34B model.
2Local setupHe loads TheBloke’s GPTQ model with ExLlamaV2 and sets the interface parameter to 2,048.
3Snake game failsThe generated Python code has missing or incorrect definitions and does not run despite attempted fixes.
4A contested drying answerThe model calculates 16 hours, which Berman accepts under a serial interpretation; the prompt describes five shirts laid out together and does not establish that serial assumption.
5A spatial reasoning failureThe model places a marble in a microwave after an inverted cup is moved, overlooking that the marble can remain on the table.

À retenir

The tested model was a GPTQ-quantized Dolphin 2.2 fine-tune of Yi 34B, not the base model itself.
01.AI also offered a separate 200K-context Yi variant; that does not establish that the tested setup used 200K context.
The Snake code had missing imports and undefined or misspelled names in Berman’s run.
The model answered some simple prompts well but made mistakes in order of operations and spatial reasoning.
Berman’s grading was informal, including a pass for a 36-word poem requested at exactly 50 words.
Quantization and the small prompt set limit conclusions about the full-precision model or general performance.

ThèmesOpen-weight models Foundation Models Benchmarks Quantization Agents de code

TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Afficher la transcription

Transcription seule

Yi models and the test setup

Berman describes 01.AI’s Yi models and Eric Hartford’s Dolphin fine-tune. He loads a quantized version in a local interface.

01.AI has released the Yi family of foundation models, trained in English and Chinese, with 6-billion- and 34-billion-parameter versions. A separate Yi variant offers a 200,000-token context window. The developer’s chart shows strong results relative to several models available in late 2023, including Llama 2 70B, but those figures are limited to the benchmarks and settings shown.

Eric Hartford fine-tuned Yi 34B with Dolphin 2.2 data. I am testing TheBloke’s GPTQ-quantized Dolphin 2.2 Yi 34B because I had trouble loading the unquantized model in the text-generation web UI. I expect the difference to be small, but quantization can change outputs, so this test does not establish how the full-precision model would behave.

DIAPOSITIVE 1 / 4Loading the quantized Dolphin Yi modelThe interface shows TheBloke’s GPTQ model selected for the local test.

After downloading the model, I refresh the list, select it and load it with ExLlamaV2. I set a parameter to 2,048 and use the Dolphin prompt template. A basic greeting works, and the model produces a Python script to print numbers from 1 to 100 that looks correct.

Coding, writing and refusal tests

The model handles a basic script but fails to produce a working Snake game. Berman also tests writing, factual recall and whether the model refuses several harmful requests.

I then ask for a Snake game in Python. Generation is not especially fast, and the code fails when I run it: pygame was not imported. Even after I add the import, I find further errors, including a misspelled variable and undefined names. I stop trying to repair it and mark this example as a failure.

DIAPOSITIVE 2 / 4Snake code produces an errorThe initial program fails because pygame is not defined.

The model writes an AI poem that I like, but it contains 36 words instead of the requested 50; I generously give it a pass despite the missed constraint. It also writes an acceptable resignation email and correctly identifies Bill Clinton as US president in 1996. These are individual checks rather than a systematic assessment.

I probe its refusal behavior with several harmful requests. It responds to one with an answer I regard as unreliable, but refuses others. I conclude that the tested model is not consistently without refusals; a handful of prompts cannot characterize its behavior or safeguards comprehensively.

Logic, math and structured outputs

Drying, ordering and arithmetic questions produce mixed results. The model also attempts summarization and JSON generation.

For the shirt-drying puzzle, five shirts laid out in the sun take four hours, and I ask how long 20 would take. The model divides 240 minutes by five and multiplies by 20, yielding 16 hours, which I accept under a serial interpretation. The prompt says the original shirts were laid out together and does not establish serial drying; with all 20 drying simultaneously under comparable conditions, the answer would be about four hours.

DIAPOSITIVE 3 / 4The shirt-drying answerThe response calculates 16 hours by treating the time as proportional to the number of shirts.

It correctly infers that Sam is not faster than Jane when Jane is faster than Joe and Joe faster than Sam, and it answers 4 + 4 correctly. On 25 − 4 × 2 + 3, however, it subtracts before multiplying and remains wrong when prompted to use the order of operations; the correct result is 20. A meal-plan request looks good, while a question about the word count of its own response fails.

DIAPOSITIVE 4 / 4Order-of-operations errorThe model subtracts before multiplying in a simple arithmetic expression.

The model also misses my killer puzzle, overlooking the person who entered and committed the killing. It does better when asked to summarize a passage about nuclear fusion with dash bullets and to create JSON describing three people. I regard the JSON as valid even though I do not like its formatting.

Hypothetical and spatial reasoning

Berman tries an unusual animal comparison and an inverted-cup problem. He concludes that this configuration performed worse than he expected.

For the hypothetical choice between many duck-sized horses and one horse-sized duck, the model first warns against fighting, then compares the risks of a single large animal and many small ones. I accept that answer. Finally, in a spatial puzzle involving a marble under an inverted cup moved into a microwave, it says the marble goes into the microwave; I mark that wrong because it may remain on the table when the cup is lifted. Overall, this configuration is less impressive than I hoped, and in my view Dolphin Mistral performed better on my tests.