HuggingChat’s 2023 Debut: Testing Writing, Reasoning and Python Code

In April 2023, I am trying HuggingChat, a Hugging Face chat interface running an OpenAssistant model based on LLaMA 30B. Its interface resembles ChatGPT and lists the active model. I am interested in how it compares with GPT-3.5, though a few prompts can only provide a limited snapshot.

Hugging Face hosts machine-learning models, datasets and libraries; here I am focusing on its chat application. I describe the project as open source, but the original LLaMA model had licensing restrictions, including limits on distribution and commercial use. The interface and the underlying model should not be assumed to have identical licenses.

First I ask for a 50-word poem about AI. The answer arrives quickly and reads reasonably well despite capitalization issues. It feels faster to me than the models I have used, but I have not measured the speeds under controlled conditions.

Next I select the example prompt for a Python Snake game. HuggingChat produces formatted code, but its answer appears incomplete and includes spelling problems. Longer code has been difficult for many models I have tried, so I copy the output into Visual Studio Code to see whether it runs.

The code needs a missing random import, and I notice indentation and method-name problems. I try a few fixes, but the game still does not work and I stop debugging. It has produced a substantial starting point, yet this example falls short of a playable game.

I move to the suggested email-writing example: a restaurant owner asks a supplier for weekly products. At a quick glance the response reads like a professional email. I also ask for a lemon cheesecake recipe and find the resulting ingredients and instructions plausible, without cooking or independently verifying the recipe.

For reasoning, I ask whether Sally is faster than Joe when Sally is faster than Nick and Nick is faster than Joe. HuggingChat correctly answers yes: the comparison is transitive. This is one simple problem, not a broad measure of reasoning.

The arithmetic task is 34 times 56 plus 109, then plus 5. HuggingChat first asks for clarification and then claims it cannot calculate; when I add parentheses it gives 2,158. That is incorrect: the expression equals 2,018.

I repeat the arithmetic prompt with GPT-3.5. In these attempts it answers 1,928 and then 2,013, both incorrect for the full expression. GPT-4 takes longer in the demonstration but gets the calculation right on the first shown attempt; this is not a controlled speed or accuracy study.

I also ask GPT-3.5 to write the Snake game. The first output produces an indentation error, so I feed the error back and try its revision. The program still does not become a playable game during my test, so I count both models’ attempts as failures on this example.

Finally, GPT-3.5 gets the Sally–Nick–Joe question wrong. Its counterexample gives Joe a faster time than Nick, contradicting the premise that Nick is faster than Joe. HuggingChat succeeded on that prompt, but these isolated 2023 examples are not enough to rank either system generally.