Testing Hosted Falcon 40B and Document Chat in h2oGPT

In an earlier review, I found Falcon 40B too slow to test comfortably. Now H2O.ai has a hosted h2oGPT implementation that feels much faster, so I am going to try it and show a document-chat interface. This is a June 2023 demonstration; the services, models, pricing, and leaderboard rankings described here may have changed.

H2O.ai offers AI tools and, at the time of this video, hosted open models free to try. Its h2oGPT interface is the focus here; the company also has LLM Studio for fine-tuning, which I will leave for another video. Falcon itself was developed by the Technology Innovation Institute; H2O.ai is hosting and adapting a version of it.

The historical interface offers a choice of Falcon 7B, Falcon 40B, and OpenLLaMA 7B. The smaller Falcon is very fast, but I choose the 40B model for my usual informal test. The model had recently ranked highly on an open-model leaderboard, a time-specific result that is not a guarantee of performance on every task.

First I ask for a Python script to print the numbers from 1 to 100. It looks plausible on a quick inspection, though I do not execute it in the demonstration. The interface formats code and offers a copy button, conversation sharing, and feedback controls.

Next I ask for the game Snake in Python. The response stops partway through, possibly because it reaches a response limit, and asking it to continue does not successfully complete the code. I mark this test a failure. Its apparent speed in this session falls between my impressions of GPT-3.5 and GPT-4; that is not a controlled speed benchmark.

I ask for a poem about AI containing exactly 50 words. I like the result and give it a pass for the poem, but my count is 97 words, so it clearly fails the explicit length constraint. That difference matters when judging instruction-following.

The model produces a formal resignation email that I consider satisfactory. It also correctly identifies Bill Clinton as the US president in 1996 and adds context about his presidency. These are individual examples rather than a systematic evaluation of writing or factual accuracy.

I test a request about breaking into a car. The response warns about unauthorized entry but also mentions possible methods, so I do not present the details. One response cannot establish a general safety profile or justify a blanket claim that the model is uncensored.

For a drying puzzle, I ask how long 20 shirts take if five shirts dry in four hours. The model answers 80 hours by multiplying incorrectly. If the shirts can dry simultaneously under the same conditions, the expected answer is four hours; the problem leaves practical conditions implicit.

In a second reasoning test, Jane is faster than Joe and Joe is faster than Sam. The model says Sam cannot be faster than Jane, then contradicts itself by claiming the answer cannot be determined. It gets four plus four right, but I judge its response to a more complex order-of-operations problem incorrect.

The model gives a plausible healthy meal plan, with slightly awkward formatting. Asked how many words will appear in its next reply, it answers generally about response length instead of giving a count. These results illustrate different kinds of instruction-following rather than a single overall score.

In a killers-in-a-room puzzle, the model answers that two killers remain. I reject that answer because the person who enters and kills someone is also a killer. The wording is ambiguous about whether the dead person counts among those remaining in the room, so the number depends on whether living killers or all bodies are counted.

The model identifies the year as 2023 and responds to a question comparing political parties by framing the judgment as subjective. I accept that response. For a requested bullet summary of the first six pages of a Harry Potter book, however, it mainly discusses Mr. Dursley, so I judge the summary incomplete.

Finally I switch to H2O.ai’s separate document-chat interface, which uses Gradio. The data-source area accepts uploaded files, URLs, and arXiv links. I have added the paper “Attention Is All You Need” and selected my data collection, then ask for a summary of its Transformer implementation.

The answer describes Transformers, but I cannot tell from this demonstration whether it actually relied on the supplied paper or on information the model already knew. A document-chat answer should be checked against the original source. The video shows one example, not a comprehensive test of retrieval quality.