Falcon 40B: A 2023 Leaderboard Result and a Slow GPU Test

In June 2023, I test Falcon 40B, a 40-billion-parameter model from the UAE's Technology Innovation Institute. Its base and instruction-tuned versions had appeared at the top of the Open LLM Leaderboard at the time. That ranking covers a particular set of evaluations and models, not every task, model or measure of usefulness.

The model card describes training Falcon 40B on one trillion tokens and highlights data filtering and deduplication. I discuss the developers' claims about using less training compute than some earlier models. Data quality is important, but my suggestion that it always outweighs dataset size is too broad to establish from these results alone.

I describe training on hundreds of AWS GPUs and mention QLoRA as a way to make adaptation more memory-efficient. QLoRA is a fine-tuning method, not evidence that Falcon's original pretraining can be reproduced in a few days. The checkpoint I later demonstrate is labeled GPTQ, a different quantization method.

The training material included filtered web data and other curated sources. I discuss possible uses such as chat, translation and coding and note that the training data was primarily English with some other European languages. The model card cautions that performance is not uniform across languages and tasks.

I choose a hosted GPU because running the 40B model exceeds the resources available on my own computer. A smaller Falcon 7B model was also available, although what runs locally depends on memory, hardware and format rather than parameter count alone.

Before publishing, I add an update that Falcon's license had changed to Apache 2.0. The Falcon 40B and instruction-tuned model cards list that license; commercial use remains subject to its terms and the terms of any other components used.

For the historical setup, I use a RunPod template and an RTX A6000 instance, open a web terminal, edit the text-generation-webui launch script and add AutoGPTQ and remote-code options. Enabling remote code permits code from the model repository to run, so the repository and its code need to be trusted and reviewed. Hosted GPU time may also incur charges.

After restarting the process, I open the web interface and download TheBloke's Falcon 40B Instruct GPTQ checkpoint. I call the variant QLoRA in the video, but the linked checkpoint is a GPTQ quantization; those terms are not interchangeable. The installation procedure and software versions are specific to this 2023 demonstration.

My first prompt asks for a Chinese translation of a sentence about wanting to learn more about AI. I paste the output into Google Translate and find that the translated meaning matches my original sentence. This is one informal back-translation, not a systematic evaluation of Chinese translation quality.

The practical problem in my setup is speed: the short translation took about a minute. I ask for a Python script to output the numbers 1 through 100 and wait while the answer appears. I say the visible code looks correct, but it uses range(101), which also outputs 0; the response does not exactly satisfy the prompt. The slow output prevents broader testing.

I explain that the leaderboard I am looking at measures quality on selected benchmarks rather than the latency of this particular deployment. A leading benchmark result and a frustrating user experience can therefore coexist. My single run does not prove that Falcon is universally the slowest model, or that quantization alone caused the slowdown.

I stop the test early and say I will revisit faster implementations or fine-tuned versions. The experience suggests that model quality, hardware, quantization and serving software all matter when choosing a system; the leaderboard and my two small prompts leave many capabilities untested.