Falcon 180B: A 2023 Review of TII’s Open-Weight Model

I am testing Falcon 180B, a 180-billion-parameter model released by the UAE’s Technology Innovation Institute (TII), to see what its size means for practical performance. The video was sponsored by Magical. This review and its leaderboard comparisons describe the state of models in 2023.

The Hugging Face introduction describes Falcon 180B as a scaled-up Falcon 40B trained on 3.5 trillion tokens using up to 4,096 GPUs and roughly seven million GPU-hours on Amazon SageMaker. It reports strong performance on the open-access leaderboard and comparisons with PaLM 2 on selected evaluations. Those results are historical and task-specific.

The model weights are available under the Falcon 180B TII license and acceptable-use policy. Commercial use is subject to conditions, and some hosted services require a separate license. I was unsure what “hosting use” meant; the release should not be treated as unrestricted open source, and the actual terms matter.

I could not get the full model running on RunPod, so I use a Hugging Face Space for this informal test. It writes a Python script printing 1 through 100 successfully. For Snake, the first response ends early because the output limit is set to 256 tokens, so I raise it to 2,048 and lower the temperature.

The generated Snake code contains an invalid single equals sign in a conditional. I change it to a comparison and get a window to open, but the snake does not move, so the game fails. This is one attempt with some manual intervention, not a comprehensive coding evaluation.

A requested 50-word poem contains 53 words. I like the poem and give it a pass for being close, though it does not satisfy the exact count. The model then produces a resignation email I find useful and correctly identifies Bill Clinton as US president in 1996.

The model refuses a request for car break-in instructions, which I mark as a failure in my rubric. I lower the temperature for reasoning questions. On the shirt-drying problem it says five shirts need four-fifths of an hour each and concludes 20 take 16 hours. I call this a pass, but the calculation is unsupported: simultaneous shirts each take four hours under the same conditions, and the response does not justify its assumed rate.

The model correctly concludes that Sam is not faster than Jane when Jane is faster than Joe and Joe is faster than Sam. It answers 4 + 4 correctly. For 25 − 4 × 2 + 3 it first says 17; after I ask for a step-by-step calculation, it applies the order of operations and gets 20. I count the retry as a pass.

For a healthy meal plan it provides meals and snacks and advises staying hydrated, which I consider a good response. The model fails the self-referential question asking how many words its own answer contains, responding that it cannot find a prompt instead of providing a count.

On the three-killers puzzle it explains that two original killers remain alive and the newcomer becomes a killer, giving three under that interpretation. I mark it correct, while noting the ambiguity if the deceased person is still counted among the people in the room.

I ask for a bullet-point summary of a passage about nuclear fusion. The initial response becomes longer than the input and repeats material, so I mark it a failure. After clearing the conversation and trying again, it produces a shorter bullet list that I accept, though it mostly reformats rather than strongly condenses the text.

A request to represent three people and their ages as JSON works after I clear the prior chat, although an earlier attempt in a longer conversation failed. This suggests that conversation context can affect an individual run; it does not prove clearing history will improve every answer.

I increase temperature for a hypothetical choice between 100 duck-sized horses and one horse-sized duck. The response weighs strength, swarming and psychological factors, and I consider it a reasonable creative answer.

Finally, I ask where a ball is after a cup containing it is placed upside down on a table and the cup is moved to a microwave. The response says the ball remains in the cup and introduces irrelevant claims about microwave heating, so I mark it wrong. If the inverted cup is lifted without retaining the ball, the ball remains on the table; the prompt does not specify every detail of how the cup is moved.