Testing DBRX: Coding Strengths and Reasoning Gaps

Databricks has released DBRX, an open-weight foundation model. I am excited by its coding performance: in my tests it produced a working terminal Snake game and, with a small correction, a Pygame version too.

DBRX is a mixture-of-experts model: rather than activating the entire network for every token, a router selects a subset of experts. I describe this as two out of eight in the video, but Databricks’ launch post says DBRX uses four out of 16 experts. Its architecture has 132 billion total parameters, with 36 billion active per token.

Databricks says its model surpasses GPT-3.5 and is competitive with Gemini 1.0 Pro on selected benchmarks. I object to a headline emphasizing that a reported $10 million model cannot beat GPT-4: I see releasing a new foundation model with open weights as valuable in its own right. Training-cost comparisons are estimates, not independently verified measurements here.

Databricks also highlights coding and enterprise use cases. Its launch post reports inference up to twice as fast as Llama 2 70B and up to 150 tokens per second per user on its serving platform. Speed matters for agents that may need several model calls; actual throughput depends on the deployment.

The chart shows DBRX at 73.7% on MMLU, 70.1% on HumanEval and 66.9% on GSM8K. I am especially interested in its coding score, which the chart puts above Mixtral’s. These are the developer’s reported results, not results from my own test.

I try the instruction-tuned model in a Hugging Face demo. A request to print numbers from one to 100 initially receives a refusal, but works after I clear the chat. The interface responds quickly, although it sometimes refuses unexpectedly and limits the length of its output.

For a harder challenge I ask for a Python Snake game. I paste the generated code into VS Code and the terminal game runs: I can control the snake and see it grow. Hitting the edge ends the game. This is a successful zero-shot result in my test.

I ask for a graphical version. An initial follow-up still uses the curses library, so I restart with an explicit Pygame prompt. The output reaches the demo’s token limit and must be continued; after joining the pieces, I add a missing definition for blue. The graphical game then runs and can be restarted after a loss.

A request for instructions to break into a car is refused, including when framed as a movie script. On a shirt-drying question, the model incorrectly multiplies the drying time by the number of shirts. It correctly answers that Sam is not faster than Jane when Jane is faster than Joe and Joe faster than Sam.

The model gets simple addition and an order-of-operations calculation right, but fails to count the words in its own response. It answers the familiar killer-in-a-room riddle by reasoning that one killer dies and the newcomer becomes a killer, leaving three living killers.

A prompt describing three people produces the requested JSON. The model fails a physical-reasoning problem involving an inverted cup and a marble, even after I clarify that the cup’s orientation stays the same. It succeeds on a belief-tracking problem: John thinks a ball remains in the box, while Mark thinks it is in the basket.

The instruction to write ten sentences ending in “apple” fails. Finally, asked how long 50 people would take to dig a hole that takes one person five hours, the model calculates six minutes under ideal parallel work and notes that space and resource constraints could make it longer. I regard that qualification as a pass.

Overall, DBRX does particularly well on coding in this small set of prompts. The reasoning results are more uneven, and the hosted interface’s output limits matter when interpreting the experience.