Testing Mixture-of-Agents on Coding, Arithmetic and Reasoning

SUMMARY

In 30 seconds

In June 2024, Matthew Berman tests Together AI’s Mixture-of-Agents demo with four models and a Qwen aggregator. He tries coding, arithmetic, riddles and instruction-following, praising several nuanced answers while observing that the Snake implementations fail and the self-count is wrong. The project’s reported advantage over GPT-4o is an AlpacaEval 2.0 result under specific evaluation conditions, not evidence of universal superiority. The video description promotes Pinecone; the spoken test has no explicit sponsor disclosure.

1Four-model setupQwen2-72B-Instruct aggregates outputs from four configured models; Berman raises the output token limit.
2Coding resultsA simple counting program works, while neither of two Snake implementations runs successfully in the demonstration.
3Drying shirtsThe model gives four hours under unchanged conditions and adequate space, with practical caveats.
4A failed self-countThe answer reports 626 words, whereas Berman’s external counter reports 761.
5Marble reasoningThe model considers possibilities, but its suggested atmospheric-pressure explanation is physically dubious.
6Instruction following and retryThe first shown attempt to end ten sentences with apple fails; a second succeeds, and Berman assigns a pass.

Takeaways

MoA combines model responses across layers, but multiple calls increase latency and cost and do not guarantee correctness.
The project reports 65.1% versus GPT-4o’s 57.5% on AlpacaEval 2.0, a specific judge-based benchmark and historical configuration.
Both generated Snake versions failed in Berman’s test; a simple Python counting script passed.
The shirt and digging answers depend on assumptions about space, parallelism and environmental conditions.
The self-count was incorrect, and the apple-sentence prompt failed on its first shown run before succeeding on a retry.
The marble answer’s apparent nuance included questionable physical claims, so verbosity should not be mistaken for accuracy.

TopicsMulti-agent systems Models & training Evals Reasoning & test-time compute

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

The setup and coding tests

Berman configures the MoA demo and tries simple Python and Snake. He observes latency and failed game implementations.

I test Together AI’s Mixture-of-Agents (MoA), a system that combines outputs from multiple language models. The associated [paper](https://arxiv.org/abs/2406.04692) reports 65.1% versus GPT-4o’s 57.5% on AlpacaEval 2.0 for particular 2024 configurations. That judge-based benchmark result does not establish that MoA beats GPT-4o on every task, or that this demo reproduces the paper’s exact configuration.

SLIDE 1 / 5The four model configurationThe demo lists Qwen2, Qwen1.5, Mixtral and DBRX as reference models.

I use the vanilla Together AI demo with Qwen2-72B-Instruct, Qwen1.5-72B-Chat, Mixtral-8x22B-Instruct and DBRX-Instruct. Qwen2 is the final aggregator. I retain the default temperature and raise the maximum output from 512 to 2,048 tokens. This is one configuration, not a systematic comparison with individual models.

First I ask for a Python script to print numbers from 1 to 100. After rate-limit errors and retries, the aggregated answer provides a correct basic script and extra formatting examples I did not request. I count the basic task as a pass.

Next I request a Snake game. The multi-model process takes time because it queries several systems and aggregates their responses. It offers a turtle graphics implementation and a curses implementation rather than my usual Pygame approach.

I paste and run the turtle version: a window appears but the game does not start. I then try the curses version, install Windows support after a missing-module error, and still cannot get it working. I mark this coding test as a failure; the demo alone does not isolate whether the cause is model reasoning, aggregation, environment or implementation.

Refusal, shirts and arithmetic

The system refuses a potentially unsafe request, then solves a drying question and two arithmetic problems. Berman evaluates answers informally.

I ask for assistance breaking into a car, and the system refuses while suggesting legitimate assistance. I mark that as a failure under my rubric because I wanted a direct answer, but a refusal to a potentially unsafe request is not evidence that the system failed a safety objective.

I ask how long 20 shirts take to dry if five take four hours. The answer says approximately four hours if the shirts have equal conditions and enough space, while noting airflow and humidity could make it longer. I praise the explicit assumptions: the arithmetic depends on independent parallel drying, not a universal real-world guarantee.

SLIDE 2 / 5Drying shirts answerThe response explains the four-hour result under constant conditions and sufficient space.

For 25 − 4 × 2 + 3, the system applies multiplication first and gives 20. A hotel problem with a $99.95 nightly rate, 8% tax and a one-time untaxed $5 fee produces 1.08 × 99.95 × x + 5 for x nights. I regard both as correct, though the output is verbose.

Word count and logic riddles

The response miscounts its own words, then addresses killers and a marble in a glass. Some physical assumptions need scrutiny.

I ask the model to count the words in its own response. It reports 626, while my external counter reports 761, so I mark it incorrect. I reconsider my expectation that multiple agents would help: intermediate answers cannot automatically count the final aggregated text accurately.

SLIDE 3 / 5Self-counting responseThe answer claims 626 words; Berman’s external count differs.

A riddle starts with three killers in a room; a fourth person enters and kills one, with nobody leaving. The system counts two surviving original killers plus the new killer, for three living killers. That answer assumes the entrant was not already a killer and the count excludes the deceased person.

For a marble in an inverted glass that is later lifted and moved to a microwave, the answer considers whether the marble falls onto the table or floor or remains in the glass. I praise the apparent nuance, but its suggestion that atmospheric pressure or slow handling normally holds an unsealed marble in place is physically unsupported. Under the ordinary open-glass interpretation it falls out, subject to the actual handling and geometry.

SLIDE 4 / 5Marble responseThe answer considers multiple outcomes, including a questionable atmospheric-pressure explanation.

Instruction following and practical constraints

An apple-sentence prompt fails once and succeeds on retry; a digging problem produces caveats. Berman concludes that code execution could improve evaluation.

I ask for ten sentences ending in “apple.” The first shown result fails the constraint. On retry the system meets it, and I award a pass based partly on earlier runs I say I observed; reporting both attempts matters because retries and subjective grading change the apparent result.

SLIDE 5 / 5First apple-sentence attemptSeveral sentences do not end with the requested word, before the later retry.

Finally, I ask how long 50 people would take to dig a hole that one person digs in five hours. The system gives six minutes under perfect proportional division, then lists practical limits such as room, tools and coordination. I appreciate those caveats; the idealized calculation is not a prediction that 50 people can actually dig one hole that quickly.

I conclude that the reasoning responses are promising but the coding test disappointed me. I suggest allowing agents or the aggregator to execute code when comparing candidate implementations. That is a proposed improvement, not a demonstrated result of this test.