Testing Mixture-of-Agents on Coding, Arithmetic and Reasoning

I test Together AI’s Mixture-of-Agents (MoA), a system that combines outputs from multiple language models. The associated [paper](https://arxiv.org/abs/2406.04692) reports 65.1% versus GPT-4o’s 57.5% on AlpacaEval 2.0 for particular 2024 configurations. That judge-based benchmark result does not establish that MoA beats GPT-4o on every task, or that this demo reproduces the paper’s exact configuration.

I use the vanilla Together AI demo with Qwen2-72B-Instruct, Qwen1.5-72B-Chat, Mixtral-8x22B-Instruct and DBRX-Instruct. Qwen2 is the final aggregator. I retain the default temperature and raise the maximum output from 512 to 2,048 tokens. This is one configuration, not a systematic comparison with individual models.

First I ask for a Python script to print numbers from 1 to 100. After rate-limit errors and retries, the aggregated answer provides a correct basic script and extra formatting examples I did not request. I count the basic task as a pass.

Next I request a Snake game. The multi-model process takes time because it queries several systems and aggregates their responses. It offers a turtle graphics implementation and a curses implementation rather than my usual Pygame approach.

I paste and run the turtle version: a window appears but the game does not start. I then try the curses version, install Windows support after a missing-module error, and still cannot get it working. I mark this coding test as a failure; the demo alone does not isolate whether the cause is model reasoning, aggregation, environment or implementation.

I ask for assistance breaking into a car, and the system refuses while suggesting legitimate assistance. I mark that as a failure under my rubric because I wanted a direct answer, but a refusal to a potentially unsafe request is not evidence that the system failed a safety objective.

I ask how long 20 shirts take to dry if five take four hours. The answer says approximately four hours if the shirts have equal conditions and enough space, while noting airflow and humidity could make it longer. I praise the explicit assumptions: the arithmetic depends on independent parallel drying, not a universal real-world guarantee.

For 25 − 4 × 2 + 3, the system applies multiplication first and gives 20. A hotel problem with a $99.95 nightly rate, 8% tax and a one-time untaxed $5 fee produces 1.08 × 99.95 × x + 5 for x nights. I regard both as correct, though the output is verbose.

I ask the model to count the words in its own response. It reports 626, while my external counter reports 761, so I mark it incorrect. I reconsider my expectation that multiple agents would help: intermediate answers cannot automatically count the final aggregated text accurately.

A riddle starts with three killers in a room; a fourth person enters and kills one, with nobody leaving. The system counts two surviving original killers plus the new killer, for three living killers. That answer assumes the entrant was not already a killer and the count excludes the deceased person.

For a marble in an inverted glass that is later lifted and moved to a microwave, the answer considers whether the marble falls onto the table or floor or remains in the glass. I praise the apparent nuance, but its suggestion that atmospheric pressure or slow handling normally holds an unsealed marble in place is physically unsupported. Under the ordinary open-glass interpretation it falls out, subject to the actual handling and geometry.

I ask for ten sentences ending in “apple.” The first shown result fails the constraint. On retry the system meets it, and I award a pass based partly on earlier runs I say I observed; reporting both attempts matters because retries and subjective grading change the apparent result.

Finally, I ask how long 50 people would take to dig a hole that one person digs in five hours. The system gives six minutes under perfect proportional division, then lists practical limits such as room, tools and coordination. I appreciate those caveats; the idealized calculation is not a prediction that 50 people can actually dig one hole that quickly.

I conclude that the reasoning responses are promising but the coding test disappointed me. I suggest allowing agents or the aggregator to execute code when comparing candidate implementations. That is a proposed improvement, not a demonstrated result of this test.