Matthew Berman Adapts Mixture of Agents to Groq Inference

Mixture of Agents combines responses from several language models to produce an answer. I am excited by reported results in which an ensemble of open models outperformed GPT-4o on a particular evaluation, but repeated model calls can make the approach slow. In this July 2024 tutorial I adapt the original code to use Groq’s fast inference service; Groq partnered on the video.

The illustrated architecture passes the prompt through several layers of models, feeding intermediate responses forward before an aggregator produces the final answer. The diagram shows three agents in each of three intermediate layers and a final aggregator. Results depend on the models, prompts and evaluation; the demonstration does not establish superiority to GPT-4o on every task.

I start with Together AI’s open-source MoA repository and make changes to use Groq. This is a pragmatic adaptation rather than a full redesign, and I expect that model choices and provider support will evolve. The code, model identifiers and service interfaces shown reflect the 2024 tutorial.

In VS Code, open a terminal and clone the repository. I create a separate conda environment with Python 3.11, activate it, enter the MoA directory and install the repository requirements with pip. Then I open the project folder and verify the environment is active.

I create a .env file for credentials. The original tutorial discusses an OpenAI key because the repository also has OpenAI benchmark scripts and uses the OpenAI Python library for compatible API calls. The video description later clarifies that a valid OpenAI API key is not required for this tutorial; the library and an actual OpenAI credential are separate things.

I obtain a Groq API key from its console and put it in the environment file. I say the keys shown in my recording will be revoked before publication. The two files containing the main changes are bot.py and utils.py; my modifications are somewhat hacky but sufficient for the demonstration.

In bot.py I replace the default reference models with Groq-supported options available at the time: Llama 3 8B, Llama 3 70B, Mixtral 8x7B and Gemma 7B. I choose Llama 3 70B as the default aggregator and update the corresponding model references. Other combinations can be tested as service support changes.

I also change the maximum token defaults to 2,048 while retaining the existing temperature setting. In utils.py, I load the .env file with python-dotenv and install that dependency. This lets the script read the configured environment variables rather than requiring them to be exported manually.

Next I replace Together’s API endpoint and key references with Groq’s OpenAI-compatible endpoint and Groq API key. The endpoint path includes /openai/v1. I make the changes in both the ordinary and streaming calls, then save the files and test the interactive bot.

Running python bot.py prompts for an aggregator model, temperature and maximum tokens. The displayed prompt still says 512 even though the actual default is 2,048; I later correct that label. My first joke request reaches the models but encounters an error when a streaming output is None and is concatenated to a string.

In bot.py I add a check that skips a None value before concatenation, and update the outdated maximum-token prompt. After restarting the program and accepting the defaults, I ask for a joke and receive the familiar bicycle punchline. This confirms that the demonstrated request completes after the fix, not that all failure modes are handled.

I next ask for ten sentences ending in the word “Apple.” The program queries the reference models and aggregates their responses; I report that the sentences satisfy the constraint. The two examples show a working interactive workflow, but they are not a controlled comparison of latency, cost or answer quality against GPT-4o.