AutoGen Advanced Tutorial: Configuration, Memory, Retrieval and Tools

In this October 2023 advanced AutoGen tutorial, I walk through agent model configurations, caching, enhanced inference, retrieval, teachable agents, Docker, function calling and reusable tools. The examples reflect AutoGen as demonstrated then; APIs and defaults have changed, so consult the documentation for the version you install.

An agent’s `llm_config` contains model-related settings, including a `config_list`. In the demonstrated implementation, the list can contain several model configurations tried in sequence when earlier requests fail. The same concept can be used for assistant agents and user proxy agents.

A prompt template can use values supplied through a context dictionary. Instead of rewriting an instruction, provide a variable such as a problem statement and reuse a fixed surrounding prompt. This is useful when multiple agents share an instruction structure but need different inputs.

The tutorial’s configuration options include enabling or disabling caching, a list of model configurations and function definitions. The shown documentation also describes how rate-limit and timeout failures can be handled, including an option to return a failure indicator after configurations are exhausted. Exact behavior depends on the version and settings.

Retry settings can specify a total retry period and a wait interval. I note that an increasing backoff would be useful, but the example I show uses an interval. Other model parameters, including temperature, token limits and sampling options, can be passed through where supported.

A cache seed selects a namespace for stored responses in the historical example. Repeating an identical cached request with the same seed can avoid another inference call; changing the seed or clearing the relevant cache forces a fresh result. This is a cache mechanism, not a general guarantee that independent model runs are deterministic.

The configuration list can be written in code or loaded from a JSON file and filtered for selected models. My example tries a GPT-4 configuration, an Azure-hosted configuration and then GPT-3.5 if earlier choices fail. Each entry specifies the model and connection details appropriate to its provider.

An OpenAI-compatible endpoint can also point to a locally hosted or remote open-weight model. In my example, Code Llama and Mistral run on separate local endpoints, and I assign the coding model to a coder agent and the other model to a product-manager agent. The servers and hardware must actually support the chosen models.

Different agents can use different model settings, and a team may combine hosted and local models. That flexibility can match a model to a role, although performance, compatibility, latency and available hardware determine whether a particular arrangement is practical.

The cache example places saved responses under a seed-specific directory. For repeated tests, a cache hit saves time and potentially API costs by returning a stored result; deleting a cache directory or changing its seed lets the application request a new response. Cached results can also conceal model or prompt changes if the application is not configured carefully.

AutoGen’s historical enhanced-inference interface is distinct from its multi-agent conversation framework. It offered a wrapper around completion calls with features such as caching, configuration fallback, error handling, templating and tuning. The description of it as a drop-in replacement applies to the API shown in 2023, not automatically to newer SDKs.

Performance tuning searches for settings that perform well under an inference budget. The example separates validation data, an evaluation function, a metric, a search space and a budget. For a math task, a user-defined evaluator can compare an answer with a known solution and report success.

The tutorial’s tuning example maximizes an aggregate success metric across test cases while exploring parameters such as model, prompt and token settings. It shows separate inference and optimization budgets. Results are only as meaningful as the validation data, evaluator and search space supplied.

API unification makes it possible in the historical interface to work with several hosted or local model endpoints through a common configuration. Fallback tries another entry when a prior one fails; it does not guarantee that every provider supports identical parameters or produces equivalent results.

Templating avoids adding the same instruction manually to every prompt. I use “explain your reasoning step by step” as an example of a reusable suffix, with the task inserted through context. That instruction helped in some of my tests, but it is not a universal guarantee of better answers.

The demonstrated logging interface records call history for debugging, and the documentation at the time marked it experimental. Next I turn to retrieval-augmented generation: an agent can use external documents to provide relevant context beyond the model’s initial knowledge.

The retrieval example combines a `RetrieveAssistantAgent` with a `RetrieveUserProxyAgent`. A retrieval configuration specifies a question-answering task and a document path. The example chunks the document, embeds the text and retrieves relevant content so the agent can answer a question about AutoGen.

A comparison without the retrieved document returns a generic answer about a different meaning of “autogen.” Retrieval can supply more relevant context, although the quality of the final answer still depends on the source, retrieval and model behavior.

The example’s embedding function can be customized, as can text splitting. I show alternatives using OpenAI or Hugging Face embeddings and a LangChain text splitter. Embeddings turn text into vectors that can be compared for relevance, but they are one component of a retrieval system.

The shown default uses Chroma, while another example customizes retrieval for Qdrant by providing its own client and retrieval logic. In a group chat, a retrieval-enabled user proxy can supply information while a separate proxy handles code execution and other assistants take on coding roles.

The tutorial then introduces a teachable agent that stores selected information as persistent memos and retrieves them in later conversations. This is external memory rather than a guarantee that the underlying model weights have been retrained or that every memory will be recalled correctly.

The historical `teach_config` example includes verbosity, whether to reset the database, its directory, a recall threshold and a maximum number of retrievals. A higher distance threshold can retrieve more memories, potentially at the expense of relevance. Resetting the database discards prior stored memos for that example.

In the demonstration, a user teaches facts about Vicuna and Orca, clears the conversation history, and then asks the agent to compare them using stored memory. The example illustrates persistence, not independent fact-checking: Vicuna was developed by LMSYS as a LLaMA-based fine-tune, rather than being simply a model released by Meta as the spoken example suggests.

The same mechanism can store a user preference, such as a three-bullet abstract summary. Feedback is converted into a memo that may be retrieved in a later task, rather than requiring the user to repeat the preference in every conversation.

For generated code, I recommend isolation using Docker and show a code-execution configuration with a Python container and working directory. Installing the Python Docker package alone is not the same as installing and running Docker itself. The video’s claim about automatic fallback to host execution is version-specific; current behavior must be checked rather than assumed.

Function calling lets an agent propose a structured call to a function defined by the application. In the example, the configuration describes Python and shell functions, including their names and parameters, while ordinary application code implements them.

The user proxy registers a mapping from the declared names to the actual functions. The model can suggest a call, and the application executes the registered implementation and returns its result. This division matters: the model does not gain a tool merely by naming it, and code execution needs appropriate restrictions.

Another example registers a Wolfram query function for a mathematical inequality. The assistant suggests the function and supplies the required query, after which the application executes it and returns the result. Tool outputs still need to be interpreted and checked for the task at hand.

Finally, I show agents working through a research workflow: search arXiv papers about trust calibration, analyze the application domains and generate a bar chart. When code fails, the user proxy feeds back errors and the agents revise it.

A follow-up asks the assistant to reflect on the successful sequence and produce a reusable recipe and generalized functions. That recipe could be supplied in a later task or stored for retrieval by a teachable agent. The aim is to reuse a tested workflow, while still reviewing generated code and results.