Andrew Ng’s Four Agentic Workflow Patterns, Explained by Matthew Berman

Matthew Berman introduces a 2024 talk by Andrew Ng on agentic workflows. Berman is enthusiastic about agents and pauses the talk throughout to explain examples and add his own interpretation. He discloses in the video description that he is an investor in CrewAI.

Berman introduces Ng as a computer scientist, Google Brain co-founder, former Baidu chief scientist and Coursera co-founder. He then plays Ng’s presentation at Sequoia on how iterative workflows can change what a language model accomplishes.

Ng contrasts a single-pass prompt with an agentic workflow using an essay as an example. Instead of typing from beginning to end without revising, a system might outline, research, draft, review and revise. The key difference is iteration across steps.

Berman expands on this with a hypothetical writer, reviewer, spell checker and fact checker passing work among themselves. Different roles and tools can help divide the work, he argues, but his example is an elaboration on Ng’s general point rather than evidence that every role is needed.

Ng presents a HumanEval coding benchmark comparison. The slide reports 48% for zero-shot GPT-3.5 and 67% for zero-shot GPT-4, with several agentic approaches using GPT-3.5 scoring higher than the zero-shot GPT-4 result in that comparison.

Berman emphasizes the gap shown on the chart and points to reflection, tool use, planning and multi-agent systems. These figures compare particular benchmark setups, not a general demonstration that GPT-3.5 always matches or exceeds GPT-4 across tasks; workflows can also consume different amounts of time and computation.

Ng groups agentic patterns into reflection, tool use, planning and multi-agent collaboration. In his experience at the time, reflection and tool use were relatively robust, while planning and multi-agent collaboration could be impressive but were less consistently reliable.

Berman describes reflection as asking a model to examine its own output and propose improvements. For tool use, he gives examples such as web retrieval, financial-data lookups and mathematical libraries: the model can select and call external functions instead of relying only on its initial response.

For planning, Berman describes breaking work into steps before execution. In multi-agent collaboration, different prompted roles or different underlying models can critique and build on one another’s outputs. He is optimistic, but acknowledges that getting agents to behave as intended can require testing and iteration.

Ng demonstrates reflection with code: ask a model to write a function, feed its code back with a request to check correctness, style and efficiency, then use that critique to generate a second version. The result is not guaranteed to improve, but in his experience it works often enough to try.

Berman compares this with his own practice of running model-generated code, finding an error and sending it back for a fix. Ng adds that an agent can run unit tests, inspect failures and revise again, turning feedback into another pass through the workflow.

A variation separates the coder and critic into different roles. They can use the same base model with different prompts: one writes code, the other reviews it. Ng describes this as a broadly useful pattern, rather than a requirement to use multiple distinct models.

Ng’s tool-use examples include web search, code execution, productivity tools and image-related functions. Earlier language models that could not directly interpret images could still invoke external tools for tasks such as object detection or image generation.

Berman emphasizes that existing libraries and APIs can become tools for a model. Their outputs are not necessarily correct or deterministic in every setting, but calling a dedicated tool can give a workflow capabilities beyond text generation.

Ng describes planning systems that can select a sequence of tools and sometimes recover from a failed step. His example, adapted from HuggingGPT, combines pose extraction, image generation, image-to-text and speech synthesis to carry out a multi-part request.

Ng cautions that these systems were finicky and did not always work reliably in his experience. Berman agrees that failure recovery and repeated attempts can be valuable, and expects improvements in models, tools and frameworks to reduce some of the friction.

Ng says he sometimes uses a research agent to gather information when he does not want to search manually. It can work or fail, but has already become part of his personal workflow. He then turns to collaboration among multiple agents.

He describes ChatDev as an open-source example where prompted roles such as software engineer, designer and tester converse, write code and iterate on a program. He says results can be surprisingly complex, while noting that attempts do not always succeed.

Ng also mentions multi-agent debate, in which agents argue over an answer. Berman adds that different models or models specialized for different roles might provide varied perspectives. Neither claims that role separation guarantees a better result in every task.

Ng summarizes the four patterns and predicts that agentic workflows will expand the tasks AI can perform. He notes that users accustomed to immediate chat responses may need to wait minutes or longer for an agent to research, act and revise.

Berman argues that fast inference could shorten the iterative part of such workflows. He cites Groq’s token generation speeds as examples from the period and suggests that web searches or external APIs could become the bottleneck instead. These are illustrative estimates, not timing measurements for all agents.

Ng likewise suggests that fast generation from a somewhat weaker model may sometimes permit more iterations than a slower stronger model. He compares that idea with the benchmark example and frames agentic reasoning as a way to improve some applications before the next model generation arrives.

Ng describes progress toward AGI as a journey and sees agentic workflows as a possible incremental step. Berman reiterates his interest in agent workflows and inference speed. The available automatic transcript skips part of his closing commentary, so the missing passage is not reconstructed here.

Berman closes by noting questions about token costs and his expectation that pricing will evolve as models become more widely available. He thanks viewers and ends the discussion.