Google’s 2024 Vertex AI Agent Builder: A First Look at Agents, Workspace and Code Assist

Google launched an agent platform at Cloud Next 2024, and I want to watch the relevant keynote excerpts and show you what I have found in Vertex AI Agent Builder. This is a first look at the April 2024 preview, not a description of every feature Google may have added later.

Google says its Vertex AI Model Garden offers access to more than 130 first-party, partner and open models. I browse Gemini, Imagen, Gemma, Chirp and third-party models and show filters for modalities and tasks. A model page provides use cases, documentation and a route into Vertex AI Studio, which reminds me of a model catalog such as Hugging Face.

The keynote announces Gemini 1.5 Pro in public preview. Its context window of up to one million tokens is striking: Google describes processing an hour of video, 11 hours of audio, hundreds of thousands of words or tens of thousands of lines of code. Those are approximate examples, not a guarantee that all material will be interpreted accurately ([Google announcement](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/google-cloud-next-2024-generative-ai-gemini/)).

I mention reports of much larger experimental contexts and look forward to the use cases. Google’s research had already evaluated retrieval at up to 10 million tokens, but that research result was not a statement that a 10-million-token product was generally available ([Gemini 1.5 paper](https://arxiv.org/abs/2403.05530)). Many real codebases still exceed even a large context window, so retrieval and codebase mapping remain useful.

Google describes searching for a spoken phrase inside a video and extracting information from a long document containing text, tables and charts. I am excited about asking questions of long PDFs and podcasts, although the demonstration alone does not establish extraction accuracy across all materials.

Google also announces CodeGemma, a lightweight coding-oriented model. I was not impressed by my earlier experience with Gemma, but I appreciate the release of open models and want to test the coding variant. Google describes Vertex as a place to tune, augment, manage and monitor models.

When I hear “agent,” I think of tools such as AutoGen and CrewAI: agents given tools and roles that can cooperate on tasks. Google’s examples initially look more like customer-service assistants or custom GPTs. That is my impression of the preview and the keynote examples, not a comprehensive comparison of every capability.

Google describes customer agents that respond across web, mobile, point-of-sale and call-center channels, including voice and video experiences. Mercedes-Benz CEO Ola Källenius describes work with Google on sales assistance, customer service, marketing and navigation, as well as cloud infrastructure for development of conditionally automated driving.

Källenius says the company’s Level 3 driving system had been certified in Germany, California and Nevada and discusses Google Cloud as infrastructure for a next-generation development and testing platform. This is a statement about specific regulatory approvals and development work, not evidence that autonomous driving works everywhere or without the system’s operating conditions.

I wonder why the keynote did not highlight a conversational agent in the car’s infotainment system: when driving, speaking to an assistant could be much more useful than typing. That is my suggested use case, not proof that Mercedes had no voice assistant or planned no such feature.

Google presents examples from hotels, home security, telecoms, retailers, public services and financial services. These are mostly customer-facing support and sales applications, which I understand have commercial value but find less exciting than autonomous agents that carry out a wider range of tasks.

My former company used Dialogflow, Google’s earlier conversational product, and I found it brittle and difficult to set up. I can see why generative models might improve these experiences, while still wishing the new announcement were more ambitious. Google also describes translation, troubleshooting and knowledge-search examples.

Google introduces Vertex AI Agent Builder as a way to create customer agents: use Gemini for conversational and multimodal inputs, write natural-language instructions and decide when to hand off to a human. Google also describes grounding responses in internal data and search, plus extensions for tasks such as booking or updating contact information. These are announced capabilities rather than proof that each integration is configured automatically.

Developer advocate Amanda Lewis demonstrates a shopping assistant: she supplies a music video and asks for a checkered shirt like the keyboard player’s, with prices and availability. The demonstration uses Gemini to interpret the video and retrieve items from the retailer’s inventory. I like the multimodal input, though the shopping use case itself is not what I hoped agent building would center on.

I open my own weather agent in Agent Builder. The interface lets me name the agent, state its goal and write instructions such as asking for the user’s location; typing a dollar sign makes it easy to insert references to agents or tools. I like how direct this part of the interface feels.

My weather tool is configured as a function with input and output schemas. What I cannot find is where to put the code that actually calls a third-party weather API. When I ask for Los Angeles weather, the tool input is formatted, but the example output is zero and the assistant says it cannot provide the weather. My unsuccessful test shows my confusion, not that function calling can never execute external code.

I notice integrations including Dialogflow Messenger, Twilio and Discord and appreciate the availability of authentication and TLS options. The tool configuration appears to use structured schemas rather than the simple inline method I wanted. At the time, I find the workflow less straightforward than the custom function interface I expected.

The keynote then moves to employee agents: Google describes connecting models to company and web data, translating content and grounding answers in services such as AlloyDB, BigQuery and enterprise applications. A mentioned HubSpot acquisition is only a rumor in the video, not an announced transaction.

Developer advocate Gabe Weiss demonstrates a benefits assistant integrated into Google Chat. He asks it to summarize an open-enrollment email and a long attached video, with relevant email and attachment context retrieved for the prompt. I like the idea because I would welcome an assistant that reads my email context and drafts a reply for me to review.

The demonstration then compares medical coverage options against a PDF in Google Drive, links to supporting source material and generates a bilingual summary. The presenter describes Gemini’s long context, retrieval and extensions as the underlying pieces. This is a staged benefits demo, so source links and a generated recommendation should still be checked against the actual plan documents.

I am impressed by the Drive integration, but I did not see an obvious way to reproduce the complete experience from the Agent Builder interface I tried. I suspect Google connected some capabilities behind the scenes; I cannot establish from the demo which parts an outside developer could build at that time. I would like more of that functionality exposed to builders.

The keynote introduces Google Vids as an AI-assisted video-writing, production and editing app for work. In the demo, a presenter supplies a prompt and an existing document, edits an outline and chooses a style; Vids assembles a first draft with scenes, stock media, music and a script. I like the integration and note that the shown workflow assembles media rather than proving it generates all video footage from scratch.

Next is Gemini Code Assist with Gemini 1.5 Pro. In the example, a new developer is asked to move product recommendations to a storefront home page using a design mockup and a repository spanning multiple services. Google says the long context helps the assistant reason about the code and suggest coordinated changes.

The presenter supplies business requirements and a design reference; the assistant identifies the recommendation function and files to change. I appreciate the ability to add a Drive link and an image as context. The presentation shows proposed changes, an apply option and edits across a Go handler and an HTML file.

Google compares the demo with the time it would take to read an entire repository. I think that is marketing: developers do not necessarily need to read every line before making a change. The presenter tests the modified app locally and shows the recommendation section; that is one controlled demonstration, and proposed code still needs human review and tests.

The storefront example is fairly simple, but I am eager to try AI-assisted coding. My overall reaction is mixed: Agent Builder’s customer-agent focus and tool setup fall short of the more sophisticated agent framework I wanted, while Google’s Workspace integration and coding demonstrations are welcome.