Google I/O 2024: Gemini, Astra and AI Agents Across Google's Products

Google I/O 2024 was full of AI announcements. I want to watch the most interesting parts together, from Gemini in Google products to a multimodal assistant demonstration, and compare the ideas with what OpenAI showed the day before.

Google is doubling down on Gemini in Docs, Sheets, Gmail and Workspace, although some announcements sound familiar. What stands out to me is the long context window: Gemini 1.5 Pro supports up to a million tokens, and Google is discussing two million. A large window enables new uses but does not by itself guarantee perfect retrieval or reasoning.

Sundar Pichai describes Gemini as natively multimodal across text, images, video and code. Google also wants to expand AI in Search, its core business. I think the competitive pressure is obvious, and I expect people to ask longer, more complex questions and search using photos.

One of my favorite products is Google Photos. Google demonstrates Ask Photos: instead of searching manually through years of pictures for a license plate, you can ask a question and Gemini uses the context in the photos to find the number. This is a product demonstration, and the feature was announced as experimental.

Another example asks when a child learned to swim and how her swimming progressed. The demo combines images, activities, dates and certificates into a summary. I already enjoy automatically assembled albums and memories, so being able to ask for a personalized version is something I would use.

The long context window excites me especially for coding: a million tokens can hold a large codebase. Google discusses expanding the window to two million; I also mention hearing that larger windows were being explored internally. That is not the same as a publicly available product or a guarantee that the model will use every detail correctly.

Google has an advantage in the services people already use. Gemini could draw on email, documents and presentations to answer questions and eventually help with tasks. I find this more useful than a model in isolation, although access and actions depend on product permissions and availability.

The keynote is polished and scripted, whereas I preferred OpenAI's more informal presentation. In Google's Gmail example, Gemini summarizes recent messages and PDF attachments from a child's school. Having a way to search and summarize a large mailbox is particularly appealing to me.

The demonstration also describes summarizing a long Google Meet recording when a parent misses a meeting. I would like an assistant to go further: notice that I was traveling, prepare a relevant summary and suggest a follow-up, with me still deciding whether to send it.

Google shows Gemini drafting a reply to volunteer for a parents' event. I would prefer not to issue a separate prompt for every such step: the assistant could prepare a draft from context for me to review and approve. That proactive workflow is my desired future behavior, not proof that the demo already does it.

NotebookLM organizes documents, PDFs and notes so a user can ask questions against them. In the demonstration, a parent adds science worksheets, slides and a textbook; the tool can create a summary, study guide, FAQ and quizzes.

Google previews an audio overview that turns the source material into a conversational discussion about physics. I like the idea, though I have not yet found my own use case. An impressive technical feature matters to me when it solves a real need.

Google defines agents as systems that reason, plan and remember across multiple steps and software systems while remaining under user supervision. Its first example is shopping and returning shoes, including finding the receipt, filling a form and arranging a pickup. I find the task less compelling than other work, even though the sequence is useful.

A second imagined workflow involves moving to Chicago: Gemini and Chrome could help find services and update an address on many sites. Finding things to do is ordinary, but updating an address across sites would save significant time. Google presents this as a prototype and says it would request more information when needed.

Demis Hassabis introduces Gemini 1.5 Flash, a smaller model intended for fast and cost-efficient multimodal tasks. Google says Flash and Pro are available with up to one million tokens in its developer tools, with a sign-up for testing two million. These are announcements from the 2024 event.

Google then presents Project Astra, an experimental multimodal assistant. I compare it with the conversational experience OpenAI showed for GPT-4o. Google presents the demonstration as a prerecorded single take at normal speed; even a compelling demo does not measure reliability across every situation.

In the Astra video, the assistant identifies a speaker and its tweeter, produces an alliteration about crayons and explains encryption code shown on a screen. It then identifies an area of London and recalls seeing glasses on a desk near an apple. I find the seamless transition between voice, visual context and memory striking.

The presentation moves to glasses with a camera. The assistant suggests adding a cache between a server and database, responds to another visual question and comes up with a band name. I would like glasses that support a natural back-and-forth like this; I don't know what hardware is being shown.

Google also announces Imagen 3 and generative music, but my attention is on video. Veo is presented as a model for generating 1080p video from text, image and video prompts, with cinematic directions and editing by further prompts. Google describes VideoFX as an experimental tool.

A generated driving scene looks consistent as the cars move through space, and I compare the result to demonstrations of Sora. A selected clip shows what the system can produce, not that every generated scene will have equally reliable motion or consistency.

Google returns to AI Overviews in Search. A complex query about yoga and Pilates studios brings together ratings and introductory offers. I see it as Google's attempt to make its search product more useful for questions that require several pieces of information.

A Workspace side-panel demonstration starts with a hotel receipt attached to an email. Gemini proposes finding the receipt and 37 others, creating a Drive folder and extracting vendor, date and cost into a spreadsheet. The user can edit or approve the proposed actions.

The demo then suggests automating the same workflow for future emails. This is the sort of agent behavior I want: explain a recurring task once and have it continue. The presentation shows a proposed experience, not an independent assessment of how reliably it works across every inbox.

Google also demonstrates a virtual teammate called Chip with its own account, a role and instructions to monitor projects. I like the ability to reference external documents and other context directly when configuring it.

In Google Chat, Chip searches project conversations to answer a question about event storyboards. I don't personally use Google Chat, but the broader capability—answering from documents, email and conversations—is valuable regardless of the interface.

That integration also creates a tension for businesses. Open-source tools may offer flexibility across services and reduce dependence on a single platform, but they still need access to the underlying data. If the providers do not expose it, an open system cannot simply assume it can retrieve it.

Google also discusses AI features on Pixel phones. For the closing joke, Pichai refers to last year's viral count of how often 'AI' was mentioned and lets Google count this year's mentions. That wraps up the announcements I found most interesting.