Google I/O 2024: Gemini, Project Astra and AI Across Google’s Products

At Google I/O 2024, CEO Sundar Pichai described Gemini as a model built to work across different kinds of information. He said more than 1.5 million developers were using the models and introduced examples spanning Search, Photos, Workspace, Android and the Gemini app. This video is an edited supercut of Google's May 2024 presentation, so its announcements and promised dates are historical.

Pichai said Ask Photos would roll out that summer, with more capabilities to follow. His larger point was that a multimodal model can connect information across formats rather than treat each input as an isolated task.

Gemini 1.5 Pro's one-million-token context was becoming available to Gemini Advanced users in 35 languages. Pichai also announced an expansion toward two million tokens. Google's developer announcement placed the two-million-token version on a waitlist, so the larger window should not be read as general availability at the time.

Josh Woodward demonstrated an early NotebookLM audio feature using material about Newton's laws. A generated dialogue between two voices explained force and motion; Woodward then asked for a basketball example for his son. The screen notes that the audio was pre-generated, and this presentation was a demonstration rather than evidence that the interaction always worked this way.

Pichai connected this sort of interaction to a broader vision of mixing input and output formats. He then defined an AI agent as a system with reasoning, planning and memory that can work across software to complete a multistep task on a user's behalf.

His shopping example was hypothetical: instead of manually returning ill-fitting shoes, a future agent might find a receipt and order number in email, complete a return form and arrange a pickup. The example illustrated the intended workflow, not an announced completed product.

Google DeepMind CEO Demis Hassabis introduced Gemini 1.5 Flash as a lighter model than Pro, intended for lower-latency, lower-cost applications while retaining multimodal and long-context capabilities. Its design trade-off was serving efficiently at scale, rather than presenting one model as ideal for every task.

Hassabis then introduced Project Astra, Google's prototype for a more universal assistant. The goal was an agent able to understand and remember what it sees, respond to a changing environment and converse naturally. Google presented a video in two parts, each said to be captured in a single real-time take.

In the first part, the demonstrator points a camera around a room. The prototype identifies a speaker and its tweeter, makes an alliteration about crayons, describes code on a screen and suggests the King's Cross area of London. Asked where it had seen the user's glasses, it recalls a desk near a red apple.

The demonstration continues with a system diagram: Astra suggests adding a cache between a server and database. It also responds to drawings and proposes a band name. These selected examples show the direction of the research prototype; they do not establish its reliability across arbitrary settings.

A Google presenter introduced Imagen 3, describing improvements in photorealism, detail, prompt following and visual artifacts. He said independent evaluators preferred it in some side-by-side comparisons and invited viewers to sign up for ImageFX; developer and enterprise access was planned. The preference claim was Google's account of those comparisons, not a universal ranking.

Google also described Music AI Sandbox, built with YouTube and artists to explore professional music tools, including generating instrumental sections and transferring styles. The presentation framed these as creative tools under development, with artists involved in shaping their use.

The presenter announced Veo, a generative video model that Google said could create 1080p video from text, image or video prompts, follow cinematic instructions such as aerial shots or time-lapses, and support further edits through prompts. The clips shown were selected demonstrations, not a guarantee of results on every request.

Google announced its sixth-generation TPU, Trillium. The company claimed a 4.7-fold improvement in peak compute performance per chip over TPU v5e and said it expected cloud availability in late 2024; that figure refers to a specific hardware comparison, not every workload's speedup.

The Workspace section said the Gemini-powered side panel was planned for general availability the following month. Google also highlighted automatic language detection and real-time meeting captions expanding to 68 languages, before turning to proposed Gemini 1.5 Pro features in Gmail mobile.

In a roof-repair example, the presenter opens a long email thread and uses a summary card to find a contractor's quote. Instead of searching separate messages by hand, she asks Gemini to compare bids by price and availability; the displayed response brings together details from different threads.

The presenter selects a contractor and shows a contextual Smart Reply. Gemini offers responses shaped by the thread and lets her preview a full draft before sending. Google said these Gmail capabilities would begin rolling out to Labs users that month, a 2024 rollout plan rather than a statement of present availability.

Looking toward 2025 and beyond, Google showed a prototype virtual teammate named Chip. In Google Chat, Tony asks whether I/O storyboards have been approved; Chip searches the conversations it was assigned to track and responds. The presentation says the teammate could build context as it joins more chats, files and email threads, while acknowledging substantial work remained.

The Gemini app presentation framed a personal assistant around text, voice and camera input. Google announced Gemini Live for later that summer, with more natural voice conversations and the ability to interrupt; it also intended to bring Astra's speed and video understanding to the app.

Google proposed opening a camera during a live conversation so Gemini could respond to the user's surroundings. It also introduced Gems: reusable customized instructions for a personal expert, such as a writing coach that specializes in short stories. Gems were described as coming in the following months, with trusted testers already trying them.

The Android section described a multiyear effort to put AI into the operating system: easier AI search, Gemini as an assistant and on-device models for faster and more private experiences. The speaker's privacy language describes Google's goal and architecture, not a blanket guarantee about all data handling.

Dave demonstrated Gemini at the system level while receiving a pickleball video in a message. Because the user was viewing YouTube, Gemini offered an 'Ask this video' action. His question about the two-bounce rule produced a short explanation, with YouTube captions among the signals used.

Next he opened an 84-page pickleball rulebook as a PDF. Gemini offered an 'Ask this PDF' action, answered a question about spin serves and pointed to a source in the document. The presentation used a specific document and example; its answer should be checked against the relevant rules and edition rather than generalized to every form of spin.

To illustrate on-device scam detection, Google staged a call from someone impersonating a bank employee and asking Dave to transfer money. Gemini Nano displayed a warning during the simulated call. It was a scenario demonstrating a proposed safeguard, not a measured fraud-detection accuracy or a promise to catch all scams.

In the developer section, Google said Gemini 1.5 Pro and the new Flash were available in preview across more than 200 countries and territories. The presenter also discussed video-frame extraction, parallel function calling and context caching; caching was planned for the following month to make repeated use of long inputs more practical and affordable.

The Gemma family was presented as open models derived from related Google research, with 2-billion- and 7-billion-parameter sizes and variants including RecurrentGemma and CodeGemma. Google introduced PaliGemma, an open-weight vision-language model for tasks including image captioning, visual questions and image labeling; its use is subject to its license terms.

Google announced a forthcoming Gemma 2 model with 27 billion parameters, planned for June 2024. The presenter said it was optimized for NVIDIA GPUs and could run on a single TPU host in Vertex AI, and claimed it could outperform models more than twice its size. No general benchmark or universal superiority follows from that stage claim.

The presentation closed by inviting developers to build with the announced tools. Across the supercut, Google moved between available previews, scheduled rollouts, staged examples and research prototypes; each should be understood in its original 2024 context.