Google’s Gemini Launch: Multimodal Demos, Benchmarks and Research Applications

Google has announced Gemini, and I want to examine its research report and demonstrations against the models available in December 2023. Google describes Gemini as natively multimodal, trained to handle different kinds of information. Its benchmark claims are historical comparisons under particular settings, not proof that it beats GPT-4 at every task.

I begin with a promotional sequence in which a person draws a bird and a voice identifies it as a duck. I describe the interaction as real time, but that impression needs correction: Google’s account of how the clip was made shows that the model received selected still images and text prompts, with responses and timing edited for the presentation. The sequence is not evidence of uninterrupted live video and voice processing.

The demonstrator colors the drawing blue, then presents a blue toy duck, and the displayed response jokes about the coincidence. I enjoy the apparent personality and discuss the question of whether the toy floats. A squeak alone does not prove an object will float; the example is a selected promotional interaction, not a physical test of every toy.

The clip also shows a map-based guessing game with country clues, followed by a game tracking paper under cups. I take this as spontaneous understanding without instruction, but the construction details matter: the demonstration used selected visual inputs and text prompts. It does not show that the model independently inferred all rules from an uninterrupted video stream.

Google’s original Gemini report introduces Ultra, Pro and Nano, model sizes intended for needs ranging from demanding reasoning to memory-constrained devices. Google reports leading results on 30 of 32 selected benchmarks. These are the company’s reported evaluations against the baselines and test settings in the report, rather than a guarantee for all tasks or current model versions.

The report describes joint training on text, images, audio and video. I interpret Ultra as the largest tier, Pro as a more deployable middle tier, and Nano as the device-focused tier. My suggestion that ordinary users could simply run Pro locally is speculation; the announcement did not provide publicly downloadable Pro weights or promise that setup.

One report example asks Gemini to read a handwritten physics problem about a skier, check the student’s calculation and explain the result using mathematical notation. The displayed answer identifies an error and provides a corrected calculation. I find the combination of image reading, reasoning and formatting compelling, though a selected example does not establish reliability across all handwritten work.

The report also introduces AlphaCode 2, a competitive-programming system combining a specialized Gemini model with search and tools. Google reports a ranking within the top 15% of participants in its Codeforces evaluation, compared with roughly the top half for its predecessor. That result concerns the full system and its evaluation conditions, not a claim that a bare model or every programming task reaches that rank.

Gemini Nano is designed for on-device applications such as summarization and text completion. The report describes Nano-1 and Nano-2 at approximately 1.8 billion and 3.25 billion parameters and four-bit quantization for deployment. On-device operation can avoid a network call for supported functions, but device compatibility and whether a particular feature works offline depend on the product.

The report describes Transformer-based models served on Google’s TPU infrastructure and a 32,768-token training sequence length. Ultra is positioned for more complex work and Pro for a balance of capability, cost and latency. These are Google’s design descriptions; model access and hardware requirements were product-specific rather than a promise of local access to all tiers.

Google says Gemini can handle text interleaved with visual or audio inputs and describes text and image outputs in the research setting. It explains video understanding by encoding frames into the context window. Research capabilities, promotional examples and features available in a particular user-facing product are separate questions.

On training data, the report mentions web documents, books, code, images, audio and video, along with quality and safety filtering. I speculate that Google could benefit from services such as Search, Gmail and YouTube, but the report’s broad data categories do not establish that private Gmail contents were used. The exact sources and permissions are not identified in the excerpt I discuss.

The report asks whether joint multimodal training can perform well across domains compared with models specialized for one kind of task. Google reports that Gemini Pro is competitive with models available at the time and that Ultra leads on many of its selected tests. Model version, prompting, evaluation design and cost all matter when comparing those results.

On MMLU, Google reports 90.04% for Gemini Ultra with a particular chain-of-thought and multi-sample procedure, compared with the benchmark authors’ 89.8% human-expert reference. I extrapolate that result to every expert and every domain, but that is not supported: a multiple-choice benchmark and a reference score do not establish superior real-world expertise across all fields.

Google attributes the MMLU result to a prompting approach that uses multiple reasoning samples and a confidence threshold. The report also gives 94.4% on GSM8K using chain-of-thought and self-consistency and 53.2% on the MATH benchmark under the stated prompting conditions. Comparisons are meaningful only when their methods and compute budgets are understood.

I examine Google’s text-and-reasoning benchmark table and note that GPT-4 scores higher on HellaSwag while Gemini Ultra leads on many other displayed entries. Some rows use different prompting procedures. The table provides a snapshot of selected 2023 evaluations, not a universal or permanent verdict about either model.

The report also discusses benchmark contamination: evaluation items can appear in training material, making it harder to interpret a model’s score as generalization. Google says it performed leak analysis and omitted some results where it found issues. Such checks can reduce a source of uncertainty but do not prove every benchmark is free of contamination.

I am particularly interested in Nano because models running on a device could support local, potentially offline experiences. The report describes results in summarization, comprehension, factual retrieval, reasoning and multilingual tasks, often with task-specific fine-tuning. I would prefer an open release of weights and training data, but that is my preference rather than a feature Google promised.

The report includes translation results and a long-context test. Google says the original Gemini models were trained with a 32,768-token sequence length; in a synthetic task, Ultra retrieved a value paired with a key across long filler text with 98% accuracy. That result is specific to the tested retrieval setup and does not establish the same accuracy for every long document or complex reasoning task.

I return to AlphaCode 2 and the idea of combining a specialized Gemini Pro with search and tools to tackle multi-step programming problems. As someone who uses AI while coding, I welcome another potential assistant. A competitive-programming result does not by itself establish performance on everyday repositories or production software.

Google’s report compares image-understanding benchmarks and presents MMMU questions across academic disciplines. I regard the results as an encouraging launch for Google, but the reported comparisons are bounded by the models, prompts and tasks evaluated. They do not show that Gemini is better at every visual question.

One illustrated example asks the model to identify four plotted functions and write Matplotlib code to rearrange their positions. The displayed output identifies a sine wave, tangent, exponential function and 3D paraboloid and produces code for the requested arrangement. The example combines image interpretation, instruction following and coding, but is a selected demonstration.

Other examples suggest craft ideas from colored yarn and provide step-by-step advice for a vegetable omelet from images. I imagine a personal assistant that could use such visual context, perhaps through a phone or wearable. That is my vision of a possible product, not evidence that a device continuously observes a user or that the examples work reliably in every kitchen.

A Google research video describes updating a genetics literature dataset derived from a 2022 review. The researchers say more than 200,000 relevant-domain open-access papers had appeared since 2021 and demonstrate prompting Gemini to identify candidates and extract fields with annotations. This describes a workflow for one research problem, not a general replacement for scientific review.

In the demonstration, Google says it ran screening at scale over a lunch break, narrowed the corpus to roughly 250 papers and extracted information from selected results. I characterize that as the model fully reading and ranking every paper in an hour, but the displayed filtering prompt includes titles and abstracts and the clip does not establish full-text reading of all 200,000 or independently validated recall and extraction accuracy.

Another promotional example shows Gemini identifying mistakes in a student’s mathematics solution and explaining the height calculation. I connect this to broader speculation about improvements in math reasoning, including Q*; the video does not establish what Q* is or how it relates to Gemini. A selected tutoring example should be checked against the underlying problem and answer.

Finally, a Google demonstration asks Gemini to turn a tree image into SVG and then generate an interactive JavaScript fractal tree with a slider. I am impressed by the movement between images and code and look forward to testing the models myself. At the time of the video, Bard was beginning to incorporate Gemini; product availability and capability should be understood in that 2023 context.