Testing Gemini 1.5 Pro on Code, Reasoning, Images and Video

I’m testing Gemini 1.5 Pro in Google AI Studio in August 2024, including text, images and a long video. Google described an updated context window of up to two million tokens and published benchmark results. Those reported figures are vendor claims for specific evaluations, not a guarantee for every task or input.

The interface offers the standard Gemini 1.5 Pro and an experimental 0801 preview. I start with the experimental version and switch between them when results are poor. The settings and preview status are specific to the historical test and may not describe later versions.

A Python script to print numbers from one to 100 looks correct, and the model also provides an explanation. For Snake, however, both versions repeatedly return output errors or blocked output. A later complete response uses the curses library, but the code produces an error when I run it; repeated attempts leave me unable to get a working game.

I then test a request about entering a car. The model refuses the direct request and responds differently when asked about historical methods. I treat that as a refusal test, but a single prompt and response cannot establish the model’s overall safety behavior or the reason an output was blocked.

For the shirt-drying puzzle, the experimental version assumes groups of five and answers 16 hours. I give it a marginal pass in the video for consistency with earlier tests, but if all 20 shirts can dry simultaneously under the same conditions, four hours is the natural answer. The standard version supplies that answer and notes the role of sunlight, humidity and airflow.

When asked how many words are in its own response, the standard model gives an incorrect count, while the experimental response gives a correct one in this attempt. I suspect luck may play a role and question whether this prompt tests much. One trial is not enough to estimate reliability.

In the killers-in-a-room puzzle, both responses miss that the newcomer who kills someone also becomes a killer. The experimental version discusses the dead person remaining in the room. The wording leaves room for disagreement about whether dead people count among the killers, so the expected answer depends on that interpretation.

In another puzzle, a marble is put in a glass, the glass is inverted onto a table and then moved to a microwave. Both versions say the marble remains in the glass. I expect it to stay on the table after the glass is lifted, although the exact sequence and handling of the glass are implicit in the prompt.

I ask for ten sentences ending with the word “apple.” The experimental response gets seven of the ten right by my count, while the standard response also misses some endings. Both versions correctly say 9.9 is larger than 9.11 and provide explanations; these examples do not amount to a broad benchmark.

For an ethical hypothetical about gently pushing someone to save humanity, the standard model discusses arguments but declines repeated requests for a yes-or-no answer. The experimental version answers yes. I prefer an explicit answer with justification, but this is my grading preference for a philosophical scenario, not an objective measure of moral ability.

On vision, I show a meme contrasting startup workers with big-company supervisors. The model recognizes the exaggerated comparison, and I mark it as a pass. It also converts a screenshot of a table to CSV in a way that appears correct on my inspection; the video does not establish accuracy across other images.

Finally, I upload a 30-minute museum video, shown in the interface at roughly 531,000 tokens, and ask what it contains. The model describes a paleontology exhibit and identifies it as the Carnegie Museum of Natural History, while the visible file label begins “American Museum of Natura…”. The demonstration does not resolve that discrepancy.

I ask for the first dinosaur’s name. The model says it identified a sign later in the video, and I find that cross-reference impressive, but I do not independently verify the sign or the answer. I conclude that the model did well on vision and long context but underperformed my usual text tests; that is my judgment from a small informal sample, not proof that it is the worst frontier model.