Gemini Advanced and Ultra 1.0: A First Look and Hands-On Test

Google has launched Gemini Advanced with Ultra 1.0, and I think it could challenge ChatGPT. I want to look at the distinctive Google features and then try coding and reasoning prompts, including my usual test of whether the model can create Snake in one go. This is a first look at the February 2024 release, not a comprehensive evaluation.

Bard is now called Gemini. The product names can be confusing: Gemini Advanced is the paid experience that gives access to the Ultra 1.0 model, while the existing Gemini experience uses Pro 1.0. Google said Ultra 1.0 was stronger at complex coding, logical reasoning, nuanced instructions and creative collaboration; I want to test some of those claims.

Google said it planned to add expanded multimodal features, interactive coding and deeper data analysis. At launch the US Google One AI Premium plan cost $19.99 per month, the same general price as ChatGPT Plus at the time, and offered a two-month trial and 2 TB of storage. Google also said subscribers would later get Gemini in Gmail, Docs, Slides and Sheets, features previously known as Duet AI.

Google was rolling out a dedicated Android app and said Gemini access would come to iOS through the Google app in the coming weeks. On Android, an optional overlay could offer contextual help, including discussion of an image. I wonder whether Gemini will eventually replace Google Assistant, but that is my speculation rather than an announcement.

After subscribing, I select Gemini Advanced in an interface that reminds me of ChatGPT. The extensions page offers connections to Google Flights, Hotels, Maps, Workspace and YouTube. I see those integrations as a major possible advantage because they can bring relevant Google services and, where enabled, a user’s own content into the experience; the available connections and permissions matter.

For my first test I ask it to create Snake in Python. The response appears very quickly, but when I paste the code into Visual Studio Code I notice that “screen” is undefined, and the game does not open correctly. I tell Gemini about the error and the lack of a visible result; it suggests updates, including a loop, but the revised program only opens a blank window.

Next I ask how long 20 shirts take to dry if five shirts laid out in the sun take four hours. Gemini correctly says approximately four hours, assuming they can all dry in parallel under comparable conditions. For a self-referential word-count question it suggests using a word counter rather than answering; GPT-4 also gives an incorrect count in my comparison.

In the killer puzzle, three killers are in a room and someone enters and kills one. Gemini answers that two killers remain, overlooking that the entrant is also a killer; on the intended interpretation, three are still alive in the room. It then succeeds on a perspective-taking puzzle: John thinks a ball remains in the box where he left it, while Mark knows he moved it to the basket.

The interface includes controls to make an answer shorter, longer, simpler, more casual or more professional without rewriting the prompt. A double-check control searches Google, and users can also see alternative drafts or have the response read aloud. I like the flexibility, though it is worth asking which of these features prove useful in practice.

For an image test I upload a picture showing a stylized llama and the Meta logo. The first draft incorrectly describes an image about Google’s LaMDA and invents text that is not present. A second draft recognizes a blue-and-white llama and is much closer to what is shown. My brief first impression is positive about the competition, but these few examples leave me unsure whether Gemini Advanced is better than GPT-4 overall.