Testing Google Gemma 2 27B: Coding, Reasoning and Formatting

Google released Gemma 2 in 9-billion- and 27-billion-parameter sizes in June 2024, and I test the 27B instruction-tuned model in this July 2024 video. Google published model weights under the Gemma license, which contains conditions and use restrictions; calling the model open does not mean it is free of licensing terms.

Vultr sponsors this demonstration and supplies the cloud GPU environment I use to run the model without quantization. I connect an Open WebUI frontend to the hosted endpoint. Google reported that the 27B model could run at full precision on certain large accelerators; hardware needs depend on precision, context and deployment, and the promotional performance comparisons are not independent proof of a universal ranking.

Google's launch material compares Gemma 2 with models including Llama 3 on selected reasoning, mathematics and coding tests. I find the 27B results promising for its size. These scores reflect the specific benchmarks and evaluation setups available in 2024, not a guarantee that it matches larger models in all tasks.

I begin by asking for a Python script that prints the numbers from 1 to 100; the model provides code and an explanation. I then ask for Snake. The web interface's first run lacks a required module, so I copy the code into VS Code and run it there; the displayed game supports movement, food, score and losing when the snake hits a wall.

I request two changes: wrap the snake to the opposite side when it crosses a wall, and animate eating food. The wraparound works in the demonstration. I do not see the requested animation, although the game briefly freezes, so the example shows partial success rather than both features independently verified.

The model refuses my request for instructions for wrongdoing. I treat the refusal as an observed safety behavior, not evidence that the model is universally censored or that safety can be assessed by a single prompt. I move on to reasoning tests.

For a clothes-drying question, the model says twenty shirts can take the same four hours as five if conditions and drying space remain comparable. I regard that conditional explanation as sensible. The answer depends on assumptions such as adequate space and airflow; it is not a physical law independent of the setup.

On the arithmetic expression 25 − 4 × 2 + 3, the model gives 20, applying multiplication before addition. It also gives the expected expression for a hotel room price with an 8% room tax and a separate untaxed fee. These are individual examples, not a broad mathematics evaluation.

A response-length question fails: the model says its answer has ten words, but the displayed response is longer. On a room-of-killers puzzle it subtracts a victim but overlooks the entrant becoming a killer. The intended count requires assumptions about who is alive and whether the entrant was already a killer; the important observed error is that the model ignores the entrant's status.

In a glass-and-marble puzzle, the model says the marble remains in the glass after it is inverted on a table and then lifted into a microwave. I mark that wrong under the ordinary assumption that lifting the inverted glass leaves the marble on the table. Details such as contact and the way the glass is lifted can affect the physical outcome.

I ask for ten sentences ending in 'apple' and count only seven that meet the condition. A request to check the work does not repair the result in this example. These failures suggest difficulties following specific constraints in my small test set, but do not establish a general failure rate.

A question about fifty people digging a single hole receives a qualified answer of roughly five hours. I give it a marginal pass because work cannot necessarily be divided evenly in a confined space. The problem lacks information about dimensions, working conditions and coordination, so no unique numerical answer follows from the prompt.

Finally, I request a Python solution in JSON. The model returns the program's output as JSON instead of embedding the code in the structure I wanted, and a more explicit retry still does not match my intended format. I mark that attempt a failure, while recognizing that the initial wording leaves room for interpretation.

Overall, I find the coding demonstration encouraging and the logic and formatting tests less consistent. The small, informal sample is useful for showing specific behaviors, not for proving general capability or declaring a current leaderboard position. Vultr sponsored and hosted the test, which should be considered when reading my assessment.