Testing Google Gemma 7B at Launch: Setup Matters

SUMMARY

In 30 seconds

In this 2024 launch-era review, Matthew Berman tests Gemma 7B Instruct locally and on HuggingChat. The local run is slow and produces malformed answers; the hosted run improves on some prompts but still fails others. Berman suspects a local configuration issue, so the informal results should not be generalized to all Gemma deployments or later models.

1The launchBerman reviews Google’s Gemma 2B and 7B model release and its reported benchmark comparisons.
2Local setupHe downloads a model after accepting terms and configures LM Studio on a cloud GPU environment.
3Unexpected local resultsA basic Python task fails and generation is slow; he suspects the setup may be responsible.
4Hosted retestHuggingChat is faster and produces better answers for some prompts, but still fails other tasks.

Takeaways

Google released early Gemma models in 2B and 7B sizes with base and instruction-tuned variants.
Berman’s local 7B test was slow and produced malformed output; he suspected a setup or prompt-template issue.
HuggingChat was faster and answered some prompts better, but still failed several of his tests.
The small informal prompt set and differing environments do not establish a general model ranking.

TopicsOpen-weight models Evals Models & training

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Gemma and Google’s model strategy

Berman introduces the open-weight release and discusses Google’s reported benchmarks and competition.

Google has released Gemma, a family of open-weight models. I want to compare the launch claims with my own tests. This video reflects a 2024 launch-era experiment, not the performance of later Gemma models.

The release includes 2B and 7B models in base and instruction-tuned variants. Google says they draw on research behind Gemini and provides weights and tools for developers. I focus on the 7B instruction-tuned version and note that terms apply to the download.

Google’s launch chart compares Gemma with Llama 2 on language understanding, reasoning, mathematics and coding. I am cautious about using benchmark scores alone to predict an individual workflow. The comparison shown is between specific model sizes and settings, not a universal ranking.

SLIDE 1 / 3Google’s launch benchmark chartThe chart compares the reported Gemma 7B scores with Llama 2 variants.

A Gemini video example

He discusses a third-party Gemini 1.5 video analysis example and an apparent refusal.

Before testing Gemma, I discuss Simon Willison’s example of Gemini 1.5 analyzing a short video of books. It identifies many books but initially does not return the requested JSON; a follow-up gives a structured response. Another video request is refused, and the proposed trigger is only a hypothesis.

Local Gemma setup and tests

Berman configures LM Studio, tests code and reasoning, and questions whether the setup is correct.

I use LM Studio in a cloud GPU environment, accept the download terms, place an unquantized model file in the expected folder and select the Gemma instruction preset. Interface and model-distribution details are historical and may have changed.

The first Python prompt asks for numbers from 1 to 100. The local output appears malformed and fails when I try it. Generation is also unexpectedly slow. I stop a Snake-game attempt early and explicitly wonder whether my prompt template or setup is wrong.

A factual question identifies Bill Clinton as US president in 1996 but adds an odd explanation. A request for car-break-in instructions is refused. The model then gives a confused answer to a shirt-drying problem and a correct conclusion with poor reasoning on an ordering problem.

SLIDE 2 / 3A confused local reasoning answerThe local model gives inconsistent reasoning about the shirt-drying problem.

The local run answers 4 + 4 correctly but adds an inconsistent explanation. It fails a more complicated arithmetic question, word counting, a killer riddle and a JSON task. It also struggles with the cup-and-marble question, while the final belief-tracking answer is broadly correct.

Retesting on HuggingChat

The hosted service is faster and improves on some prompts, but several answers remain wrong.

Because the local results are so poor, I retry on HuggingChat. It responds much faster and gives a plausible answer to the simple Python task, though the Snake attempt is unsatisfactory. This difference supports my suspicion that the local setup affected the experience, but does not identify the cause.

On the hosted service, the killer and arithmetic questions still fail, while the JSON result is much better. The marble answer again introduces irrelevant heating. I remain disappointed in this small test, but the two environments and possible configuration issue limit what the results can establish.