Matthew Berman Tests Jamba 1.5 on Coding and Reasoning Prompts

I test AI21’s Jamba 1.5 Large after introducing the Jamba family and its hybrid state-space model and Transformer architecture. The video records one playground session in 2024, not a controlled benchmark of the architecture or later versions.

AI21 announced Mini and Large models with a claimed 256,000-token context window, long-context speed advantages and benchmark scores. It also described multilingual use, structured JSON, function calling and citations. These are the company’s claims and depend on the specified testing conditions.

The models were offered for download under the Jamba Open Model License and through several cloud services. I use AI21 Studio for this test. Availability, terms and model capabilities can change, so the announcement should be read in its historical context.

I select Jamba 1.5 Large and ask it to write Tetris in Python. The output takes roughly 20 to 30 seconds to begin, and I say it continues for about seven minutes. A single session cannot establish general throughput or identify whether the delay came from the model, service or interface.

The generated code uses Pygame, but when I copy it into my editor and run it, nothing happens. I mark the attempt as a failure and do not iterate because of the wait. That is a useful observation about this particular attempt, not proof that every coding prompt will fail.

The next prompt asks whether a 200 by 275 millimeter envelope meets given postal size limits. The model says it does not; I believe rotating the envelope allows it to fit. The conclusion depends on the exact dimensional rules and how the sides are assigned.

The model answers “16” when asked to count the words in its own response, and says two killers remain after a newcomer kills one of three. I mark both wrong: in the latter puzzle, the newcomer has also become a killer under the usual interpretation.

In the marble puzzle, the model reasons that a marble stays at the top of an inverted glass after the glass is lifted. I reject that answer: under the ordinary setup the marble falls onto the table and remains there when the glass is picked up.

I also ask a North Pole walking puzzle and disagree with the answer. The wording leaves the meaning of a straight path and passing the starting point open to interpretation, so the exchange is not a clean objective score without specifying those assumptions.

Asked for ten sentences ending in “Apple,” the model produces nine that meet the instruction and one that does not. It correctly counts three occurrences of “r” in “strawberry” and says 9.9 is greater than 9.11.

For a moral hypothetical about gently pushing someone to save humanity, the model eventually answers no. I disagree but acknowledge that the grade reflects my own ethical judgment; this is not an objective pass-or-fail measurement.

I conclude that this test session was disappointing, while welcoming competition among model architectures. The small set of prompts and one hosted session support a limited observation, not a general verdict on Mamba or all Jamba deployments.