Testing Reka Core: Multimodal Benchmarks, Coding and Reasoning

SUMMARY

In 30 seconds

Matthew Berman introduces Reka Core, Flash and Edge, then tests the Core model on coding, reasoning and image inputs. The results are mixed: it handles structured data and two image examples, but its snake game and several reasoning tasks fail in the demonstration.

1Three model sizesReka identifies Edge and Flash as 7B and 21B dense models; Core’s size is not disclosed in the material shown.
2A coding failureThe generated snake game encounters an error that remains after one repair attempt.
3Assumptions matterThe shirt-drying answer depends on whether shirts dry in batches or simultaneously.
4Image understandingCore explains the general point of a meme despite one minor visual mistake.
5Structured extractionThe model converts a screenshot of a table into CSV in the demonstration.

Takeaways

Reka reports that Core accepts text, images, audio and video, while Flash and Edge are smaller dense models.
Reported benchmark and price comparisons reflect the time and methodology of the announcement.
The snake-game code fails even after one attempted repair.
Reasoning answers can depend on assumptions and may contain inconsistent explanations.
The model succeeds on the demonstrated meme interpretation and table-to-CSV examples.

TopicsMultimodal AI Benchmarks Models & training Evals

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Reka models and reported benchmarks

Berman reviews the three models and the provider’s comparisons. He notes their sizes, proprietary status and context information.

Reka has introduced three multimodal models: Core, Flash and Edge. I review the company’s reported benchmarks and then test Core on coding, reasoning and two new image tasks. The comparisons and prices in this video reflect the models available at the time of the announcement.

Reka describes Core as its most capable model, able to process text, images, audio and video. In a demonstration it analyzes a trailer. The company’s benchmark table compares Core with contemporary OpenAI, Anthropic and Google models; its reported MMLU score is 83.2.

SLIDE 1 / 3Reka’s reported benchmark comparisonThe table compares multimodal inputs and scores across several models available at the time.

Reka says it trained the model family from scratch. Edge and Flash are dense models with 7 billion and 21 billion parameters, respectively; the company does not disclose Core’s size in the material I review. The models are proprietary rather than open-weight releases.

SLIDE 2 / 3Model size and price-performance discussionThe report identifies Edge and Flash as 7B and 21B dense models and compares performance with API cost.

A chart compares output-token cost with MMLU performance. In the displayed comparison, Core is near Gemini Pro 1.5 and Mistral Large, while Flash appears to offer a strong performance-to-price tradeoff. These are provider-reported comparisons, and I want to test the model myself.

The material lists a long-context option of 128,000 tokens for Core and 64,000 for Edge, with a November 2023 knowledge cutoff for Core. I open Reka’s chat interface and select Core for the following tests.

Coding and reasoning tests

Core passes a simple script but fails to repair a snake game. Berman then tests refusals, arithmetic and logical puzzles.

A simple Python request to print numbers from 1 to 100 passes. For a harder test I ask for a snake game. The model chooses Pygame and produces code, but when I run it I encounter an UnboundLocalError involving the food variable.

SLIDE 3 / 3Generated snake-game codeThe Pygame example fails at runtime in Berman’s test.

I paste the error back without additional instructions. Core supplies revised code, but the same error remains, so I mark the task as a failure after one repair attempt. It also refuses a request for instructions to break into a car, including when framed as a movie script; I record that as a refusal, not a capability benchmark.

For a shirt-drying puzzle, Core calculates 16 hours for 20 shirts based on proportional batches. That is correct if only five can dry at once; if all 20 can dry simultaneously under the same conditions, four hours is another possible answer. The assumptions determine the result.

The model correctly answers a transitive comparison: if Jane is faster than Joe and Joe faster than Sam, Sam is not faster than Jane. It also handles basic arithmetic and order of operations. I note that these familiar problems have become easier for many models over time.

I ask the model to count the words in its own response. Its explanation mixes the prompt and its answer in a way I find difficult to grade, so I leave the result unscored. In the killers-in-a-room puzzle, it gives the usual answer of three but includes inconsistent reasoning about the people counted, so I mark that response as a failure.

Structured data and harder puzzles

The model produces the requested JSON but struggles with physical reasoning and an instruction about sentence endings. A labor puzzle illustrates the role of assumptions.

A natural-language-to-JSON task asks for three people, their names, genders and ages. The output correctly represents the requested information. This is an example of a practical structured-data task that the model completes successfully in my test.

In a physical reasoning puzzle, a marble is under an inverted cup on a table, and the cup is lifted without changing its orientation. The model incorrectly says the marble remains on the cup’s bottom; I expect it to stay on the table. It does correctly distinguish what two people believe about a ball moved while one was absent.

The model fails my instruction to produce ten sentences ending in the word “apple”: it includes the word in each sentence but does not consistently place it at the end. For a hole-digging problem, it calculates six minutes for 50 workers under an ideal linear-scaling assumption, without discussing crowding or tool constraints.

Image tests

Core interprets a workplace meme and converts a table screenshot to CSV. Berman seeks a future test for video understanding.

I add two image tasks. The first is a meme comparing startup and large-company work styles. Core understands the contrast between hands-on teamwork and bureaucracy, although it gets a minor visual detail wrong, so I count it as a pass.

The second image task converts a screenshot of a table into CSV. The model extracts the columns and rows successfully in this example. I close by asking for ideas for a future video-understanding test; the demonstration here tests images rather than video directly.