Testing MosaicML MPT-7B Chat and StoryWriter in 2023

SUMMARY

In 30 seconds

Matthew Berman reviews MosaicML’s 2023 MPT-7B release, compares its vendor-reported benchmarks and tries Chat locally and StoryWriter in a hosted demo. The model succeeds on some simple tasks and fails several reasoning and coding prompts. Licensing differs across variants and the small test does not establish a current ranking.

1Different model licensesThe base, Instruct and StoryWriter were presented for commercial use; Chat carried a noncommercial license.
2Vendor benchmarksMosaicML’s 2023 comparison found MPT-7B and LLaMA-7B each led six of 12 selected tasks.
3Mixed coding resultsA simple loop worked, but the unmodified Snake game did not.
4StoryWriter testThe hosted demo produced a continuation but a follow-up was less coherent.

Takeaways

MPT-7B includes base, Instruct, Chat and StoryWriter variants with different purposes and licenses.
The original Chat variant was noncommercial, unlike the commercially usable base model.
StoryWriter’s 65k training context was distinct from the 10k limit of the hosted demo shown.
The Chat model gave mixed answers, including a working simple loop and a failed Snake game.
A refusal to assist a harmful request is a safety behavior, not an ordinary task failure.

TopicsOpen-weight models Benchmarks Models & training Generative media

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

The MPT-7B family

Berman introduces the base and three fine-tunes, their historical context and licensing. He reviews vendor benchmarks and context length.

MosaicML's MPT-7B release includes a base model and three fine-tuned variants: Instruct, Chat and StoryWriter. In this May 2023 review I look at the company's announcement and run informal tests. MosaicML reports training the base model on one trillion text and code tokens in roughly 9.5 days at a cost of about $200,000; that is its stated training cost, not a complete comparison with proprietary models.

The base model was released with a license permitting commercial use, and Instruct and StoryWriter were also presented as commercially usable. The Chat variant had a noncommercial license, so the family should not be treated as uniformly cleared for commercial use. Licenses and data restrictions vary by checkpoint; check the specific model before use.

Instruct is tuned to follow instructions, Chat for conversation, and StoryWriter for long-form text. StoryWriter was fine-tuned with a 65,536-token context, which was unusually long among the models compared in MosaicML's 2023 chart. A long context is a capacity claim, not proof that the model uses every detail reliably.

SLIDE 1 / 4MosaicML benchmark tableThe vendor reports results on selected zero-shot tasks for 2023 models.

MosaicML's own benchmark table compares MPT-7B with LLaMA-7B and other models on a set of zero-shot tasks. Each of the two leading models tops six of the 12 listed tasks in that table. Those are vendor-reported results for particular benchmarks and historical models, not a universal or current ranking.

SLIDE 2 / 4Historical context-length comparisonThe chart compares StoryWriter’s training context with selected models at launch.

At the time, GPT4All's interface offered downloadable Chat and Instruct files, each shown at about 4.5 GB, while I used a Hugging Face Space for StoryWriter. Running a model locally depends on its format, memory and hardware, and long-context inference can be demanding. The Space shown in the video used an A100 and limited its demo sequence length to 10,000 tokens.

Testing MPT-7B-Chat

Local prompts cover coding, writing, facts, reasoning and safety. The results are mixed and informal.

I begin with MPT-7B-Chat, which I preferred in this session. It correctly produces a Python loop to print the numbers one through 100. On a request for a Snake game, the generated code looks plausible at first but has missing methods and does not work when I try it without debugging. This is a small, informal test, not a comprehensive coding benchmark.

The model writes a short poem about AI, though it falls below the requested 50 words, and gives a conventional resignation email. It correctly names Bill Clinton as US president in 1996. When asked for instructions for breaking into a car, it refuses; that refusal is a safety behavior, not a failure to follow an ordinary benign instruction.

I test a drying-shirts puzzle without specifying whether shirts can dry in parallel or whether capacity is limited. The model answers ten hours without a sound explanation; the problem needs assumptions before a definite answer can be assessed. On a separate ordering question, it incorrectly says that Sam is faster than Jane despite the stated comparisons.

SLIDE 3 / 4Chat model responsesThe example shows a refusal and an unsupported answer to an ambiguous drying puzzle.

The model gets simple addition and an order-of-operations question right. Asked for a healthy meal plan, it first recommends consulting a dietitian and, when prompted again, announces a sample menu but does not actually provide the meals. I consider that an incomplete answer rather than a successful plan.

The model fails to answer how many words will be in its next reply and gives two for a killers-in-a-room puzzle, while my intended interpretation counts the newcomer as a killer and gives three. It answers 2021 when asked the current year; that output alone does not establish the model's training cutoff. A political comparison produces a general description instead of choosing a party.

StoryWriter in a hosted demo

Berman tries a continuation and a follow-up. The displayed demo has its own sequence limit.

For StoryWriter, I use the hosted demonstration and supply a short opening excerpt from a novel, then ask it to continue. The first continuation is acceptable to me, but a second generation after appending the output is less coherent. I speculate that more context might help; this one trial does not establish that it would.

SLIDE 4 / 4StoryWriter hosted demoThe screen states that this demo limited the sequence to 10,000 tokens.

MosaicML's announcement also gives a long-context demonstration using The Great Gatsby and an invented epilogue. I find that example promising and suggest GPT4All as one way to try the smaller variants locally. My assessment is based on a handful of prompts and 2023 tools, not a controlled or present-day ranking of open models.