Testing MosaicML MPT-7B Chat and StoryWriter in 2023

MosaicML's MPT-7B release includes a base model and three fine-tuned variants: Instruct, Chat and StoryWriter. In this May 2023 review I look at the company's announcement and run informal tests. MosaicML reports training the base model on one trillion text and code tokens in roughly 9.5 days at a cost of about $200,000; that is its stated training cost, not a complete comparison with proprietary models.

The base model was released with a license permitting commercial use, and Instruct and StoryWriter were also presented as commercially usable. The Chat variant had a noncommercial license, so the family should not be treated as uniformly cleared for commercial use. Licenses and data restrictions vary by checkpoint; check the specific model before use.

Instruct is tuned to follow instructions, Chat for conversation, and StoryWriter for long-form text. StoryWriter was fine-tuned with a 65,536-token context, which was unusually long among the models compared in MosaicML's 2023 chart. A long context is a capacity claim, not proof that the model uses every detail reliably.

MosaicML's own benchmark table compares MPT-7B with LLaMA-7B and other models on a set of zero-shot tasks. Each of the two leading models tops six of the 12 listed tasks in that table. Those are vendor-reported results for particular benchmarks and historical models, not a universal or current ranking.

At the time, GPT4All's interface offered downloadable Chat and Instruct files, each shown at about 4.5 GB, while I used a Hugging Face Space for StoryWriter. Running a model locally depends on its format, memory and hardware, and long-context inference can be demanding. The Space shown in the video used an A100 and limited its demo sequence length to 10,000 tokens.

I begin with MPT-7B-Chat, which I preferred in this session. It correctly produces a Python loop to print the numbers one through 100. On a request for a Snake game, the generated code looks plausible at first but has missing methods and does not work when I try it without debugging. This is a small, informal test, not a comprehensive coding benchmark.

The model writes a short poem about AI, though it falls below the requested 50 words, and gives a conventional resignation email. It correctly names Bill Clinton as US president in 1996. When asked for instructions for breaking into a car, it refuses; that refusal is a safety behavior, not a failure to follow an ordinary benign instruction.

I test a drying-shirts puzzle without specifying whether shirts can dry in parallel or whether capacity is limited. The model answers ten hours without a sound explanation; the problem needs assumptions before a definite answer can be assessed. On a separate ordering question, it incorrectly says that Sam is faster than Jane despite the stated comparisons.

The model gets simple addition and an order-of-operations question right. Asked for a healthy meal plan, it first recommends consulting a dietitian and, when prompted again, announces a sample menu but does not actually provide the meals. I consider that an incomplete answer rather than a successful plan.

The model fails to answer how many words will be in its next reply and gives two for a killers-in-a-room puzzle, while my intended interpretation counts the newcomer as a killer and gives three. It answers 2021 when asked the current year; that output alone does not establish the model's training cutoff. A political comparison produces a general description instead of choosing a party.

For StoryWriter, I use the hosted demonstration and supply a short opening excerpt from a novel, then ask it to continue. The first continuation is acceptable to me, but a second generation after appending the output is less coherent. I speculate that more context might help; this one trial does not establish that it would.

MosaicML's announcement also gives a long-context demonstration using The Great Gatsby and an invented epilogue. I find that example promising and suggest GPT4All as one way to try the smaller variants locally. My assessment is based on a handful of prompts and 2023 tools, not a controlled or present-day ranking of open models.