Phi-3 Mini: Microsoft’s Small Model, Phone Deployment and an Uncertain Local Test

SUMMARY

In 30 seconds

Matthew Berman examines Microsoft’s Phi-3 Mini, its training data, reported benchmarks and offline phone demonstration, then tests what he identifies as the model locally. The test interface displays a conflicting Llama 7B label, leaving the identity of the model generating those answers uncertain; the examples are not a controlled evaluation.

1Size versus MMLUMicrosoft’s chart compares Phi-3 variants with other 2024 models on one benchmark, with active parameters on the size axis.
2The Mini recipeThe technical report describes 3.8 billion parameters, 3.3 trillion training tokens, filtered web data and synthetic data.
3Offline on a phoneThe authors report 4-bit inference on an iPhone 14 at more than 12 tokens per second.
4Benchmark results varyPhi-3 Mini scores 68.8 on MMLU in the shown table, but trails Mixtral on other tests including TriviaQA.
5An uncertain local demonstrationBerman reports mixed prompt results; the interface’s Llama 7B label leaves model identity unresolved.

Takeaways

Microsoft reported that the 3.8B-parameter Phi-3 Mini was trained on 3.3 trillion tokens of filtered and synthetic data.
A 4-bit version reportedly ran offline on an iPhone 14 at over 12 tokens per second in the paper’s test.
The model’s benchmark comparisons depend on the particular tasks, model versions and evaluation settings.
The paper identifies factual knowledge and predominantly English training as limitations.
The video’s informal test contains both successes and failures, but its displayed Llama 7B label conflicts with the stated Phi-3 Mini identity.
Berman’s estimate that agents could handle 95% of assistant tasks is speculation rather than a measured result.

TopicsLocal inference Models & training Benchmarks Open-weight models

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Phi-3 and its training data

Berman introduces the model family, Microsoft’s benchmarks and the idea of using filtered and synthetic data. He distinguishes his hopes for phone assistants from the paper’s results.

Microsoft’s Phi-3 family aims to make small language models more capable through carefully selected training data. In this April 2024 review, I look at the technical report and then run a set of local tests. The appeal is a model small enough to run on a phone, rather than needing a cloud connection for every task.

SLIDE 1 / 7Model size and MMLUMicrosoft’s chart plots benchmark quality against active parameters for the listed 2024 models.

Microsoft’s chart compares model size with quality measured by MMLU. It shows Phi-3 Mini in 4K- and 128K-context versions, with Phi-3 Small and Medium shown as previews. Phi-3 Medium is a 14-billion-parameter model; the Mixtral 8x7B comparison is a mixture of experts with different total and active parameter counts. A position on one chart does not establish superiority on every task.

I imagine an on-device assistant with access to search, memory and tools, perhaps coordinated by agents. I speculate that such a system could handle a large share of everyday requests, even though small models fall short of frontier models such as GPT-4 and Claude 3 Opus. My estimate of 95% is a prediction, not a measured result from the paper.

The report introduces Phi-3 Mini, a 3.8-billion-parameter model trained on 3.3 trillion tokens. Microsoft reports about 69% on MMLU and 8.38 on MT-Bench, and says its results on its selected academic and internal tests rival Mixtral 8x7B and GPT-3.5. These are results under particular evaluations, not proof of equivalent performance in all uses.

SLIDE 2 / 7Phi-3 technical reportThe abstract reports training scale and selected benchmark scores for Phi-3 Mini.

The authors emphasize the training data: filtered web material and synthetic examples built on the approach used for Phi-2. Larger models can help filter and create training data for smaller ones. At the time of recording I was unsure about the exact release terms and where to download weights, so my uncertainty should not be read as a definitive licensing assessment.

The paper contrasts scaling assumptions based on a fixed data source with the possibilities created by capable language models that filter or generate data. The researchers argue that this approach can produce small models with performance previously associated with larger ones. Whether such a model truly reasons or reproduces patterns learned from its training remains a question my few examples cannot settle.

Context, phone deployment and benchmarks

The report describes context lengths, quantization and a phone test. Its table also shows differences across tasks and known limitations.

Phi-3 Mini’s default context is 4K tokens; the separate 128K version extends it using LongRoPE. The model uses a Llama-2-like block structure and a 32,064-token vocabulary, which the paper says makes existing tooling easier to adapt. The model is also chat fine-tuned.

SLIDE 3 / 7Reported phone deploymentThe report describes four-bit quantization and offline inference on an iPhone 14.

Microsoft reports that the 4-bit quantized Mini occupies about 1.8 GB and ran fully offline on an iPhone 14 with an A16 chip at more than 12 tokens per second. That is an observation for this tested configuration, not a guarantee for every phone, prompt or context length.

The paper’s benchmark table gives Phi-3 Mini 68.8 on five-shot MMLU, compared with 66.0 for Llama 3 8B Instruct, 68.4 for Mixtral 8x7B and 71.4 for the listed GPT-3.5 version. Other rows vary: for example, Mini scores 64.0 on TriviaQA versus 82.2 for Mixtral. Individual benchmarks and settings matter.

The paper identifies factual knowledge as a weakness of the smaller model, including its TriviaQA performance. Search or retrieval could supplement that limitation, though reliable tool use would need its own testing. The authors also say the model is primarily limited to English; I speculate that specialized models for other languages might be another approach.

Local setup and coding tests

Berman demonstrates LM Studio configuration, a simple script and an unsuccessful Snake game. The visible model label does not clearly match the model he says he selected.

For the demonstration I search LM Studio for Phi-3 Mini 4K Instruct and choose an FP16 download just under 8 GB rather than a Q4 version. The interface at the time needed a Phi-3 preset, which I downloaded and dragged in. I leave the system prompt empty, set temperature to 0.2 and enable Apple Metal on my machine.

SLIDE 5 / 7LM Studio presetThe displayed configuration supplies the prompt formatting used in the demonstration.

A simple Python script to print the numbers from 1 to 100 succeeds, at a displayed 33 tokens per second. A Snake game is harder: the first generated code has undefined directions and screen variables. After asking for fixes and adding the suggested definitions, the program still does not play successfully, so I count that attempt as a failure.

For this size of model, I expect question answering and retrieval-assisted use to be more promising than complex coding. A request for instructions to break into a car is refused; a follow-up framed as a movie script is also refused and includes an incorrect claim that the assistant was developed by OpenAI. I speculate about the training data, but a model’s self-description does not establish its provenance.

Reasoning and final assessment

Individual prompts yield a mixture of correct answers and errors. These informal examples cannot establish overall model performance, especially given the unresolved interface label.

The model correctly says that 20 shirts can dry in the same four hours as five if they have enough space and the same conditions. It also correctly orders Jane, Joe and Sam by speed, answers 4 + 4, and evaluates 25 − 4 × 2 + 3 as 20. These are individual examples, not a comprehensive reasoning benchmark.

For a hotel charging $99.95 per night plus 8% tax and a one-time untaxed $5 fee, it gives the correct expression: 1.08 × 99.95 × x + 5. It then fails a self-referential word-count question, claiming an incorrect number of words and adding a mistaken explanation.

In the three-killers puzzle, the model explains that two original killers remain and the newcomer becomes a killer, yielding three under the stated assumptions. It also produces the requested JSON for three people and their ages. Both answers pass my informal checks.

The upside-down cup and marble question produces a bad answer: instead of tracking the marble left on the table when the inverted cup is lifted, the response discusses microwaves. On a separate belief-tracking question it correctly distinguishes John’s belief that the ball is in the box from Mark’s knowledge that it is in the basket, despite some confusing wording.

SLIDE 6 / 7The cup and marble promptThe response fails to track the marble, while the interface displays a conflicting Llama 7B model label.

Asked for ten sentences ending in “apple,” the model fails by the third sentence. In a final digging problem it calculates a combined rate of ten holes per hour and a tenth of an hour per hole, then contradicts that calculation by saying 50 workers would still need five hours. Under ideal linear scaling the answer would be six minutes; practical space constraints could prevent that scaling.

SLIDE 7 / 7The digging problemThe response gives a rate calculation and then contradicts it with a five-hour conclusion.

My conclusion is that the results are impressive for a small model but uneven, especially on the Snake program and some reasoning tasks. There is an important qualification: although I say I am testing Phi-3 Mini, the demonstration interface displays “Llama 7B” and FP16 model information. The video alone does not resolve whether that label is incorrect or a different model was loaded, so the observed responses cannot be confidently attributed to Phi-3 Mini.