Orca: Learning from GPT-4 Explanations

In June 2023, I compared two recent papers about learning from larger language models. One found that imitation could reproduce style without closing important factuality gaps; Microsoft researchers’ [Orca paper](https://arxiv.org/abs/2306.02707) reported stronger results from a 13-billion-parameter model trained on richer explanations.

The Orca authors argue that shallow question-and-answer imitation and weak evaluation can overstate a small model’s capabilities. They train a LLaMA-based model using explanations generated by stronger teachers, with ChatGPT assisting before GPT-4 examples.

The paper reports that Orca substantially exceeded Vicuna-13B on its zero-shot Big-Bench Hard and AGIEval comparisons and approached ChatGPT on BBH, while generally trailing GPT-4. These are results under the paper’s particular benchmarks and prompting conditions.

At the time of the video, the authors said they were working with legal counsel on a possible release of a weight difference under LLaMA’s policy. That historical statement should not be read as a current claim about model availability or license.

A 13-billion-parameter model is much smaller than GPT-3’s publicly reported 175 billion parameters. Smaller models can be deployed with fewer resources, although whether a particular laptop can run one depends on memory, quantization and software.

The earlier [imitation study](https://arxiv.org/abs/2305.15717) found that systems could impress raters through style while performing poorly on tasks less supported by imitation data. Orca’s results challenge a simple dismissal of imitation, but do not invalidate that study’s observations in its own setting.

The authors use system instructions to elicit explanations, not just final answers. In a common-sense example, the teacher explains why a flood is undesirable before giving its choice, giving the student more information to imitate.

The training sequence also matters in the reported experiment. The authors describe an intermediate ChatGPT teacher before GPT-4 examples, and the video compares that progressive approach with training only on the GPT-4 subset.

The paper reports open-ended assessments using GPT-4 as a judge, but also notes position bias in pairwise comparisons. Model-judged scores therefore need caution and should be considered alongside other evaluations.

Academic and professional exam questions offer another comparison. Orca outperformed the selected Vicuna baseline in the reported evaluations, but scores and relative improvements should not be generalized into a claim of across-the-board parity with a larger model.

Big-Bench Hard contains challenging tasks selected from BIG-bench. I point to individual examples such as logical statements and temporal sequences; performance varies substantially by task, so a few wins do not summarize all capabilities.

The authors characterize their reported results as zero-shot and note that other contexts, including multi-turn use and advanced prompting, remained to be investigated. A baseline under one setup is neither a guarantee nor a fixed ceiling.

I discuss research on stronger models creating reusable tools for smaller models, and speculate that tool use or process-based feedback might improve later systems. Those ideas were not demonstrated as Orca results in the paper discussed here.

The video notes that the model had not been released at the time and discusses the authors’ research-use framing. Questions about subsequent access, usage terms and later models require separate, current sources.

I speculate that Microsoft researchers may have been investigating whether a smaller model can cheaply imitate a larger one. This is my theory about motivation, not a documented statement from the research team.

The paper concludes that learning from detailed explanations is a promising way to improve models and calls for better evaluation. Its evidence concerns the tested model and datasets; reliable reasoning and factuality still require direct assessment.

The video ends with 2023 comments from Ilya Sutskever and Sam Altman on competition between openly available and proprietary systems. Their predictions and business arguments reflect the views expressed then, not a verified account of what happened later.

Altman emphasizes the difficulty of finding and executing the next idea as a possible competitive advantage. The discussion closes by inviting further examination of open models, an area that has continued to change since this 2023 video.