Orca: Teaching a Smaller Model with GPT-4 Explanations

There is a debate in AI between large proprietary foundation models and smaller open models. In June 2023, a new Microsoft Research paper on Orca adds another perspective. I want to compare it with a leaked Google memo, ‘We Have No Moat,’ and a paper questioning whether models trained to imitate proprietary systems really acquire their capabilities.

The Google memo argued that open models were improving rapidly as developers gained access to models and efficient training techniques such as LoRA and QLoRA. I found much of it compelling: technical barriers are not the whole of a business moat, but the open community has been iterating quickly.

A study led by researchers at UC Berkeley challenged the value of imitation. Its experiments found that models trained on ChatGPT outputs could look convincing to people while closing little of the capability gap on tasks poorly represented in their imitation data. It argued that style can hide weaknesses in factuality and that stronger base models matter.

I think of it like a student who matches questions to answers rather than learning a subject deeply: a small variation can expose the limits of memorized patterns. That is an analogy, not proof of what either a human or a language model understands. Orca’s paper asks whether more informative teacher outputs can improve the smaller model’s performance.

The authors identify shallow imitation signals, small homogeneous datasets and weak evaluation as problems. Their proposed Orca model has 13 billion parameters and learns from detailed GPT-4 explanation traces, with ChatGPT acting as a teaching assistant. A model of this size may be more accessible than a 65-billion-parameter model, though local requirements depend on memory, quantization and hardware.

Instead of using only a question and short answer, the researchers ask the teacher for step-by-step explanations and use those responses as training signals. Berman describes this as learning how an answer is reached; the generated explanations are observable text, and the experiments do not establish that a model has human-like understanding.

The training process uses a large set of five million ChatGPT examples followed by one million GPT-4 examples. The authors call this progressive learning, with an intermediate teacher helping the smaller model learn before it encounters the stronger teacher’s more complex output.

The paper reports that Orca exceeds Vicuna-13B by more than 100% on Big-Bench Hard and by 42% on AGIEval in its chosen comparisons. It reports near parity with ChatGPT on Big-Bench Hard and competitive examination results, while still trailing GPT-4. These are zero-shot results under the paper’s settings, not proof that Orca is better on every task.

Ordinary instruction tuning trains on instructions and responses. The paper’s explanation tuning adds a more detailed account of the steps leading to a response. I find this promising because it supplies richer training material than a bare answer, although the resulting scores do not settle the broader question of model understanding.

In one GPT-4-judged evaluation shown in the paper, Orca scores slightly above ChatGPT and above Bard and the listed LLaMA-based models. The authors also compare performance on academic exams and Big-Bench Hard. A model judging responses is itself an evaluation method with limitations, so the figure should be read alongside the other tests.

The Orca authors respond to the earlier imitation paper’s concern that matching ChatGPT would require much larger and more diverse data. They argue their results show that such data can be assembled and that the gap can be reduced on several zero-shot reasoning benchmarks. That is a narrower claim than resolving every limitation identified by the earlier study.

Earlier methods offer useful contrasts. Alpaca and WizardLM use forms of self-instruction, and WizardLM progressively rewrites instructions to make them more complex. Vicuna and Koala draw on natural conversations; these can teach a conversational style, but the authors question whether short conversational examples teach enough about the steps behind an answer.

The first contribution is explanation tuning. System instructions ask the teacher to explain, justify its answer or think step by step. These prompts produce detailed responses for the student model to learn from. I find it fascinating that prompting techniques people use to get more elaborate answers are also used to produce training data.

The second contribution is scale and diversity. The authors draw tasks from the FLAN collection. Their table lists 52,000 examples for Alpaca, 70,000 for Vicuna and 250,000 for WizardLM, compared with five million ChatGPT examples for Orca and a one-million-example GPT-4 subset. These are the paper’s comparisons with models available at that time.

The third contribution is evaluation. The researchers use GPT-4 to compare responses, academic tests including Big-Bench Hard and TruthfulQA, professional and academic examinations, and ToxicGen for toxic-language evaluation. Each method measures a particular slice of behavior rather than every aspect of model quality or safety.

One figure illustrates a conventional instruction-response format with a median problem: sort five numbers and identify the middle value, seven. Orca’s approach adds a system instruction asking the teacher to explain its steps. The resulting longer explanation is then used in training.

System messages are the main tool the authors use to elicit the explanations. They might ask the teacher to give a detailed answer, explain for a five-year-old, or justify its reasoning. Such instructions change the text available to the student; they do not independently verify that every explanation is correct.

Another example asks which statement is less logical: people in a desert looking forward to a flood or looking forward to rain. A short answer identifies the flood because it causes damage. ChatGPT’s illustrated answer instead lays out steps, compares the statements and explains why rain can be beneficial while floods can be dangerous. The displayed GPT-4 response is more detailed still.

Why use ChatGPT as an intermediate teacher? The authors propose a capacity-gap explanation: a smaller student may learn more effectively when it first encounters an intermediate teacher before the stronger GPT-4 examples. They compare the sequence to a curriculum that moves from easier to harder material, like learning addition before calculus.

There is also a practical reason: at the time, GPT-3.5 Turbo was faster and cheaper than GPT-4. Berman says this made it attractive to generate five million intermediate examples before using GPT-4 on a million. The paper presents an experiment in which the progressive sequence improves the reported score from 37.18 to 41.7 compared with GPT-4-only training.

The examination comparisons show Orca substantially ahead of Vicuna on the questions discussed and closer to ChatGPT, though still behind GPT-4. On Big-Bench Hard, the paper reports Orca slightly ahead of ChatGPT in aggregate, 113% above Vicuna, and well behind GPT-4. Aggregate results do not imply a lead on every individual task.

To me, the work illustrates how quickly open-model techniques were advancing in 2023, while GPT-4 still appeared to retain a considerable advantage on these tests. I find it striking that Microsoft Research produced the paper given Microsoft’s investment in OpenAI. My expectation is that models will continue to improve and become cheaper; that is a forecast, not a result demonstrated by this study.

At the time of this video, Orca’s code and dataset had not yet been released, so I could not run my own evaluation. I plan to revisit it if the materials become available and see how it performs in practice.