IBM Granite 3.0 and InstructLab: Small Models for Enterprise AI

IBM invited me to New York to see its AI projects and partnered with me on this video. Two announcements caught my attention: the Granite 3.0 models and InstructLab. IBM released the Granite family under the Apache 2.0 license, and InstructLab is an open-source project.

Granite 3.0 is a family of relatively small general-purpose models that IBM positions for enterprise use. Berman is particularly interested in supplementing a model with private or domain-specific knowledge. Whether a model can run on a particular laptop or phone depends on its memory, software and configuration.

Berman argues that useful data can lie behind authentication or within enterprises, and sees it as an opportunity alongside synthetic data and additional test-time computation. His claim that essentially all public internet data has already been used is his assessment, not a verified inventory of training data.

An insurance company, for example, might want to adapt a model with information that is not public. Berman sees Granite and InstructLab as a way to work with such domain data. The quality of the result depends on the data, training process and evaluation rather than following automatically from adding data.

The dense family includes Granite 3.0 8B and 2B in base and instruction-tuned versions. IBM also announced small mixture-of-experts models: 3B-A800M and 1B-A400M. The names indicate three billion or one billion total parameters, with 800 million or 400 million active during inference, respectively.

IBM presents the dense models as workhorses for retrieval-augmented generation, classification, summarization, entity extraction and tool use. Berman also points to Granite Guardian, a separate family intended to help detect risks and support guardrails in business applications.

Berman walks through IBM's charts comparing Granite 3.0 8B Instruct with similarly sized Llama and Mistral models. IBM reports competitive results on the displayed leaderboard, RAG, cybersecurity and tool-calling tests. These vendor-reported comparisons describe particular tests and model versions, not performance on every workload or a current ranking.

Mixture-of-experts models activate only a subset of their parameters for inference. IBM positions the smaller Granite variants for low-latency, CPU and on-device settings, and says they were trained on more than ten trillion tokens. Base and instruction-tuned variants were offered; actual hardware needs and speed depend on deployment.

IBM also announced a Granite-powered coding assistant. Berman connects this work with IBM's ownership of Red Hat and its open-source enterprise tools, including Linux, OpenShift, Ansible and OpenStack. He notes an announcement integrating CrewAI with watsonx for enterprise agent workflows.

Berman contrasts retrieval-augmented generation, which supplies information at query time, with adapting a model itself. He initially describes InstructLab as not quite fine-tuning. More precisely, InstructLab's LAB method uses alignment tuning: curated seed examples are expanded with synthetic data, then used in a training process to add skills and knowledge without pretraining from scratch.

The open-source workflow supports collaborative contributions and aims to reduce the cost of creating training data. Its multi-phase training and replay approach is intended to retain existing capabilities; it should not be treated as a guarantee that no knowledge is lost. Practical suitability depends on the model, data, compute and testing.

Finally, Berman briefly mentions IBM's long-running quantum computing work and a talk at the event about combining quantum computing and AI. He says he is still learning about the subject and hopes to interview a researcher. He closes by encouraging viewers to explore Granite and InstructLab, again thanking IBM for the partnership.