Beyond GPT-7: The Customization Toolkit, Meta's Open Source Bet and the Return of Consumer AI

The hosts begin with a question on many minds: what does it take to get 10x or 100x better AI systems? Beyond scaling data, compute and parameters, one host argues that many gains are available on existing models, so you do not need to wait for a hypothetical GPT-7. The levers fall into five or six areas that people in the field expect to arrive quickly.

The first is multimodality, meaning text, voice, images and video as both input and output. The second is long context windows, with the example of a company called Magic letting you dump an entire code repository into a coding model. The third, the focus of the episode, is customization through fine-tuning, RAG, data cleaning and labeling.

The remaining three are memory, so the AI remembers what it is doing; recursion, or looping models back to reuse them; and many small specialized models orchestrated by a central model or router. The comparison offered for the last is the human brain, which uses the visual cortex for vision and other regions for decisions. These are coming fast, he says, and it is a matter of when, not if.

The hosts discuss OpenAI opening up fine-tuning for more sophisticated models such as GPT-3.5, aimed at more enterprise use cases. One host notes that labs working on general capability and scaling laws are only part of the picture, because models in business and consumer settings are used against specific tasks. The discussion adds that we do not yet know how good models of 30, 70 or 100 billion parameters can be when trained on very high quality, carefully curated data.

There was a period when the research narrative held that fine-tuning does not matter and the general model does. The other host says ChatGPT showed otherwise: it took GPT-3.5, which was seeing little use unless people were skilled at prompting, and had people rank its outputs so the model could be tuned on that feedback. The underlying model had not changed much, yet usefulness jumped for students, lawyers and many others, which the host calls the starting gun for the current AI revolution.

In an enterprise, the same approach could tune a model on proprietary medical data to build a better doctor assistant, or on a company's own HR answers so employees get good responses. The hosts see these specializations as a way to improve a model's output dramatically against something you care about.

The basic premise of retrieval-augmented generation, one host explains, is to retrieve against a specific corpus such as legal cases, internal documents or medical information. You still want the model's reasoning, for example to reach a diagnosis, but you want it grounded in chosen data rather than all the random pre-training material on the internet. The core driver is trustworthiness: citations and control over information sources.

The host adds two further reasons. Retraining or even fine-tuning a model takes compute, so being able to incorporate new information without retraining is cheaper, and it keeps answers fresh, such as the latest medical research or this year's cases. Architecturally it combines traditional information retrieval and search with the models.

The hosts note that RAG also helps with hallucination, a common criticism of current systems, such as citing legal cases that do not exist. By restricting answers to things known to exist, you can filter outputs, which matters because critics use hallucination to argue for regulation. They say it is worth calling out explicitly.

A recent Google paper compared human feedback with AI-generated feedback, showing that for certain use cases an AI can judge output about as well as people. That would mean fine-tuning without hiring an army of raters. As early evidence, the host cites Med-PaLM 2, where a model trained on medical data produced answers more correct than human physician experts, and says RLAIF aims to generalize this with cheap models in place of expensive people.

On the open source side, the hosts discuss Meta emerging as the main sponsor with Llama and Llama 2. One host compares it to MySQL, where early internet-scale companies like Facebook wanted to avoid dependence on commercial database vendors, made it more robust and contributed back. Meta may not see models as its business model but wants open options, and if enough developers find them strategic, the ecosystem is hard to picture fading.

The other host raises Linux, which IBM sponsored in the late 1990s at times to the tune of a billion dollars a year as an offset to Microsoft. The hosts see why the analogy may break: Meta does not seek B2B revenue from it, and centralized training is hard to coordinate with open source communities, though some groups are working on it. One guess is that Meta wants to use the models in its core businesses, such as data center efficiency, ad serving and consumer engagement, without being locked into a vendor.

The hosts argue that generative AI products such as Character, Inflection, Midjourney, Pika and Can of Soup have captured consumer attention in a way few things have in years, and that Meta's Instagram chatbots show the interest. One host points out that TikTok, launched seven years earlier, was the last large new social network, partly because it is hard to pry users from established products. The same host adds that social innovation for startups had stagnated into variations like Twitter but different.

They sketch how social products can be mapped by broadcast versus mutual follows, by modality such as image or video, and by length and persistence, with Snapchat starting as short-form, ephemeral broadcast and one-on-one. New forms of content creation could upend some of those quadrants. The hosts note that big networks such as Meta and Twitter may be the biggest beneficiaries, though there should be room for startups, and one cites AI-native content aggregators in China that solved cold start on relevance, where generation might solve cold start on content itself.

One host closes with advice for founders. When a technology shift disrupts a market, the advice is to do the easy things because there is so much low-hanging fruit; if something is hard, make it the technology that is hard, not the market. The other host agrees, noting that of accelerator founders with strong research backgrounds were reaching for ambitious, solve-it-all code generation problems, and that more at-bats with the wind at your back beat a multi-year research goal. The slogan they land on: no GPU before product-market fit.