In 30 seconds
Karpathy discusses the challenges of specialization, capability retention and the cost of deeper model customization.
- 1Capability retentionFine-tuning can risk existing abilities.
- 2ContextKarpathy calls context a relatively cheap customization mechanism.
- 3WeightsChanging weights can affect the whole model.
Takeaways
- Context is a relatively cheap way to adapt behavior.
- Fine-tuning changes model weights.
- Specialization must preserve capability and justify its cost.
Read the edited text by chapter
Adapting models
Andrej Karpathy argues that the methods for deeply adapting AI models are still developing. He gives fine-tuning without loss of existing capabilities as one challenge.
Andrej Karpathy argues that the methods for deeply adapting AI models are still developing. He gives fine-tuning without loss of existing capabilities as one challenge.
He contrasts context windows, which he describes as a relatively cheap and useful way to customize behavior, with changing the model itself.
Weights and specialization
Karpathy says continual learning and specialization require better ways to adjust model weights and improve performance in a particular area.
Karpathy says continual learning and specialization require better ways to adjust model weights and improve performance in a particular area.
Changing weights can affect the whole model and its capabilities, making it trickier than changing context. The speakers add that such customization must also be economical.
More from this channel
38:06EN
No Priors: AI, Machine Learning, Tech, & Startups· Oct 29, 2026Podcast · 3 speakersServal’s Jake Stauch on coding agents for enterprise ITGuest Jake Stauch · Host Sarah Guo, Elad GilStauch explains why Serval is challenging ServiceNow, how it approaches permissions and approvals, and why background agents remain difficult to deploy safely.
1:07ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 27, 2026PodcastGeorge Kurtz on How AI May Change CyberattacksGuest George KurtzThe CrowdStrike CEO describes his assessment of evolving threats.
39:23ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 15, 2026Podcast · 2 speakersThe Defenders' Window: Greg Brockman on AI-Generated Knowledge, Drug Discovery, and the New Physics of CybersecurityHost Sarah Guo, Elad GilOpenAI's president explains why solving open math problems is a proof point for science, how proactive agents change product form, and why security spending must exceed software spending.
1:04ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 14, 2026PodcastWalter Goodwin on the Slow Steps in Chip DesignGuest Walter GoodwinA chip founder explains why automation does not remove every computational bottleneck.
1:09:44ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 9, 2026Podcast · 2 speakers10,000 GB300s for Four Weeks: Reflection’s Open-Model LedgerHost Sarah Guo, Elad GilMisha Laskin tells No Priors how Reflection trained Beam, why reinforcement learning outspent pre-training, and why the West needs its own open weights.
0:39ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 8, 2026PodcastMichael Lee on Rethinking Organizations with AIGuest Michael LeeA business leader argues that the opportunity goes beyond faster tasks.