Andrej Karpathy on Context Windows and Model Fine-Tuning

Andrej Karpathy argues that the methods for deeply adapting AI models are still developing. He gives fine-tuning without loss of existing capabilities as one challenge.

He contrasts context windows, which he describes as a relatively cheap and useful way to customize behavior, with changing the model itself.

Karpathy says continual learning and specialization require better ways to adjust model weights and improve performance in a particular area.

Changing weights can affect the whole model and its capabilities, making it trickier than changing context. The speakers add that such customization must also be economical.