Why AI Judges Still Need Evaluation

The interviewer asks how to interpret claims that a model is more robust to jailbreaks: do obvious failures remain, or have the remaining cases become more subtle?

The speaker says mitigation can make failures subtler, but subtle does not mean harmless. Language models used as judges can have unusual, long-tail failure cases, as can refusal mechanisms.

There is a tradeoff between compute spent on an automated judge and its accuracy. The speaker says failures can still warrant human supervision; the clip does not quantify an error rate or define a universally safe threshold.