En 30 segundos
Shira Lazar asks a guest how AI tools are tested. The guest describes MMLU, multi-step reasoning, human evaluation of code generation, and ARC science questions; the short ends before the conversation is complete.
- 1MMLU and reasoningThe guest describes broad subject knowledge and multi-step reasoning tests.
- 2Human evaluationThey mention examining code generation, including Python.
- 3Science questionsThe guest describes ARC as school-level science questions requiring reasoning.
Conclusiones
- The guest describes MMLU as a test spanning many subjects.
- They mention multi-step reasoning, human evaluation of Python code and ARC science questions.
- The clip is an excerpt and does not show the complete evaluation process.
Lee el texto editado por capítulos
Testing a language model
Lazar asks about evaluations, and the guest describes several benchmarks and human checks. The excerpt ends mid-conversation.
Shira Lazar asks a guest about tests used for an AI tool like ChatGPT. The guest describes MMLU, expanding the name as Massive Multitask Language Understanding, and says it checks performance across many subjects. They also mention BBH as a multi-step reasoning test.
The guest goes on to mention human evaluation of code generation, including Python, and ARC questions that they describe as grade-school science questions requiring reasoning. Lazar says she had expected simpler tests; the clip ends before the guest finishes responding.
Más de este canal
0:45EN
Shira Lazar· 21 oct 2026PódcastA Visit to Paris Hilton’s Private SpaA visitor describes a pink spa, sensory treatments and a chilly cryotherapy session.
0:55ENShira Lazar· 19 oct 2026PódcastTyler Denk on Building a Creator Business Across FormatsInvitado: Tyler DenkThe Beehiiv founder describes creators combining channels, revenue models, and in-person events.
0:48ENShira Lazar· 12 oct 2026PódcastA Second Map of the Creator EconomyPresenta: Shira LazarShira Lazar introduces an updated ecosystem map with a section for independent creators.
0:54ENShira Lazar· 3 oct 2026Pódcast · 2 vocesChelsea Clinton on AI Anxiety, Health and GovernanceInvitado: Chelsea Clinton · Presenta: Shira LazarClinton discusses examples that give her hope and argues for treating AI as a collection of different technologies.
0:38ENShira Lazar· 21 sept 2026PódcastA Creator’s Visit to an LTK Event in DallasA short event recap highlights programming, brand displays, and conversations with women building creator businesses.
1:01ENShira Lazar· 19 sept 2026PódcastAffinity 100: A Creator Ranking Focused on Audience ConnectionA co-creator explains why the list emphasizes fandom, trust, and engagement instead of follower counts alone.