30초 요약
Shira Lazar asks a guest how AI tools are tested. The guest describes MMLU, multi-step reasoning, human evaluation of code generation, and ARC science questions; the short ends before the conversation is complete.
- 1MMLU and reasoningThe guest describes broad subject knowledge and multi-step reasoning tests.
- 2Human evaluationThey mention examining code generation, including Python.
- 3Science questionsThe guest describes ARC as school-level science questions requiring reasoning.
핵심 정리
- The guest describes MMLU as a test spanning many subjects.
- They mention multi-step reasoning, human evaluation of Python code and ARC science questions.
- The clip is an excerpt and does not show the complete evaluation process.
챕터별로 다듬은 전문 읽기
Testing a language model
Lazar asks about evaluations, and the guest describes several benchmarks and human checks. The excerpt ends mid-conversation.
Shira Lazar asks a guest about tests used for an AI tool like ChatGPT. The guest describes MMLU, expanding the name as Massive Multitask Language Understanding, and says it checks performance across many subjects. They also mention BBH as a multi-step reasoning test.
The guest goes on to mention human evaluation of code generation, including Python, and ARC questions that they describe as grade-school science questions requiring reasoning. Lazar says she had expected simpler tests; the clip ends before the guest finishes responding.
같은 채널의 다른 글
0:45EN
Shira Lazar· 2026년 10월 21일팟캐스트A Visit to Paris Hilton’s Private SpaA visitor describes a pink spa, sensory treatments and a chilly cryotherapy session.
0:55ENShira Lazar· 2026년 10월 19일팟캐스트Tyler Denk on Building a Creator Business Across Formats게스트 Tyler DenkThe Beehiiv founder describes creators combining channels, revenue models, and in-person events.
0:48ENShira Lazar· 2026년 10월 12일팟캐스트A Second Map of the Creator Economy진행자 Shira LazarShira Lazar introduces an updated ecosystem map with a section for independent creators.
0:54ENShira Lazar· 2026년 10월 3일팟캐스트 · 2명Chelsea Clinton on AI Anxiety, Health and Governance게스트 Chelsea Clinton · 진행자 Shira LazarClinton discusses examples that give her hope and argues for treating AI as a collection of different technologies.
0:38ENShira Lazar· 2026년 9월 21일팟캐스트A Creator’s Visit to an LTK Event in DallasA short event recap highlights programming, brand displays, and conversations with women building creator businesses.
1:01ENShira Lazar· 2026년 9월 19일팟캐스트Affinity 100: A Creator Ranking Focused on Audience ConnectionA co-creator explains why the list emphasizes fandom, trust, and engagement instead of follower counts alone.