30 秒でわかる
Shira Lazar asks a guest how AI tools are tested. The guest describes MMLU, multi-step reasoning, human evaluation of code generation, and ARC science questions; the short ends before the conversation is complete.
- 1MMLU and reasoningThe guest describes broad subject knowledge and multi-step reasoning tests.
- 2Human evaluationThey mention examining code generation, including Python.
- 3Science questionsThe guest describes ARC as school-level science questions requiring reasoning.
ポイント
- The guest describes MMLU as a test spanning many subjects.
- They mention multi-step reasoning, human evaluation of Python code and ARC science questions.
- The clip is an excerpt and does not show the complete evaluation process.
チャプターごとに整えた全文を読む
Testing a language model
Lazar asks about evaluations, and the guest describes several benchmarks and human checks. The excerpt ends mid-conversation.
Shira Lazar asks a guest about tests used for an AI tool like ChatGPT. The guest describes MMLU, expanding the name as Massive Multitask Language Understanding, and says it checks performance across many subjects. They also mention BBH as a multi-step reasoning test.
The guest goes on to mention human evaluation of code generation, including Python, and ARC questions that they describe as grade-school science questions requiring reasoning. Lazar says she had expected simpler tests; the clip ends before the guest finishes responding.
同じチャンネルの記事
0:45EN
Shira Lazar· 2026年10月21日ポッドキャストA Visit to Paris Hilton’s Private SpaA visitor describes a pink spa, sensory treatments and a chilly cryotherapy session.
0:55ENShira Lazar· 2026年10月19日ポッドキャストTyler Denk on Building a Creator Business Across Formatsゲスト Tyler DenkThe Beehiiv founder describes creators combining channels, revenue models, and in-person events.
0:48ENShira Lazar· 2026年10月12日ポッドキャストA Second Map of the Creator Economyホスト Shira LazarShira Lazar introduces an updated ecosystem map with a section for independent creators.
0:54ENShira Lazar· 2026年10月3日ポッドキャスト · 2 人Chelsea Clinton on AI Anxiety, Health and Governanceゲスト Chelsea Clinton · ホスト Shira LazarClinton discusses examples that give her hope and argues for treating AI as a collection of different technologies.
0:38ENShira Lazar· 2026年9月21日ポッドキャストA Creator’s Visit to an LTK Event in DallasA short event recap highlights programming, brand displays, and conversations with women building creator businesses.
1:01ENShira Lazar· 2026年9月19日ポッドキャストAffinity 100: A Creator Ranking Focused on Audience ConnectionA co-creator explains why the list emphasizes fandom, trust, and engagement instead of follower counts alone.