In 30 Sekunden
Shira Lazar asks a guest how AI tools are tested. The guest describes MMLU, multi-step reasoning, human evaluation of code generation, and ARC science questions; the short ends before the conversation is complete.
- 1MMLU and reasoningThe guest describes broad subject knowledge and multi-step reasoning tests.
- 2Human evaluationThey mention examining code generation, including Python.
- 3Science questionsThe guest describes ARC as school-level science questions requiring reasoning.
Das Wichtigste
- The guest describes MMLU as a test spanning many subjects.
- They mention multi-step reasoning, human evaluation of Python code and ARC science questions.
- The clip is an excerpt and does not show the complete evaluation process.
Den bearbeiteten Text nach Kapiteln lesen
Testing a language model
Lazar asks about evaluations, and the guest describes several benchmarks and human checks. The excerpt ends mid-conversation.
Shira Lazar asks a guest about tests used for an AI tool like ChatGPT. The guest describes MMLU, expanding the name as Massive Multitask Language Understanding, and says it checks performance across many subjects. They also mention BBH as a multi-step reasoning test.
The guest goes on to mention human evaluation of code generation, including Python, and ARC questions that they describe as grade-school science questions requiring reasoning. Lazar says she had expected simpler tests; the clip ends before the guest finishes responding.
Mehr aus diesem Kanal
0:45EN
Shira Lazar· 21. Okt. 2026PodcastA Visit to Paris Hilton’s Private SpaA visitor describes a pink spa, sensory treatments and a chilly cryotherapy session.
0:55ENShira Lazar· 19. Okt. 2026PodcastTyler Denk on Building a Creator Business Across FormatsGast: Tyler DenkThe Beehiiv founder describes creators combining channels, revenue models, and in-person events.
0:48ENShira Lazar· 12. Okt. 2026PodcastA Second Map of the Creator EconomyModeration: Shira LazarShira Lazar introduces an updated ecosystem map with a section for independent creators.
0:54ENShira Lazar· 3. Okt. 2026Podcast · 2 StimmenChelsea Clinton on AI Anxiety, Health and GovernanceGast: Chelsea Clinton · Moderation: Shira LazarClinton discusses examples that give her hope and argues for treating AI as a collection of different technologies.
0:38ENShira Lazar· 21. Sept. 2026PodcastA Creator’s Visit to an LTK Event in DallasA short event recap highlights programming, brand displays, and conversations with women building creator businesses.
1:01ENShira Lazar· 19. Sept. 2026PodcastAffinity 100: A Creator Ranking Focused on Audience ConnectionA co-creator explains why the list emphasizes fandom, trust, and engagement instead of follower counts alone.