A First AI Livestream: Model Rumors, RAG, Agents, and Audience Questions

Matthew Berman tests a livestream in August 2024 and answers viewers’ questions about AI tools, model evaluation, company knowledge, and future agents.

MBMatthew BermanAnime · Forward Future
Chaîne
Matthew Berman
Publié
12 août 2024
Vidéo
57:55
Lecture
8 min
Diapositives
4
Original
anglais
Ouvrir sur YouTube
En faisant défiler, la vidéo se réduit en bas à droite et continue. Les sous-titres suivent votre langue de lecture.
Langue de lectureDisponible en 2 langues
Cet article n’est pas encore disponible en français ; il s’affiche dans l’original en anglais.
RÉSUMÉ

En 30 secondes

In an informal August 2024 test livestream, Matthew Berman discusses rumors about OpenAI, universal basic income, ARC-AGI, retrieval-augmented generation, hardware, AI safety, and his own work. He also tests GPT-4o on word and letter counting and a family puzzle. Much of the conversation concerns speculation or historical product details; his advice and judgments are not controlled evaluations.

  1. 1Exploring universal basic incomeBerman hopes to interview Scott Santens and acknowledges he is still learning about UBI and inflation.
  2. 2Rumors and social-media hypeHe discusses an anonymous account and possible OpenAI releases while stressing that the claims are unconfirmed.
  3. 3ARC-AGIHe examines visual pattern tasks and a 2024 leaderboard, viewing the benchmark as interesting evidence about skill acquisition.
  4. 4RAG or fine-tuningHe generally recommends retrieval for supplying company documents, though the choice depends on requirements and evaluation.
  5. 5Citing source documentsHe discusses storing source metadata alongside document chunks so a retrieval system can link back to passages.
  6. 6Live GPT-4o testsThe model miscounts letters in strawberry but answers a family puzzle correctly in the shown interactions.

À retenir

  1. Berman repeatedly identifies GPT-4o “large” and Strawberry claims as unconfirmed rumors at the time of the stream.
  2. He favors retrieval for questions over company documents, while fine-tuning and retrieval solve different problems and can be combined.
  3. Document citations require preserving and retrieving source information; they do not by themselves guarantee an answer is correct.
  4. His claims that language models or autonomous wrappers are not dangerous are personal assessments, not a comprehensive safety finding.
  5. His ARC-AGI leaderboard, pricing, hardware, and product remarks reflect the 2024 stream and are not current recommendations.
  6. A few live prompt results illustrate behavior but do not establish general model performance.
TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Transcription seule
01

Testing the stream and discussing UBI

Berman adjusts audio and talks about a possible interview on universal basic income. He asks viewers for feedback and resources.

This is my first test livestream, so I begin by checking whether viewers can hear me and adjusting the sound. I plan a more formal weekly-news stream and invite feedback on the setup. Several early minutes are devoted to resolving audio and screen-sharing issues.

I have been asking for resources on universal basic income because I may interview Scott Santens, who has discussed the subject extensively. The interview is not yet arranged. My initial question is whether giving everyone money would lead to inflation, but I am not an economist and want to learn enough to have a substantive conversation rather than assume the answer.

02

OpenAI rumors and community questions

He discusses social-media speculation about model releases, a poll, and product preferences. He treats the purported release as unverified.

Viewers ask about the anonymous X account “I rule the world” and its Strawberry-related posts. I have watched its audience grow and suspect it is adept at generating attention. It may know something or may simply be amplifying rumors; I do not know its identity, and I have seen claims from the account that I believe were false.

The account has also mentioned a purported GPT-4o “large” and a repository related to Strawberry and Q*. I do not have independent confirmation, and I cannot say whether anything in its posts is genuinely new. I start a poll about a possible release and speculate that a recognizable emoji and engagement helped the story spread. None of this verifies the rumored model or its relationship to other projects.

My editor and channel manager Alex McCraw is helping moderate. I explain that he took over editing after I had made roughly 180 to 200 videos myself, allowing me to produce more. Asked whether “GPT-4o large” refers to a 64K output window, I say no: the rumor concerns a possible more capable model, not a confirmed context-window specification.

I like Anthropic’s models, though I often return to ChatGPT and find Anthropic’s website somewhat slow in my experience. A viewer asks about speaking to people building AI products; I talk with founders regularly and could invite some to a future Q&A. I also like ChatGPT’s web-search experience, but these are impressions of products at the time of the stream.

Asked whether AI could control matter at the atomic level or create unimaginable supercomputers, I mention Mustafa Suleyman’s book The Coming Wave and its discussion of AI and biotechnology. I do not know enough about atomic-scale engineering to make a prediction. The question is interesting, but the stream offers no evidence that such capabilities exist.

03

ARC-AGI, brain interfaces, and investment

He reviews the ARC-AGI site and responds to questions about Neuralink and Tesla investing in xAI. These are personal views and historical observations.

I pull up ARC-AGI, a set of visual grid tasks in which a system must infer a transformation from examples. I find the contrast between apparently simple human puzzles and difficult model tasks compelling. The historical 2024 prize page displays a leaderboard entry around 43% and an 85% grand-prize target, but a score on this benchmark is one measure of task performance, not proof of or a complete definition of AGI.

ARC-AGI visual tasks
DIAPOSITIVE 1 / 4ARC-AGI visual tasksThe page shows example input and output grids used to explain the benchmark.

I am partway through The Coming Wave and find it interesting. I would also like to interview Yann LeCun. A viewer asks about Elon Musk and brain interfaces; after remembering the name Neuralink, I discuss his possible long-term idea of augmenting people alongside more capable AI. That is speculative commentary, not a demonstrated route to superintelligence or a medical claim.

On a question about Tesla potentially investing in xAI, I refer to Musk’s social-media poll and say the businesses might complement one another, including in robotics and communication. I would like xAI to release open weights and build an ecosystem. A poll and my opinion are not an announced investment decision or financial recommendation.

Asked whether I have access to GPT-4o “large,” I reiterate that I do not know whether it exists. I speculate about how online hype and a company’s response could interact, but I have no evidence for any coordination. A viewer challenges me on language-model safety; I ask them to explain their concern.

04

Safety, retrieval, and company documents

The stream turns to AI risks, RAG versus fine-tuning, laptop choices, and ways to build a document chatbot. Berman and viewers suggest several tools.

I consider live model testing for a future stream and answer a question about Rabbit R1: I no longer use it, though I appreciate the attempt to build a new device and have not tested its later updates. When a viewer worries about model knowledge and autonomous systems, I say I do not consider current systems dangerous in that form. That is my assessment, not a general safety conclusion: risks depend on capabilities, access, deployment, and misuse.

A viewer asks whether to use RAG or train a model on customer information. My default recommendation is retrieval-augmented generation: retrieve relevant material and supply it as context when answering. I contrast that with fine-tuning for behavior and style. The distinction is useful, but fine-tuning can also affect skills and knowledge, and RAG, memory, and fine-tuning are not identical or mutually exclusive; the right design depends on evaluation and requirements.

Asked for an AI laptop under $1,000 for a college student, I say the choice depends on whether she wants to run models locally or use a cloud chatbot. I mention then-new Copilot+ laptops and their neural processing units, while noting that they will not match a powerful desktop GPU. This is a general 2024 impression, not a specific tested purchase recommendation.

Another viewer wants a natural-language chatbot for company policies and procedures. I describe RAG options ranging from a hosted service to a locally built pipeline with document chunking and a vector database. I mention a previous Pinecone-sponsored tutorial and note that Pinecone is not required. The choice also depends on how much customization and coding the team wants.

Viewers suggest uploading documents to a custom GPT or using Claude projects for a simple start, and mention GraphRAG, AWS Bedrock, and Azure AI Studio for other setups. I have not tried all of these suggestions. A document system also needs appropriate access controls, data handling, retrieval testing, and citation checks; uploading documents alone does not establish security or correctness.

05

Tools, pricing, and career

He looks at LangGraph Studio and Pinecone’s then-current pricing. He discusses his background, investing, and why he wants to keep making videos.

I am interested in GraphRAG and LangGraph Studio. I show a LangGraph Studio interface for visualizing an agent graph and its execution steps, which I find appealing because it exposes where an agent is in a workflow. This is a look at the interface, not a detailed evaluation of its reliability.

LangGraph Studio interface
DIAPOSITIVE 2 / 4LangGraph Studio interfaceA graph view illustrates an agent workflow and the ability to inspect its execution.

Asked about Pinecone pricing, I disclose that the company has sponsored videos and that I know people there. I show the free tier and describe the storage and vector limits displayed in August 2024; I had not needed a paid account. Prices and limits change, so viewers should consult the provider rather than rely on the historical numbers shown.

Pinecone’s historical pricing page
DIAPOSITIVE 3 / 4Pinecone’s historical pricing pageThe page shows tiers and limits as displayed in the 2024 stream; current terms may differ.

A viewer asks whether anyone is building something like an idea I previously called abstract AI. I have received messages from people exploring similar ideas. My view is that execution and a capable team often matter more than an idea alone, and this influences how I think about investments and confidentiality agreements.

I describe my own background: I started programming young, switched from a computer-science major to economics, and learned much of my programming independently. I have built prototypes and parts of software businesses, though I have not held a full-time engineering position. For now I enjoy making videos, do some consulting, and do not want to start another venture-backed company.

The informal poll about a rumored GPT-4o release has roughly 700 votes, with 63% saying yes. That measures what participating viewers expect, not whether the rumor is true. Asked about investing in an abstract-AI team, I say I would consider the market and my own expertise, then focus heavily on the team and its execution.

A viewer asks for document retrieval with source links on a Mac. I describe storing metadata alongside document chunks in a vector index, so retrieved passages can point back to their originating document and location. A complete implementation also needs to extract text from PDFs and map citations accurately; the explanation is an outline, not a turnkey implementation.

06

Source citations and future platforms

Berman explains a possible document-metadata approach for citations, then shares views on Google, Apple, AI agents, and AI rights.

I say I am enjoying the stream and plan another session. Asked about Gemini, I report that its earlier performance in my own test disappointed me, while seeing potential in Google’s access to tools such as Gmail, Calendar, and Drive. I also think Apple’s hardware and applications could help it deliver useful agents. These are personal 2024 assessments, not proof of future product performance.

I speculate that a company with hardware, models, and access to everyday applications could build agents that act on a user’s behalf. I include Meta and Microsoft among companies with potentially useful platforms. A viewer raises rights for a hypothetical sentient AI; I do not have a settled answer and say that, if systems were ever understood to be sentient, their moral status would need thought.

I have not tried Apple Intelligence developer previews because my iPhone at the time is not supported. I suggest Apple often enters markets after others and then integrates its products in its own way, while acknowledging I do not know how well its AI approach will work.

07

Live tests and closing questions

He tests GPT-4o on counting and a family puzzle, discusses video tools and FAISS, and looks up Google’s YouTube acquisition. He closes the test stream.

A viewer asks whether GPT-4o now counts characters correctly. I test how many words are in its next response and find the answer wrong, then ask how many r’s are in “strawberry.” The model says two; there are three. These are individual interactions, not a statistical measure of the model’s overall performance.

A viewer offers a family puzzle: Alice has three brothers and two sisters; how many sisters does one of her brothers have? The model answers three—Alice plus her two sisters—which is correct under the stated family relationships. I ask viewers to include solutions when suggesting future tests so I can check the result.

Another question concerns AI choosing camera angles for streams or sports. I mention existing game-playing systems and automated video tools, but I am not certain which live-broadcast systems perform the exact requested task. I also look at Flowise and FAISS, saying I have not studied them closely enough to offer a detailed assessment.

Asked about jailbroken or uncensored models, I say I have tested some and that I have sometimes observed lower quality. I do not know enough about the causes to generalize from those examples. I mention video-editing tracking tools and sports production as possible related technologies, without claiming they solve the viewer’s exact problem.

The discussion ends with Google’s YouTube acquisition. I look up the 2006 announcement and see the $1.65 billion stock transaction, which I regard as an exceptionally successful acquisition in hindsight. I note that YouTube was smaller and faced copyright-related uncertainty at the time.

Google’s YouTube acquisition announcement
DIAPOSITIVE 4 / 4Google’s YouTube acquisition announcementThe historical press release states the proposed $1.65 billion stock transaction.

I thank viewers for helping test the stream and say I plan another session on Friday at 10 a.m. Pacific. The live questions and poll shaped the conversation, and I intend to make future sessions more organized.

Sur la même chaîne

12 diapositives34:21EN
Matthew Berman· 27 oct. 2026Matthew Berman Examines Anthropic’s IPO Filing, AI Growth and GovernanceMatthew Berman, Sam Altman · Forward FutureThe presenter reviews the S-1’s financial metrics, compute strategy and voting structure while questioning the tension between rapid AI development and safety.AnthropicAI SafetyIA en entreprise+2
2 diapositives0:36EN
Matthew Berman· 25 oct. 2026Kimi K3.1 and a Rubik’s Cube Solver DemonstrationMatthew Berman · Forward FutureMatthew Berman shows a scrambled virtual cube being solved and gives the Kimi K3.1 test a favorable assessment.ÉvaluationsAgents de codeOpen-weight models
6 diapositives25:50EN
Matthew Berman· 24 oct. 2026Kimi K3.1 Pro Tested: Strong 3D Demos, Uneven Results and Competitive PricingMatthew Berman · Forward FutureMatthew Berman tests the open-weight model on coding, 3D rendering and visual explanations, then compares benchmark results and prices.Open-weight modelsBenchmarksAI cost & performance+1
2 diapositives32:20EN
Matthew Berman· 23 oct. 2026Who Is Local AI For? The Costs, Trade-offs and a Hybrid FutureMatthew Berman · Forward FutureMatthew Berman weighs data control and independence against hardware cost, setup complexity and inference speed.Local inferenceOpen-weight modelsAI cost & performance+1
6 diapositives1:03:37EN
Matthew Berman· 23 oct. 2026Matthew Berman’s Team Tests Kimi K3.1 on Animation, Games, and ChartsMatthew Berman, Alex, Brian · Forward FutureA live team AMA turns into an informal test of Kimi K3.1, from personal-agent workflows to motion graphics, a browser game, and a benchmark chart.Agents de codeGenerative mediaÉvaluations+2
0:53EN
Matthew Berman· 22 oct. 2026Sam Altman on AI and the Discovery of New KnowledgeSam Altman · OpenAIAltman expects progress in mathematical discovery to extend to biology, materials and energy.AI for scienceMathematicsComputational Biology+1