AI News in September 2024: Copilot, NotebookLM, o1 and World Models

Sam Altman posted about being home in the Midwest and looking forward to winter constellations. I see a possible hint at Orion, a rumored future OpenAI model, and wonder whether “winter” points to a release later in the year. That is my interpretation of a cryptic post, not a confirmed model name or release date.

Microsoft is revamping Copilot, which I thought needed a stronger reason to use it. Recall had been delayed, and simply opening an AI chat from a Windows key did not add much beyond the web experience. The new demonstrations put AI more directly into everyday work.

Copilot Pages combines web-backed answers and references with an editable page where teammates can collaborate and continue prompting AI. The interface is what interests me: an answer becomes a shared work product that people and AI can revise together.

In Excel, the demonstration uses natural-language requests to produce analysis and charts with Python. In PowerPoint, a prompt about commercial electric-vehicle charging stations produces a deck that can be reconfigured. Outlook adds summaries and follow-up suggestions. These are Microsoft demonstrations, so their performance in real work still needs testing.

Copilot Studio’s agent builder looks to me more like custom GPTs than the full agents built with other frameworks. You can connect documents and web search, including internal business information. Satya Nadella argues that as models become more capable and commoditized, value shifts toward grounding and adapting them to business data and workflows, and the interface around them.

I have made a similar argument about model commoditization, particularly after Llama’s release narrowed the gap on some tasks. I also see Microsoft’s relationships across the industry as a way to avoid putting all its AI bets on one model provider. Those are my assessments, not proof that all models are interchangeable.

A report about Meta’s use of public Facebook and Instagram posts raises another question: what data is being used to train AI? The reporting describes Meta using public adult posts and photos, including older material, and an Australian inquiry in which a Meta privacy executive confirmed the broad description after questioning.

Meta’s policy at the time distinguished public posts and comments from content shared with a nonpublic audience. I thought I remembered Mark Zuckerberg saying Facebook data was not used, but I cannot establish from this discussion whether he was referring to private data. The exact scope and opt-out rules depend on the region and time; the account should not be read as proof that every public post was necessarily used.

Mistral released Pixtral 12B, a vision model that can take an image and answer questions about it. I had not yet tested it when recording this video, so I cannot evaluate its performance from personal experience.

A former OpenAI employee introduced ell, an open-source language-model programming library. The idea is to treat prompts as programs and model parameters, with automatic versioning, tracing, and local visualization. In the example, a short Python function defines a joke prompt and is called with a topic.

The accompanying ell Studio interface offers local prompt version control, monitoring, and visualization, including multimodal examples. Tracking prompt changes matters when building production systems, especially agent workflows, because it lets developers see what changed and compare results.

Adobe announced a Firefly video model, with a public beta planned for later in 2024. Adobe described it as designed for commercial use and said its training approach relies on material it has rights to use. The examples include text-to-video and image-based generation, with camera controls and integration into Adobe tools presented as goals.

One example uses real footage of a child looking at a flower as a reference, then shows a generated clip for a possible additional shot. I like the idea of creating missing footage during editing, although these are selected promotional examples rather than an independent assessment of quality or rights.

Klarna’s CEO said the company was closing down Salesforce internally and planned to stop using Workday as part of a broader effort to simplify its software. I see a potential challenge to expensive software subscriptions as AI makes creating software easier, but the claim does not establish that every replacement has already been completed or that AI built the entire stack.

I am torn about that strategy. Engineers can underestimate the work of building and maintaining internal systems. At my own previous company, integrations with other platforms and the ongoing work of maintaining them were a major burden. Whether AI can reduce that burden enough to replace established software remains an open question.

Google’s new NotebookLM Audio Overview turns uploaded sources into a conversation between two AI hosts. The example sounds natural, and I can imagine listening to a research paper, though I am not yet sure when listening will be more effective than reading. Google notes that the overview reflects the uploaded sources rather than providing a comprehensive or objective treatment of a subject.

The ARC Prize team tested OpenAI’s newly released o1 models on visual reasoning puzzles intended to measure adaptation to unfamiliar tasks. In its September 2024 public evaluation using a baseline testing harness, o1-preview scored 21.2%, Claude 3.5 Sonnet 21%, o1-mini 12.8%, GPT-4o 9%, and Gemini 1.5 8%. These are results under that test setup, not a general measure of intelligence.

The chart also shows higher scores from systems using additional techniques; for example, MindsAI’s 46% is shown for the private evaluation, while the model baselines are on the public evaluation. Those numbers should not be treated as an identical test. I speculate that combining a stronger model with a specialized approach might improve results, but this is a possibility, not a demonstrated outcome.

Tencent’s GameGen-O project presents generated game-like video, including characters, environments, and actions. The project describes a diffusion transformer for open-world game generation; the examples show scenes resembling familiar game genres, including driving, flying, and walking. I think work like this could change games, but generated demonstration footage does not by itself establish a complete playable game.

Finally, Fei-Fei Li’s World Labs launched with reported funding of $230 million and a valuation of about $1 billion. Its stated aim is to build large world models with spatial intelligence: systems that can model and reason about objects, places, and interactions in three dimensions over time.

This connects to Yann LeCun’s argument that language models alone are insufficient to model the world. I also wonder what Tesla might do with its large collection of real-world video. Those are future possibilities; the World Labs announcement describes a goal, not a demonstrated general solution to spatial reasoning.