AI News: Copilot Agents, Claude Computer Use and New Generative Media

This week brought a striking number of AI announcements, including enterprise agents, new models, and tools for generating images, video and voices. I want to run through the releases and explain what caught my attention as of October 2024.

Microsoft announced that autonomous agents in Copilot Studio would enter public preview in November. Instead of waiting for a person to prompt them each time, agents can be configured to react to events and work through a process using business data and connected systems. Microsoft also announced ten autonomous agents for Dynamics 365 across areas such as sales, service, finance and supply chains.

The examples include monitoring a shared inbox and identifying customer requests, gathering relevant information, and handing work to the right person. The appeal is the possibility of handling routine steps around the clock. I think the next phase will be more agents embedded in the software companies already use, although a demo or announcement does not establish how reliably every workflow will run.

Salesforce CEO Marc Benioff has criticized Microsoft’s Copilot, calling it “Clippy 2.0,” while promoting Salesforce’s own Agentforce. I disagree with dismissing Microsoft’s position: its distribution and established enterprise relationships could help it reach a great many customers. That is my view of a competitive market, not proof that either vendor has already won it.

IBM introduced Granite 3.0, including 8-billion- and 2-billion-parameter language models and mixture-of-experts variants. IBM released the models under Apache 2.0 and positioned them for business use. I am interested in InstructLab, IBM’s approach to adapting models with synthetic data and training, and plan to cover the technology in more detail after visiting IBM.

Anthropic released an upgraded Claude 3.5 Sonnet and announced Claude 3.5 Haiku for later release. Anthropic said the forthcoming Haiku matched or exceeded the earlier, larger Claude 3 Opus on selected evaluations; that is a comparison on those tests, not a guarantee that it is better at every task.

The headline feature is experimental computer use: developers can let Claude inspect screenshots and issue mouse and keyboard actions through an API. In one example, the model moves between a spreadsheet and a web form to enter information. Anthropic also showed that it can make mistakes or become distracted, so the early demonstration is useful evidence of potential, not proof of dependable autonomous operation.

Meta’s FAIR team announced several research releases. Segment Anything Model 2.1 is an update for identifying and tracking objects in images and video, while Meta Spirit LM combines text and speech. These are different research projects, illustrating the range of ways models can work across visual and audio inputs.

A report said former OpenAI CTO Mira Murati was exploring a new AI startup and could seek more than $100 million, potentially with former OpenAI colleagues. At the time, the reporting described preliminary discussions; the funding and plans were not a confirmed completed deal. Given her experience, I will be watching what she builds.

I also show a Chinese humanoid robot whose appearance reminds me of Tesla’s Optimus. The resemblance is noticeable in the footage, but the clip does not establish copying, theft or the robot’s comparative capabilities. A short demonstration cannot tell us how well it would perform in ordinary work.

Stability AI announced Stable Diffusion 3.5 Large and Large Turbo, with a Medium model planned for the following week. The examples show a range of image styles and detail. The models use Stability’s community license: qualifying organizations with annual revenue up to $1 million could use them commercially under that license, while larger commercial users needed an enterprise agreement.

Ideogram introduced Canvas, a workspace for editing and expanding images. Magic Fill lets a user select an area and describe what should replace it; Extend adds content beyond the original image’s boundaries. These editing tools are useful because a generated picture often needs a local change rather than an entirely new prompt.

LM Studio 0.3.5 added a headless mode, allowing local models to run as a background service without keeping the graphical interface open. That can make a local model easier to use from other applications. I am an investor in LM Studio, so readers should take that relationship into account when considering my enthusiasm.

Perplexity also introduced a reasoning-oriented search experience intended to work through harder questions. I speculate about whether OpenAI’s o1 might be involved, but that is not confirmed by the information I show. At the time Perplexity also offered the updated Claude 3.5 Sonnet to Pro users, giving subscribers another model choice.

Genmo released Mochi 1, an open video-generation model, along with code and weights. The clips shown have compelling motion and visual quality. They are selected examples, however, and running the model locally requires suitable hardware; the examples alone do not show its average performance or consistency.

Runway announced Act-One for Gen-3 Alpha, using a driving performance and a character reference to produce animated character footage. The demonstrations suggest a way to transfer facial expression and movement to a generated character. As with other product videos, individual results depend on the input and workflow.

xAI opened a Grok API beta with prices at the time of $5 per million input tokens and $15 per million output tokens. Those are historical launch prices rather than a statement about today’s pricing. An API gives developers a way to integrate the model into their own software.

ElevenLabs introduced Voice Design in beta, allowing users to describe a voice in text and generate a synthetic voice from that description. I play examples including a yeti and a witch to show the range of characters the tool can create. The voices are generated examples, not additional guests in the video.

That wraps up this week’s collection of AI news. Between agent builders, new models and creative tools, there is a lot to try and compare as these releases become available.