GPT-4o's Early Demos: Voice, Tutoring, Meetings and Accessibility

A few days after GPT-4o's announcement in May 2024, Matthew Berman reviews the model and OpenAI's examples. He says some model capabilities were available, but the new voice experience was not yet accessible to him. His assessment of voice is therefore based on demonstrations rather than his own extended testing.

In the first example, an OpenAI employee shows the assistant a production setup and leads it toward guessing that an announcement concerns the model itself. Berman is impressed by the combination of camera input and conversation, while finding the selected voice unusually flirtatious and sometimes awkward.

Greg Brockman introduces two assistants to one another. One has a camera view and is asked to describe the environment; the other asks questions. Berman notices a change in delivery when Brockman asks one assistant to pause, and highlights how quickly the interaction appears to proceed.

The visually enabled assistant describes Brockman's clothing and the room to the other system. Berman says the conversational style is more complimentary and verbose than he would prefer. At the time of recording, he had tried an older voice experience and could not reproduce the new demonstration's latency or expression.

A person briefly appears behind Brockman and makes a playful gesture. The assistants continue their conversation until Brockman asks if anything unusual happened, after which the visually enabled assistant describes the event. Berman says he could not clearly see the gesture in the phone view himself.

The assistants then attempt to sing alternating lines about the scene. Berman finds the combination of recent visual context, rhyming and alternating voices striking. The performance demonstrates generated speech in a staged interaction; expressive delivery does not mean the systems feel emotions.

An interview-preparation example asks the assistant to comment on a presenter's appearance and clothing. Berman considers how a personal-sounding voice and conversational history could affect companion-style applications, while making clear that he is extrapolating beyond the particular demo.

In a rock-paper-scissors demonstration, two people ask the assistant to act like a sports commentator and count them in. It describes the rounds and identifies a winner from the camera view. Berman is interested in its apparent ability to associate names with the participants, though a demonstration is not a general identification test.

A shorter example asks for a sarcastic delivery. The voice changes accordingly, illustrating the range of tone a spoken interface can convey. Berman sees such expressive control as one of the features that distinguishes the new voice demonstrations from earlier interactions.

Khan Academy founder Sal Khan and his son Imran try a geometry tutoring exercise. Khan asks the assistant to guide the student with questions instead of simply giving the answer. Berman sees promise in this approach for learning, provided that explanations and feedback are checked for accuracy.

The assistant asks the student to identify the sides of a right triangle relative to an angle. When he is uncertain, it describes the hypotenuse and adjacent side, then asks him to try again. Berman emphasizes the use of voice alongside the exercise displayed on the device.

Berman imagines assistants that can see device context and eventually take actions, but distinguishes that possibility from the demo. He notes that operating-system access, personal information and actual task execution introduce separate product and permission questions.

A meeting example features people debating cats and dogs while an assistant listens from one participant's desktop. Berman corrects his initial description: the assistant is not actually a meeting participant. It recalls preferences and summarizes the discussion; he imagines future follow-up messages and action items.

Another clip demonstrates translation between English and Spanish. Berman sees real-time interpretation as a valuable use of voice interaction, while the short example does not establish translation quality across languages, accents or difficult settings.

A Be My Eyes example shows visual descriptions for a person with visual impairment, including a landmark, ducks and a taxi. Berman considers responsiveness important for accessibility. Such demonstrations should not be treated as a guarantee of accurate navigation or a substitute for established mobility aids.

A staged customer-service call has an assistant seek a replacement for a faulty phone. Berman imagines delegating calls or returns to an assistant, but also raises concerns about scammers and spam using similar tools. The example is a demonstration, not a live test against an independent customer-service representative.

Berman closes by browsing other exploratory examples: turning a photo into a caricature, summarizing a lecture and producing views used in a 3D reconstruction. He wonders about the underlying context and capabilities, and looks forward to testing the voice mode when he can access it.