A Camera Narrator Using GPT-4 Vision and ElevenLabs

RÉSUMÉ

En 30 secondes

Matthew Berman demonstrates Charlie Holtz's Narrator project, which combines GPT-4 Vision with ElevenLabs speech synthesis. He walks through setup, creates a voice resembling David Attenborough and troubleshoots the output. The demonstration produces audio files, although direct playback in his terminal fails; the interfaces and prices described are historical.

1Nature-style narrationThe opening example narrates a person and their surroundings in a nature-documentary style.
2Vision and speechThe project uses GPT-4 Vision to describe camera frames and ElevenLabs to generate a spoken voice.
3Local setupBerman clones the project, sets up a Python environment and configures API credentials.
4Voice setupHe creates a voice clone, selects a faster speech model and locates its voice ID.
5Troubleshooting outputAfter adjusting API-key handling, he finds an audio file and plays it outside the terminal.

À retenir

The project captures webcam frames, describes them with a vision model and generates spoken narration.
Berman configures Python dependencies and separate API credentials for OpenAI and ElevenLabs.
His 2023 demonstration uses a voice clone and a voice ID, and he says Attenborough has rights to his own voice.
He fixes a key-handling problem and plays the resulting audio file after terminal playback fails.
The example's model availability, interface and pricing reflect the time of recording.

ThèmesIA vocale Multimodal AI Vision et vidéo

TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Afficher la transcription

Transcription seule

A camera narrator

Berman presents the narration example and credits Charlie Holtz's project. It combines image analysis and generated speech.

A generated David Attenborough-like voice narrates what my camera sees, describing my colorful clothing, microphone and movements in the style of a nature documentary. I'm going to show how this works. Charlie Holtz first shared the project, which uses GPT-4 Vision for camera images and ElevenLabs for speech.

Install and configure

He sets up the repository and credentials, then starts camera capture and narration processes.

I clone the Narrator repository, enter its directory and create a Python 3.11 Conda environment. After activating it, I install the requirements using that environment's Python. The demonstration requires access to the vision API and an ElevenLabs account; I configure their API keys as environment variables.

In the 2023 ElevenLabs interface I locate the API key under the account profile. I also create an OpenAI secret key and say I will revoke the one displayed in the video. The process uses two terminal sessions: one runs the camera capture script and asks for camera permission; the other analyzes frames and produces narration.

Voice setup and troubleshooting

Berman creates a voice clone and sets its ID. He adjusts key handling and plays a generated audio file after direct playback fails.

The first attempt reports that the configured voice ID cannot be found. I say Attenborough has rights to his own voice and describe my use as educational. I upload an audio sample to ElevenLabs' instant voice cloning feature; at the time, this requires a paid subscription, while a separate voice-design option is free. These are the settings and prices shown in 2023.

I test the cloned voice with text already generated by the project and find that it sounds convincing. I switch from the multilingual voice model to a faster Turbo option. Then I use the voices API to find the clone's ID and replace the hardcoded ID in the narrator script.

DIAPOSITIVE 1 / 1Testing the generated narrationThe speech interface shows generated narration text and the selected voice settings.

The next run appears not to use my API key, so I change the script to set the ElevenLabs key explicitly. Direct playback in the terminal still does not work, but I find the generated audio files and open one separately. It plays a narration describing me at my computer.