A Camera Narrator Using GPT-4 Vision and ElevenLabs

SUMMARY

In 30 seconds

Matthew Berman demonstrates Charlie Holtz's Narrator project, which combines GPT-4 Vision with ElevenLabs speech synthesis. He walks through setup, creates a voice resembling David Attenborough and troubleshoots the output. The demonstration produces audio files, although direct playback in his terminal fails; the interfaces and prices described are historical.

1Nature-style narrationThe opening example narrates a person and their surroundings in a nature-documentary style.
2Vision and speechThe project uses GPT-4 Vision to describe camera frames and ElevenLabs to generate a spoken voice.
3Local setupBerman clones the project, sets up a Python environment and configures API credentials.
4Voice setupHe creates a voice clone, selects a faster speech model and locates its voice ID.
5Troubleshooting outputAfter adjusting API-key handling, he finds an audio file and plays it outside the terminal.

Takeaways

The project captures webcam frames, describes them with a vision model and generates spoken narration.
Berman configures Python dependencies and separate API credentials for OpenAI and ElevenLabs.
His 2023 demonstration uses a voice clone and a voice ID, and he says Attenborough has rights to his own voice.
He fixes a key-handling problem and plays the resulting audio file after terminal playback fails.
The example's model availability, interface and pricing reflect the time of recording.

TopicsVoice AI Multimodal AI Vision & video

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

A camera narrator

Berman presents the narration example and credits Charlie Holtz's project. It combines image analysis and generated speech.

A generated David Attenborough-like voice narrates what my camera sees, describing my colorful clothing, microphone and movements in the style of a nature documentary. I'm going to show how this works. Charlie Holtz first shared the project, which uses GPT-4 Vision for camera images and ElevenLabs for speech.

Install and configure

He sets up the repository and credentials, then starts camera capture and narration processes.

I clone the Narrator repository, enter its directory and create a Python 3.11 Conda environment. After activating it, I install the requirements using that environment's Python. The demonstration requires access to the vision API and an ElevenLabs account; I configure their API keys as environment variables.

In the 2023 ElevenLabs interface I locate the API key under the account profile. I also create an OpenAI secret key and say I will revoke the one displayed in the video. The process uses two terminal sessions: one runs the camera capture script and asks for camera permission; the other analyzes frames and produces narration.

Voice setup and troubleshooting

Berman creates a voice clone and sets its ID. He adjusts key handling and plays a generated audio file after direct playback fails.

The first attempt reports that the configured voice ID cannot be found. I say Attenborough has rights to his own voice and describe my use as educational. I upload an audio sample to ElevenLabs' instant voice cloning feature; at the time, this requires a paid subscription, while a separate voice-design option is free. These are the settings and prices shown in 2023.

I test the cloned voice with text already generated by the project and find that it sounds convincing. I switch from the multilingual voice model to a faster Turbo option. Then I use the voices API to find the clone's ID and replace the hardcoded ID in the narrator script.

SLIDE 1 / 1Testing the generated narrationThe speech interface shows generated narration text and the selected voice settings.

The next run appears not to use my API key, so I change the script to set the ElevenLabs key explicitly. Direct playback in the terminal still does not work, but I find the generated audio files and open one separately. It plays a narration describing me at my computer.