Text-to-Video in 2023: Runway, Colab and a Local Zeroscope Setup

Text-to-video is starting to produce impressive examples. I will show Runway’s hosted Gen-2 and an alternative using a Zeroscope notebook in Google Colab or on a local machine. This walkthrough was published in June 2023, so product access, pricing, dependencies and available models may have changed.

Runway Gen-2 was publicly available with a limited free allowance in the interface I used. I enter “ducks on a lake” and start generation; my screen shows 82 seconds remaining and a charge of five credits per generated second. The resulting four- to five-second clip looks plausible, though there is little movement and one duck appears to have two heads.

I mention a paid Runway plan at $12 per editor per month and describe higher resolution, watermark removal and shorter waits. Those are details from the 2023 interface, not current prices or entitlements. My favorable comparison with other tools is my judgment from the examples I have seen, not a benchmark.

For another approach, I open the [text-to-video Colab repository](https://github.com/camenduru/text-to-video-synthesis-colab) and choose the Zeroscope v1.1 notebook. Running its first cell installs dependencies and clones the needed repositories. I use the same “ducks on a lake” prompt and leave the negative prompt empty.

The notebook screen shows 33 inference steps, guidance scale 23, 30 frames per second and 24 frames. At that playback rate, 24 frames give about 0.8 seconds. Increasing frames can make the output longer, but in my tests it also consumes more memory and the visual quality can degrade; Colab’s available resources may impose limits.

I run the generation cell and wait while it processes. When it finishes, I open the notebook’s file panel, find the output and download it to my desktop. The resulting very short clip looks comparable to the Runway example to me, though the examples differ in length and do not constitute a controlled test.

For a local installation I use Windows and an Nvidia GPU. I recommend an isolated Conda environment to manage Python and library versions; in my demonstration I create an environment with Python 3.10.11. Compatibility depends on the particular versions of the tools, drivers and hardware, so the 2023 commands should be checked against their project documentation.

After activating the environment, I install PyTorch and related packages, clone the text-to-video fine-tuning code and the model, and install the repository requirements. I then run a checker script to confirm the PyTorch and CUDA versions and whether CUDA is available.

To generate locally, I run `inference.py` with the model path and an output directory, creating that directory if needed. My GPU monitor shows the job running. The generated “ducks on a lake” clip looks good to me but is only about a second long.

I find that quality degrades noticeably as I increase length beyond roughly two seconds. Members of the project’s Discord tell me the model was trained on short clips; I present that as their explanation, not an independently verified cause. They suggest another model, which I have not yet tested.

I generate another example with 48 frames, which I describe as about two seconds, and it still looks reasonably good. In a roughly three-second example, ducks begin appearing and disappearing; longer tests deteriorate further. Frame count, playback rate, model and hardware settings all affect what a particular run can produce.

There are multiple models and notebooks to explore, and the project community can help with setup. Running locally can avoid a hosted per-generation quota, but a suitable GPU, processing time and electricity still cost resources; Colab availability and model licenses have their own limits. These early examples show progress as well as clear length and consistency constraints.