2.0 KiB
What is latent blending?
Latent blending allows you to generate smooth video transitions between two prompts. It is based on stable diffusion 2.0 and remixes the latent reprensetation using spherical linear interpolations. This results in imperceptible transitions, where one image slowly turns into another one.
Example 1: simple transition
(mp4), code
Example 2: inpainting transition
(mp4), code
Example 3: concatenated transition
(mp4), code
Relevant parameters
Installation
Packages
pip install -r requirements.txt
Models
Download the Stable Diffusion 2.0 Standard Model
Download the Stable Diffusion 2.0 Inpainting Model (optional)
xformers efficient attention (copied from stability)
For more efficiency and speed on GPUs, we highly recommended installing the xformers library.
Tested on A100 with CUDA 11.4. Installation needs a somewhat recent version of nvcc and gcc/g++, obtain those, e.g., via
export CUDA_HOME=/usr/local/cuda-11.4
conda install -c nvidia/label/cuda-11.4.0 cuda-nvcc
conda install -c conda-forge gcc
conda install -c conda-forge gxx_linux-64=9.5.0
Then, run the following (compiling takes up to 30 min).
cd ..
git clone https://github.com/facebookresearch/xformers.git
cd xformers
git submodule update --init --recursive
pip install -r requirements.txt
pip install -e .
cd ../stable-diffusion
Upon successful installation, the code will automatically default to memory efficient attention for the self- and cross-attention layers in the U-Net and autoencoder.
How does it work
what makes a transition a good transition?
- absence of movement
- every frame looks like a credible photo