What is latent blending?

Latent blending allows you to generate smooth video transitions between two prompts. It is based on stable diffusion 2.1 and remixes the latent reprensetation using spherical linear interpolations. This results in imperceptible transitions, where one image slowly turns into another one.

Example 1: simple transition

(mp4), code

Example 2: inpainting transition

(mp4), code

Example 3: concatenated transition

(mp4), code

Relevant parameters

Installation

Packages

pip install -r requirements.txt

Download Models from Huggingface

Download the Stable Diffusion v2-1_768 Model

Download the Stable Diffusion 2.0 Inpainting Model (optional)

(Optional) Install Xformers

With xformers, stable diffusion 2 will run a bit faster. The recommended way of installation is via the supplied binaries (Linux).

conda install xformers -c xformers/label/dev

Alternatively, you can build it from source:

# (Optional) Makes the build much faster
pip install ninja
# Set TORCH_CUDA_ARCH_LIST if running and building on different GPU types
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
# (this can take dozens of minutes)

How does it work

what makes a transition a good transition?

• absence of movement
• every frame looks like a credible photo