2023-01-09 08:08:51 +00:00
Latent blending enables you to generate super smooth video transitions between prompts. It is based on [stable diffusion 2.1 ](https://stability.ai/blog/stablediffusion2-1-release7-dec-2022 ). The key idea is to take the intermediate latent representations and mix them together before further developing them. You can fully specify how this happens or use presets.
2022-11-21 23:20:07 +00:00
2023-01-09 08:13:33 +00:00
Latent blending is an advanced technique that enables users to generate super-smooth transitions between different video prompts. Powered by [stable diffusion 2.1 ](https://stability.ai/blog/stablediffusion2-1-release7-dec-2022 ), this method involves specific mixing of intermediate latent representations to create a seamless transition –
2023-01-09 08:06:01 +00:00
# Quickstart
```python
fp_ckpt = 'path_to_SD2.ckpt'
fp_config = 'path_to_config.yaml'
2023-01-09 08:08:51 +00:00
2023-01-09 08:09:11 +00:00
sdh = StableDiffusionHolder(fp_ckpt, fp_config, 'cuda')
2023-01-09 08:06:01 +00:00
lb = LatentBlending(sdh)
2023-01-09 08:08:51 +00:00
2023-01-09 08:06:01 +00:00
lb.load_branching_profile(quality='medium', depth_strength=0.4)
lb.set_prompt1('photo of my first prompt1')
lb.set_prompt2('photo of my second prompt')
2023-01-09 08:08:51 +00:00
2023-01-09 08:06:01 +00:00
imgs_transition = lb.run_transition()
```
2023-01-09 08:08:51 +00:00
## Gradio UI
To run the UI on your local machine, run `gradio_ui.py`
2023-01-09 08:06:01 +00:00
## Example 1: Simple transition
2023-01-09 07:58:03 +00:00
2023-01-09 08:00:24 +00:00
To run a simple transition between two prompts, run `example1_standard.py`
2022-11-21 23:20:07 +00:00
2023-01-09 08:06:01 +00:00
## Example 2: Inpainting transition
2023-01-09 08:00:24 +00:00
To run a transition between two prompts where you want some part of the image to remain static, run `example2_inpaint.py`
2022-11-21 23:20:07 +00:00
2023-01-09 08:06:01 +00:00
## Example 3: concatenated transition
2023-01-09 08:00:24 +00:00
To run multiple transition between K prompts, resulting in a stitched video, run `example3_multitrans.py`
2022-11-21 23:20:07 +00:00
2022-12-02 11:42:09 +00:00
# Relevant parameters
2022-11-21 23:20:07 +00:00
2022-12-02 11:42:09 +00:00
# Installation
#### Packages
```commandline
2022-12-02 12:08:17 +00:00
pip install -r requirements.txt
2022-12-02 11:42:09 +00:00
```
2022-12-02 12:08:17 +00:00
#### Download Models from Huggingface
2022-12-09 11:52:50 +00:00
[Download the Stable Diffusion v2-1_768 Model ](https://huggingface.co/stabilityai/stable-diffusion-2-1 )
2022-12-02 11:42:09 +00:00
[Download the Stable Diffusion 2.0 Inpainting Model (optional) ](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting )
2022-12-09 11:52:50 +00:00
#### (Optional) Install [Xformers](https://github.com/facebookresearch/xformers)
2022-12-02 12:43:35 +00:00
With xformers, stable diffusion 2 will run a bit faster. The recommended way of installation is via the supplied binaries (Linux).
2022-12-02 11:42:09 +00:00
```commandline
2022-12-02 12:08:17 +00:00
conda install xformers -c xformers/label/dev
2022-12-02 11:42:09 +00:00
```
2022-12-02 12:08:17 +00:00
Alternatively, you can build it from source:
2022-12-02 11:42:09 +00:00
```commandline
2022-12-02 12:08:17 +00:00
# (Optional) Makes the build much faster
pip install ninja
# Set TORCH_CUDA_ARCH_LIST if running and building on different GPU types
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
# (this can take dozens of minutes)
2022-12-02 11:42:09 +00:00
```
# How does it work
2022-11-21 23:20:07 +00:00
2022-12-02 11:42:09 +00:00
what makes a transition a good transition?
* absence of movement
* every frame looks like a credible photo
