technicalkiwi
/
latentblending
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'main' of github.com:lunarring/latentblending

This commit is contained in:
DGX 2024-01-10 08:47:44 +00:00
parent a0f35f2a41 f5965154ba
commit 1775c9a90a
2 changed files with 22 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
README.md
View File

@ -2,22 +2,32 @@
Latent blending enables video transitions with incredible smoothness between prompts, computed within seconds. Powered by [stable diffusion XL](https://stability.ai/stable-diffusion), this method involves specific mixing of intermediate latent representations to create a seamless transition – with users having the option to fully customize the transition directly in high-resolution. The new version also supports SDXL Turbo, allowing to generate transitions faster than they are typically played back! Latent blending enables video transitions with incredible smoothness between prompts, computed within seconds. Powered by [stable diffusion XL](https://stability.ai/stable-diffusion), this method involves specific mixing of intermediate latent representations to create a seamless transition – with users having the option to fully customize the transition directly in high-resolution. The new version also supports SDXL Turbo, allowing to generate transitions faster than they are typically played back!
```python ```python
from diffusers import AutoPipelineForText2Image
from latentblending.blending_engine import BlendingEngine
from latentblending.diffusers_holder import DiffusersHolder
pipe = AutoPipelineForText2Image.from_pretrained("stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16").to("cuda") pipe = AutoPipelineForText2Image.from_pretrained("stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16").to("cuda")
dh = DiffusersHolder(pipe) dh = DiffusersHolder(pipe)
lb = LatentBlending(dh) be = BlendingEngine(dh)
lb.set_prompt1("photo of underwater landscape, fish, und the sea, incredible detail, high resolution") be.set_prompt1("photo of underwater landscape, fish, und the sea, incredible detail, high resolution")
lb.set_prompt2("rendering of an alien planet, strange plants, strange creatures, surreal") be.set_prompt2("rendering of an alien planet, strange plants, strange creatures, surreal")
lb.set_negative_prompt("blurry, ugly, pale") be.set_negative_prompt("blurry, ugly, pale")
# Run latent blending # Run latent blending
lb.run_transition() be.run_transition()
# Save movie # Save movie
lb.write_movie_transition('movie_example1.mp4', duration_transition=12) be.write_movie_transition('movie_example1.mp4', duration_transition=12)
``` ```
# Installation
```commandline
pip install git+https://github.com/lunarring/latentblending
```
## Gradio UI ## Gradio UI
Coming soon again :) Coming soon again :)
@ -90,12 +100,6 @@ lb.set_parental_crossfeed(crossfeed_power, crossfeed_range, crossfeed_decay)
``` ```
# Installation
#### Packages
```commandline
pip install -r requirements.txt
```
# How does latent blending work? # How does latent blending work?
## Method ## Method
![](animation.gif) ![](animation.gif)

6
latentblending/blending_engine.py
View File

@ -8,6 +8,7 @@ from PIL import Image
from latentblending.movie_util import MovieSaver from latentblending.movie_util import MovieSaver
from typing import List, Optional from typing import List, Optional
import lpips import lpips
import platform
from latentblending.utils import interpolate_spherical, interpolate_linear, add_frames_linear_interp, yml_load, yml_save from latentblending.utils import interpolate_spherical, interpolate_linear, add_frames_linear_interp, yml_load, yml_save
warnings.filterwarnings('ignore') warnings.filterwarnings('ignore')
torch.backends.cudnn.benchmark = False torch.backends.cudnn.benchmark = False
@ -64,7 +65,10 @@ class BlendingEngine():
self.multi_transition_img_first = None self.multi_transition_img_first = None
self.multi_transition_img_last = None self.multi_transition_img_last = None
self.dt_unet_step = 0 self.dt_unet_step = 0
self.lpips = lpips.LPIPS(net='alex').cuda(self.device) if platform.system() == "Darwin":
self.lpips = lpips.LPIPS(net='alex')
else:
self.lpips = lpips.LPIPS(net='alex').cuda(self.device)
self.set_prompt1("") self.set_prompt1("")
self.set_prompt2("") self.set_prompt2("")