From 13d800634751d5e525ed8a478bfa303c20490e52 Mon Sep 17 00:00:00 2001 From: Johannes Stelzer Date: Thu, 16 Nov 2023 15:47:10 +0100 Subject: [PATCH] upd --- README.md | 59 +++++++++++++++++++------------------------------------ 1 file changed, 20 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 108e554..b239e08 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,25 @@ # Quickstart -Latent blending enables video transitions with incredible smoothness between prompts, computed within seconds. Powered by [stable diffusion 2.1](https://stability.ai/blog/stablediffusion2-1-release7-dec-2022), this method involves specific mixing of intermediate latent representations to create a seamless transition – with users having the option to fully customize the transition and run high-resolution upscaling. +Latent blending enables video transitions with incredible smoothness between prompts, computed within seconds. Powered by [stable diffusion XL](https://stability.ai/stable-diffusion), this method involves specific mixing of intermediate latent representations to create a seamless transition – with users having the option to fully customize the transition directly in high-resolution. -[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1I77--5PS6C-sAskl9OggS1zR0HLKdq1M?usp=sharing) ```python -fp_ckpt = hf_hub_download(repo_id="stabilityai/stable-diffusion-2-1-base", filename="v2-1_512-ema-pruned.ckpt") - -sdh = StableDiffusionHolder(fp_ckpt) -lb = LatentBlending(sdh) +pretrained_model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0" +pipe = DiffusionPipeline.from_pretrained(pretrained_model_name_or_path, torch_dtype=torch.float16).to('cuda') +dh = DiffusersHolder(pipe) +lb = LatentBlending(dh) lb.set_prompt1('photo of my first prompt1') lb.set_prompt2('photo of my second prompt') depth_strength = 0.6 # How deep the first branching happens t_compute_max_allowed = 10 # How much compute time we give to the transition -imgs_transition = lb.run_transition(depth_strength=depth_strength, t_compute_max_allowed=t_compute_max_allowed) +imgs_transition = lb.run_transition( + depth_strength=depth_strength, + num_inference_steps=num_inference_steps, + t_compute_max_allowed=t_compute_max_allowed) ``` ## Gradio UI -To run the UI on your local machine, run `gradio_ui.py` -If you want to specify the output directory, you can create a `.env` file in the latentblending git directory. -In here, specify: -``` -DIR_OUT="SET_PATH_HERE" -``` +Coming soon again :) ## Example 1: Simple transition ![](example1.jpg) @@ -32,12 +29,6 @@ To run a simple transition between two prompts, run `example1_standard.py` To run multiple transition between K prompts, resulting in a stitched video, run `example2_multitrans.py`. [View a longer example video here.](https://vimeo.com/789052336/80dcb545b2) -## Example 3: High-resolution with upscaling -![](example3.jpg) -You can run a high-res transition using the x4 upscaling model in a two-stage procedure, see `example3_upscaling.py`. [View as video here.](https://vimeo.com/787639426/f88dae2ea6) - -## Example 4: Multi transition with high-resolution with upscaling -You can run a multi transition movie and upscale it, see `example4_multitrans_upscaling.py`. # Customization @@ -46,8 +37,8 @@ You can find the [most relevant parameters here.](parameters.md) ### Change the height/width ```python -lb.set_height(512) -lb.set_width(1024) +size_output = (1024, 768) +lb.set_dimensions(size_output) ``` ### Change guidance scale ```python @@ -87,22 +78,6 @@ lb.set_parental_crossfeed(crossfeed_power, crossfeed_range, crossfeed_decay) pip install -r requirements.txt ``` -#### (Optional but recommended) Install [Xformers](https://github.com/facebookresearch/xformers) -With xformers, stable diffusion will run faster with smaller memory inprint. Necessary for higher resolutions / upscaling model. - -```commandline -conda install xformers -c xformers/label/dev -``` - -Alternatively, you can build it from source: -```commandline -# (Optional) Makes the build much faster -pip install ninja -# Set TORCH_CUDA_ARCH_LIST if running and building on different GPU types -pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers -# (this can take dozens of minutes) -``` - # How does latent blending work? ## Method ![](animation.gif) @@ -120,6 +95,8 @@ imgs_transition = lb.run_transition(num_inference_steps=10, depth_strength=0.2, With latent blending, we can create transitions that appear to defy the laws of nature, yet appear completely natural and believable. The key is to surpress processing in our [dorsal visual stream](https://en.wikipedia.org/wiki/Two-streams_hypothesis#Dorsal_stream), which is achieved by avoiding motion in the transition. Without motion, our visual system has difficulties detecting the transition, leaving viewers with the illusion of a single, continuous image, see [change blindness](https://en.wikipedia.org/wiki/Change_blindness). However, when motion is introduced, the visual system can detect the transition and the viewer becomes aware of the transition, leading to a jarring effect. Therefore, best results will be achieved when optimizing the transition parameters, particularly the crossfeeding parameters and the depth of the first injection. # Changelog +* SD XL support +* Diffusers backend, greatly simplifing installation and use (bring your own pipe) * New blending engine with cross-feeding capabilities, enabling structure preserving transitions * LPIPS image similarity for finding the next best injection branch, resulting in smoother transitions * Time-based computation: instead of specifying how many frames your transition has, you can tell your compute budget and get a transition within that budget. @@ -128,9 +105,13 @@ With latent blending, we can create transitions that appear to defy the laws of * Inpaint support dropped (as it only makes sense for a single transition) # Coming soon... +- [ ] Gradio interface - [ ] Huggingface Space -- [ ] More manipulations to the latent (translation, zoom, masking) -- [ ] Transitions with Depth model +- [ ] Controlnet +- [ ] IP-Adapter +- [ ] Latent Consistency + + Stay tuned on twitter: ```@j_stelzer```