From f0ae2b5da9df445076ab36711034f9eca0f35f7b Mon Sep 17 00:00:00 2001 From: lunarring <78172771+lunarring@users.noreply.github.com> Date: Mon, 9 Jan 2023 11:53:14 +0100 Subject: [PATCH 1/3] Update README.md --- README.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 91f489a..ad43276 100644 --- a/README.md +++ b/README.md @@ -62,24 +62,22 @@ lb.load_branching_profile(quality, depth_strength) ### Autosetup tree setup ```python +depth_strength = 0.5 # see above (Most relevant parameters) num_inference_steps = 30 # the number of diffusion steps -list_nmb_branches = [2, 4, 8, 20] -list_injection_strength = [0.0, 0.3, 0.5, 0.9] +nmb_branches_final = 20 # how many diffusion images will be generated for the transition lb.autosetup_branching(num_inference_steps, list_nmb_branches, list_injection_strength) ``` ### Fully manual ```python -depth_strength = 0.5 # see above (Most relevant parameters) num_inference_steps = 30 # the number of diffusion steps -nmb_branches_final = 20 # how many diffusion images will be generated for the transition +list_nmb_branches = [2, 4, 8, 20] +list_injection_strength = [0.0, 0.3, 0.5, 0.9] -lb.setup_branching(depth_strength, num_inference_steps, nmb_branches_final) +lb.setup_branching(num_inference_steps, list_nmb_branches, list_injection_strength=list_injection_strength) ``` - - # Installation #### Packages ```commandline @@ -108,9 +106,27 @@ pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg= # (this can take dozens of minutes) ``` -# How does it work +# How does latent blending work? ![](animation.gif) +In the figure above, a diffusion tree is illustrated. The diffusion steps are represented on the y-axis, with temporal blending on the x-axis. The diffusion trajectory for the first prompt is the most left column, with the trajectory for the second prompt to the right. At the third iteration, three branches are created, followed by seven at iteration six and the final ten at iteration nine. + +This example can be manually set up using the following code +```python +num_inference_steps = 10 +list_nmb_branches = [2, 3, 7, 10] +list_injection_idx = [0, 3, 6, 9] + +lb.setup_branching(num_inference_steps, list_nmb_branches, list_injection_idx=list_injection_idx) +``` + +Instead of specifying the absolute injection indices using list_injection_idx, we can also pass the list_injection_strength, which are independent of the total number of diffusion iterations (num_inference_steps). +```python +list_injection_strength = [0, 0.3, 0.6, 0.9] +lb.setup_branching(num_inference_steps, list_nmb_branches, list_injection_strength=list_injection_strength) +``` + + what makes a transition a good transition? * absence of movement * every frame looks like a credible photo From b29960c65fba53ec8b822fd25a8fc23814f70c52 Mon Sep 17 00:00:00 2001 From: lunarring <78172771+lunarring@users.noreply.github.com> Date: Mon, 9 Jan 2023 12:09:25 +0100 Subject: [PATCH 2/3] Update README.md --- README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index ad43276..5046d9e 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,7 @@ pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg= ``` # How does latent blending work? +## Technology ![](animation.gif) In the figure above, a diffusion tree is illustrated. The diffusion steps are represented on the y-axis, with temporal blending on the x-axis. The diffusion trajectory for the first prompt is the most left column, with the trajectory for the second prompt to the right. At the third iteration, three branches are created, followed by seven at iteration six and the final ten at iteration nine. @@ -125,8 +126,6 @@ Instead of specifying the absolute injection indices using list_injection_idx, w list_injection_strength = [0, 0.3, 0.6, 0.9] lb.setup_branching(num_inference_steps, list_nmb_branches, list_injection_strength=list_injection_strength) ``` +## Perception +With latent blending, we can create transitions that appear to defy the laws of nature, yet appear completely natural and believable. The key is to surpress processing in our [dorsal visual stream](https://en.wikipedia.org/wiki/Two-streams_hypothesis#Dorsal_stream), which is achieved by avoiding motion in the transition. Without motion, our visual system has difficulties detecting the transition, leaving viewers with the illusion of a single, continuous image. However, when motion is introduced, the visual system can detect the transition and the viewer becomes aware of the transition, leading to a jarring effect. Therefore, best results will be achieved when optimizing the transition parameters, particularly the depth of the first injection. - -what makes a transition a good transition? -* absence of movement -* every frame looks like a credible photo From a9faae1ee320ef5c250e5218980fe2c07ffa162c Mon Sep 17 00:00:00 2001 From: lunarring <78172771+lunarring@users.noreply.github.com> Date: Mon, 9 Jan 2023 12:11:35 +0100 Subject: [PATCH 3/3] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 5046d9e..8d5e56b 100644 --- a/README.md +++ b/README.md @@ -129,3 +129,7 @@ lb.setup_branching(num_inference_steps, list_nmb_branches, list_injection_streng ## Perception With latent blending, we can create transitions that appear to defy the laws of nature, yet appear completely natural and believable. The key is to surpress processing in our [dorsal visual stream](https://en.wikipedia.org/wiki/Two-streams_hypothesis#Dorsal_stream), which is achieved by avoiding motion in the transition. Without motion, our visual system has difficulties detecting the transition, leaving viewers with the illusion of a single, continuous image. However, when motion is introduced, the visual system can detect the transition and the viewer becomes aware of the transition, leading to a jarring effect. Therefore, best results will be achieved when optimizing the transition parameters, particularly the depth of the first injection. +# Future work +* Depth model +* Zooming etc. +