Experimental fork of the Deforum extension for Stable Diffusion WebUI Forge, fix'd up to work with Flux.1 and integrate Parseq keyframe redistribution logic. Integrates dynamic camera shake effects with data sourced from EatTheFutures 'Camera Shakify' Blender plugin.
This fork of the extension is basically working.
Install, update and run the 'one-click installation package' of [Stable Diffusion WebUI Forge](https://github.com/lllyasviel/stable-diffusion-webui-forge as described. Includes:
- Python 3.10.6
- CUDA 12.1
- Pytorch 2.3.1
Other versions may work with this extension, but have not been properly tested.
Get flux1-dev-bnb-nf4-v2.safetensors from huggingface and put it into your <forge_install_dir>/models/Stable-diffusion/Flux:
https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors
Get the following 3 files from huggingface and put them into <forge_install_dir>/models/VAE
ae.safetensorshttps://huggingface.co/black-forest-labs/FLUX.1-schnell/tree/mainclip_l.safetensorshttps://huggingface.co/comfyanonymous/flux_text_encoders/tree/maint5xxl_fp16.safetensorshttps://huggingface.co/comfyanonymous/flux_text_encoders/tree/main
Restart Forge, set mode to "flux", select the flux checkpoint and all the 3 VAEs in "VAE / Text Encoder" and test with Txt2Img.
Go to tab "Extensions" - "Install from URL" and use this: https://github.com/Tok/sd-forge-deforum.git
Open commandline and run <forge_install_dir>/venv/Scripts/activate.bat
to activate the virtual environment (venv) for Python used by Forge.
With the venv from Forge activated, do:
cd <forge_install_dir>/extensions
git clone https://github.com/Tok/sd-forge-deforum
cd sd-forge-deforum
pip install -r requirements.txt
Get the latest default-settings.txt and place it directly into your 'webui' directory, then click "Load All Settings":
https://raw.githubusercontent.com/Tok/sd-forge-deforum/main/scripts/default_settings.txt
Rename it to deforum_settings.txt (or whatever matches the name of your settings file in the UI) and put it directly into your 'webui' directory.
default_settings.txt and may need to be set manually the first time:
- Tab "Prompts" - "Prompts negative" not resetting
- Consider removing the defaults because they're not used with Flux.
Recommendation: Use ForgeUIs "Settings" - "Defaults" to save your settings.
After installation, you can test the setup by generating the default bunny with "Distribution" set to "Keyframes Only" and "Animation Mode" set to "3D". This also downloads the MiDaS model for depth warping when ran for the first time and demonstrates prompt synchronization in a no-cadence setup.
The default bunnies contain 333 frames at 720p, but only 19 of them are actually diffused. The diffused frames are placed in the clip according to the keyframes defined in the prompts. The prompts themselves are aligned to be synchronized at 60 FPS with the beat of an amen break you can find linked in the settings (enable sound):
default_bunny_2.mp4
If you used other versions of the Deforum plugin before, it may also be necessary to update or adjust your Deforum settings. The latest example settings with for the default bunny can also be downloaded here:
https://github.com/Tok/sd-forge-deforum/blob/main/scripts/default_settings.txt
Causes the rendering to run on an experimental core that can rearrange keyframes, which makes it possible to set up fast generations with less jitter at high or no cadence.
- Can now be used with- or without- Parseq.
- Allows for precise sync at high cadence.
- Detailed info and recommendations on new tab.
All subtitles are now generated and written to an .srt file in advance. Complex subtitle generations should work fine with Parseq but are currently limited with Deforum-only setups.
- New Deforum setting for skipping the prompt-part to a new line in .srt files.
- New Deforum setting for choosing simple (non-technical) subtitles that contain only the text from the prompt.
- Complex subtitles should work fine when Parseq is used, but are otherwise limited to essential information only.
- Recommendation: turn on for now if not using Parseq
- Removed emtpy "--neg" param from being written into the subtitles because negative prompts are ignored in Flux workflows.
- Improved padding of technical information so subtitles jitter less.
Add camera shake effects to your renders on top of your other movement.
This feature enhances the realism of your animations by simulating natural camera movements, adding a layer of depth and engagement to your visuals. Perfect for creating action sequences or adding a sense of spontaneity, it allows for customizable shake parameters to fit your specific needs.
The shake data is available under Creative Commons CC0 1.0 Universal license and was sourced from the 'Camera Shakify' Blender plugin by EatTheFuture.
- Flux schnell
- There's not a lot of precision for fine-tuning strength values when only 4 steps are required.
- Control Net
- Hybrid Video
- Non-Flux workflows
- Kohya HR Fix
- may need to be left disabled
- FreeU
- may need to be left disabled
- Control Net
- CLI progress bar may show incorrect values in some setups.
- Check progress on the webUI for better estimations.
- Includes a new default setup to generate default bunny at 60 FPS in 720p with keyframes only.
- Non-essential emojis can be turned off with a checkbox under "Settings" - "Deforum".
- Seed and Subseed tabs unified.
During active development, content and structure of the deforum_settings.txt file
can change quickly been updated. Settings from older versions may not behave as expected.
If necessary, the latest deforum-settings.txt are available for download here:
https://github.com/Tok/sd-forge-deforum/blob/main/scripts/default_settings.txt