Skip to content

Latest commit

 

History

History
108 lines (68 loc) · 4.79 KB

README.md

File metadata and controls

108 lines (68 loc) · 4.79 KB

Add Temporal Coherence to Single Image AI Upscaling Models in Vapoursynth

Also known as temporal consistency, line wiggle fix, stabilization, temporal denoising, or temporal fix.
This runs on the CPU in parallel to the upscaling on the GPU. Intended for animation.

Check out hddvddegogo's comparisons here and here.


Requirements

Setup

Put the vs_temporalfix.py file into your vapoursynth scripts folder.
Or install via pip: pip install git+https://github.com/pifroggi/vs_temporalfix.git


Usage

from vs_temporalfix import vs_temporalfix
clip = vs_temporalfix(clip, strength=400, tr=6, denoise=False, exclude=None, debug=False)

clip
Temporally unstable clip. Should have no black borders.

strength
Suppression strength of temporal inconsistencies. Higher means more aggressive. 400 works great in most cases.
The best way to finetune is to find a static scene and adjust till lines and details are stable.
Reduce if you get blending/ghosting on small movements, especially in dark or hazy scenes.

tr
Temporal radius sets the number of frames to average over.
Higher means more stable, especially on slow pans and zooms, but is slower. 6 works great in most cases.
The best way to finetune is to find a slow pan or zoom and adjust till lines and details are stable.

denoise (optional)
By default only temporal inconsistencies in the frequency range produced by super resolution models are fixed. Setting denoise to True will fix very high and low frequency noise in addition with only a slight slowdown. Only use this if there is actually noise/grain, or low frequency flicker! This risks to remove some details like every denoiser, but is useful if you're planning to denoise anyway.

exclude (optional)
Optionally exclude scenes with intended temporal inconsistencies, or in case this doesn't work.
Example setting 3 scenes: exclude="[10 20] [600 900] [2000 2500]"
First number in the brackets is the first frame of the scene, the second number is the last frame (inclusive).

debug (optional)
Shows areas that will be left untouched in pink. This includes areas with high motion, scene changes and previously excluded scenes. May help while tuning parameters to see if the area is even affected.

Caution

  • If fps are much lower than the benchmarks, try adding core.max_cache_size = 15000 (15GB) near the top of your vapoursynth script to allow higher RAM usage. Very high tr or resolution may need more.

Tip

  • There is a big drop in performance for tr > 6, due to switching from mvtools to mvtools-sf, which is slower.
  • mvtools-sf release r9 and the r10 pre-release are both supported, but r9 is faster for me.

Benchmarks

Hardware Resolution TR Average FPS
Ryzen 5900X 1440x1080 6 ~8 fps
Ryzen 5900X 2880x2160 6 ~5 fps

Alternative Usage Options

Several projects integrated this script to simplify usage without the need for Vapoursynth knowledge.

  • py_temporalfix (Windows only)
    Simple portable command line tool to just do the temporal fix.
  • Hybrid (Windows only)
    Video filter toolbox with a GUI. Can be a bit overwhelming due to the amount of features, but can upscale and do the temporal fix at the same time, as well as many more filters.
  • VSGAN-tensorrt-docker (Windows and Linux)
    Command line AI upscale and interpolation toolbox. Rudimentary knowledge of Docker and Vapoursynth is recommended, but the readme also explains it. Can upscale and do the temporal fix at the same time.