Increase the frame rate of a video source by a factor of ~2, i.e. interpolate a video sequence of 30 frames/sec to 59 frame/sec.
Use Lucas Kanade and Farneback Optical Flow to generate artificial frames by using the two consecutive real frames from the original video sequence.
Got a free video from pexels.com/videos and down sampled the sequence of 60 frames/sec to 30 frames/sec by taking every other frame
First downsample a video sequence of 60 frames/sec to 30 frames/sec by taking every other frame; then interpolate based on video of 30 frames/sec; finally judge the quality of interpolated video by viewing it with human eyes or comparing the generated frames with the excluded half of the original frames.
The methods from cv2 such as calcOpticalFlowFarneback(), which is the working algorithm for our problem. We also used cv2.cvtColor() to gray out our frames.
Interpolate down sampled video sequence using Farneback Optical Flow. The generated video sequence will have a frame rate about twice that of the original/source video sequence(i.e. original frame rate 30fps---> new frame rate 59fps).
SampleVideo1.mov
Interpolated.mov
InterpolatedSampleVideo1.mov
Comparing to Lucas Kanade optical flow, Farneback optical flow provides better interpolation with less artifact in interpolated frames. There are still some artfacts visible in the area of image with drastic brightness change. We suspect this is due to the assumption of algorithm that the pixel values remain invariant between frames. As for interpolation with Lucas Kanade method, the resulting images exhibit large amount of artifacts. This is potentially due to limitation of Lucas Kanade method used as a dense optical flow even though conceptually it is a sparse motion estimation based only on good features (i.e. corners). Difference of being second order and first order taylor expansion plays an important role in (pixel) motion approximation, where second order expansion yields approximation with less error.
We had to learn many tools that why haven't had much experience using and it took a quite some time to learn how to navigate and implement what we wanted.
Uploading video for testing and implementation to google colab took very long. We had to adapt to use shorter videos with lower resolutions.
If we had more time and resources, we would like to implement our own working Farneback Optical Flow algorithm that is similar to the one the is implemented by OpenCV. We would also like to experiment with different frame rates using a variety of video lengths and resolution.
We learned and implemented the sparse Lucas Kanade in class and wanted to explore the dense Farneback Optical Flow method in our project. It was challenging but it was a great learning experience.