[WIP] short blogpost on tooling around processing video datasets #2631
Conversation
| @@ -0,0 +1,149 @@ | |||
| --- | |||
| title: "Build awesome datasets for video generation" | |||
| thumbnail: /blog/assets/vid_ds_scripts/thumbnail.png | |||
There was a problem hiding this comment.
We also need the changes to _blog.yml etc.
Co-authored-by: hlky <hlky@hlky.ac>
|
@hlky I updated some content now that we support NSFW filtering. |
|
@pcuenca, a friendly ping. |
|
Sure, I can take a look tomorrow if that works. |
|
@pcuenca that works. Thanks! |
| @@ -0,0 +1,149 @@ | |||
| --- | |||
| title: "Build awesome datasets for video generation" | |||
| thumbnail: /blog/assets/vid_ds_scripts/thumbnail.png | |||
There was a problem hiding this comment.
We also need the changes to _blog.yml etc.
|
|
||
| # Build awesome datasets for video generation | ||
|
|
||
| Tooling for image generation datasets is well established, with [`img2dataset`](https://github.com/rom1504/img2dataset) covering large scale and various community guides, scripts and UIs covering the small scale. |
There was a problem hiding this comment.
| Tooling for image generation datasets is well established, with [`img2dataset`](https://github.com/rom1504/img2dataset) covering large scale and various community guides, scripts and UIs covering the small scale. | |
| Tooling for image generation datasets is well established, with [`img2dataset`](https://github.com/rom1504/img2dataset) being a fundamental tool used for large scale dataset preparation, and complemented with various community guides, scripts and UIs that cover smaller scale initiatives. |
|
|
||
| Tooling for image generation datasets is well established, with [`img2dataset`](https://github.com/rom1504/img2dataset) covering large scale and various community guides, scripts and UIs covering the small scale. | ||
|
|
||
| Our goal is to make tooling for video generation datasets as established by creating open video dataset scripts suited for small scale, with [`video2dataset`](https://github.com/iejMac/video2dataset) covering large scale. |
There was a problem hiding this comment.
| Our goal is to make tooling for video generation datasets as established by creating open video dataset scripts suited for small scale, with [`video2dataset`](https://github.com/iejMac/video2dataset) covering large scale. | |
| Our ambition is to make tooling for video generation datasets equally established, by creating open video dataset scripts suited for small scale, and leveraging [`video2dataset`](https://github.com/iejMac/video2dataset) for large scale use cases. |
I'm not sure if this is what you want to say, or rather than we'll focus on the small scale because the large scale is "solved".
|
|
||
| *“If I have seen further it is by standing on the shoulders of giants”* | ||
|
|
||
| In this post, we will provide a overview of the tooling we are developing to make it easy for the community to build their own datasets for fine-tuning video generation models. If you cannot wait to get started already, we welcome you to check out the codebase [here](https://github.com/huggingface/video-dataset-scripts/tree/main/video_processing). |
There was a problem hiding this comment.
| In this post, we will provide a overview of the tooling we are developing to make it easy for the community to build their own datasets for fine-tuning video generation models. If you cannot wait to get started already, we welcome you to check out the codebase [here](https://github.com/huggingface/video-dataset-scripts/tree/main/video_processing). | |
| In this post, we provide an overview of the tooling we are developing to make it easy for the community to build their own datasets for fine-tuning video generation models. If you cannot wait to get started already, we welcome you to check out the codebase [here](https://github.com/huggingface/video-dataset-scripts/tree/main/video_processing). |
|
|
||
| #### Entire video | ||
|
|
||
| - predict a motion score with [OpenCV](https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html) |
There was a problem hiding this comment.
I think we should state what it is and what it will be used for. The link to optical flow methods might be a bit confusing in my opinion.
| Florence-2 [`microsoft/Florence-2-large`](http://hf.co/microsoft/Florence-2-large) to run `<CAPTION>`, `<DETAILED_CAPTION>`, `<DENSE_REGION_CAPTION>` and `<OCR_WITH_REGION>`. | ||
|
|
||
| We can bring in any other captioner in this regard. We can also caption the entire video (e.g., with a model like [Qwen2.5](https://huggingface.co/docs/transformers/main/en/model_doc/qwen2_5_vl)) as opposed to captioning individual frames. |
There was a problem hiding this comment.
Are we captioning individual frames with Florence 2? What do those tags <DENSE_REGION_CAPTION> etc mean?
In general, I think we should previously introduce the task earlier in the post: we are going to grab a lot of videos from the Internet, and we are going to caption them so that the models to be trained have enough information to match text descriptions with frames/sequences of frames/full videos. And of course we need to filter bad quality content, videos that are too short or too long, etc.
|
|
||
| ## Filtering examples | ||
|
|
||
| In the [dataset](https://huggingface.co/datasets/finetrainers/crush-smol) for the model [`finetrainers/crush-smol-v0`](https://hf.co/finetrainers/crush-smol-v0) we filtered on `pwatermark < 0.1` and `aesthetic > 5.5`. This highly restrictive filtering resulted in 47 videos out of 1493 total. |
There was a problem hiding this comment.
This one appears to have been captioned with Qwen actually, but we said we'd use Florence.
I'd explain that this is an example dataset we obtained using this process.
| Let's review the example frames from `pwatermark` - two with text have scores of 0.69 and 0.61, the "toy car with a bunch of mice in it" scores 0.60 then 0.17 as the toy car is crushed. All example frames were filtered by `pwatermark < 0.1`. `pwatermark` is effective at detecting text/watermarks however the score gives no indication whether it is a text overlay or a toy car's license plate. Our filtering required all scores to be below the threshold, an average across frames would be a more effective strategy for `pwatermark` with a threshold of around 0.2 - 0.3. | ||
|
|
||
| Let's review the example frames from aesthetic scores - the pink castle initially scores 5.5 then 4.44 as it is crushed, the action figure scores lower at 4.99 dropping to 4.84 as it is crushed and the shard of glass scores low at 4.04. Again fitlering required all scores to be below the threshold, in this case using the aesthetic score from the first frame only would be a more effective strategy. |
There was a problem hiding this comment.
I'm a bit lost here. Screenshots would go a long way imo.
|
|
||
| We provide some visual examples for each filters as well as the captions from Florence-2. | ||
|
|
||
| ### Watermarks |
There was a problem hiding this comment.
Ah, the example images are here! I'd put them above and maybe interwoven with the text explanation.
| <figcaption>Prompt: <i>DIFF_crush A red candle is placed on a metal platform, and a large metal cylinder descends from above, flattening the candle as if it were under a hydraulic press. The candle is crushed into a flat, round shape, leaving a pile of debris around it.</i></figcaption> | ||
| </figure> | ||
|
|
||
| ## Call to action |
There was a problem hiding this comment.
I wouldn't use "call to action", as I think it's a marketing term. Maybe "Your Turn", or "Follow Along", or "Conclusion" could work.
With @hlky.
TODO