Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MediaProcessing.asCIImageSequence should produce an array of downsampled frames #223

Open
davidkoski opened this issue Mar 5, 2025 · 1 comment

Comments

@davidkoski
Copy link
Collaborator

MediaProcessing.asCIImageSequence should take a visitor that can apply processing to frames and downsample (requires rendering as CIImage is just a promise!)

Right now we do this:

            var videoFrameResult = await try MediaProcessing.asCIImageSequence(
                video, maxFrames: maxVideoFrames, targetFPS: targetVideoFPS)

            var processedFrames: [MLXArray] = []
            for frame in videoFrameResult.frames {
                let image =
                    frame
                    .toSRGB()
                    .resampled(
                        to: CGSize(width: fixedImageSize, height: fixedImageSize), method: .lanczos
                    )
                    .normalized(mean: config.imageMeanTuple, std: config.imageStdTuple)
                    .asMLXArray()
                processedFrames.append(image)
            }

if the video is 4k, we might have 64 very large frames that we then downsample rather than downsampling as we read.

@davidkoski
Copy link
Collaborator Author

See also #206 and #222

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant