Download Images asynchronously in Image Processing

[nrjadkry]

Enhance Asynchronous S3 Image Downloading

• Refactor DroneImageProcessor to support efficient async image downloads
• Implement batched downloading with asyncio and aiohttp
• Add concurrent download capabilities with configurable batch size
• Improve error handling and logging during image retrieval

Changes include:

• Updated download_images_from_s3 method to support async downloads
• Implemented batch processing of image downloads
• Added error handling and logging
• Optimized pre-signed URL generation and download process

@spwoodcock
One question I would like to ask you is if we can support more than 10 parallel uploads. There is an option for parallel uploads in ODM. By default, it is 10 parallel downloads.

[spwoodcock]

Nice work!

See comment.

This isn't a perfect solution due to the difficulty of requiring pre-signed URLs.

Is having a private S3 bucket an integral part of the DroneTM platform? It adds significant complications (we can't have NodeODM simply download the files itself - best case we have to relay via our API for presigned URLs)

[spwoodcock]

This all looks great, except this is synchronous and may cause a bottleneck.

How did this perform during testing?

Any speed or memory issues as previously?

[nrjadkry]

Downloading looks good with this implementation.
Also, the images are public in S3. So, instead of using presigned url, we can directly use download urls.
I will update this.

[spwoodcock]

Nice! That makes things much easier and reduces the number of requests by half 😁

You can probably list all the image file names with one S3 call, then pass through the list of constructed URLs to the async downloader

[spwoodcock]

Do we need presigned URLs for the downloads @nrjadkry ?

It will significantly slow down the download process.

Can't we just construct the URL ourselves and download directly from the public bucket?

The purpose of pre signing is:

• Private buckets
• Access from a frontend that has no credentials for the private bucket

[nrjadkry]

I have kept the presigned urls in case the bucket is private.
I have used both the cases here assuming users do not have download url in the environment if the bucket is private.
Here, in our case, since the bucket is public, it wont generate the presigned urls and instead construct the urls directly.

        s3_download_url = settings.S3_DOWNLOAD_ROOT
        if s3_download_url:
            object_urls = [f"{s3_download_url}/{obj.object_name}" for obj in objects]
        else:
            # generate pre-signed URL for each object
            object_urls = [
                get_presigned_url(bucket_name, obj.object_name, 12) for obj in objects
            ]

[spwoodcock]

I have kept the presigned urls in case the bucket is private.
I have used both the cases here assuming users do not have download url in the environment if the bucket is private.
Here, in our case, since the bucket is public, it wont generate the presigned urls and instead construct the urls directly.

        s3_download_url = settings.S3_DOWNLOAD_ROOT
        if s3_download_url:
            object_urls = [f"{s3_download_url}/{obj.object_name}" for obj in objects]
        else:
            # generate pre-signed URL for each object
            object_urls = [
                get_presigned_url(bucket_name, obj.object_name, 12) for obj in objects
            ]

Oooh thanks for that snippet - missed it reviewing on my phone!

Makes sense, thanks 😁

[nrjadkry]

@spwoodcock
This is working nicely. Thank you.
Also, can we do anything on uploads to ODM ? Have you got any idea ?

[spwoodcock]

@spwoodcock
This is working nicely. Thank you.
Also, can we do anything on uploads to ODM ? Have you got any idea ?

So sorry @nrjadkry, its going to be a good few days until I can look at this. It needs some dedicated time to work out. I believe I had some ideas / possible solutions in threads here somewhere though.

OAM had a second issue today & I'm so far behind with FMTM 😅

Ping me if I forget please! 🙏