Post-update Comprehensive Catalogue takes too long

[vix35]

Hi SDLF community, and sorry if there is many mistakes in my english.

I am implementing SDLF with my team, and we are at the point of running the complete process from raw to post-stage, the problem i encountered as i mention on the title of the discussion is that the post update takes too long, and the lambda timeouts before completing all the logging.

here is some images that could help with some more context:

image of the failed step function and parameters:

Image of the size of the bucket where is logging the objects:

Image of the time it takes between each iterarion of the logging and the code that prints, as you can see it takes near to 300ms for each file to be logged in Dynamo, and inside the "folder" of the bucket which is logging there is more than 3600 objects, and there is not even half of the whole data we want to put, so i was wondering what we are doing wrong here.

[cnfait]

Hello @vix35

Instead of updating the DynamoDB table record by record, we can probably batch them:
9a178cd

Can you try and let me know if that helps? I'll run a few tests on my side as well. This may be added to SDLF by default.

[vix35]

Hi @cnfait i took 3 tests:

the post-update as it came originally: 625.679 [ms]
With only accessing once to the metadata in S3: 317.860,48 [ms]
With only accessing once to the metadata in S3 and Dynamo Batch: 297.777

[cnfait]

That's a noticeable improvement, thank you for your tests. Can I also ask which version of Python is used for this lambda in your environment?

[vix35]

Python 3.9

[cnfait]

Python 3.11 should be faster, if you have the opportunity to try it. I don't expect a major speedup as most of the duration is due to network calls but if your use case is still failing this might help as well.

I'm also thinking about other ways to handle pipeline metadata updates (an asynchronous process executed entirely outside pipelines) but this is not even at the draft stage at this point.

Also stageb is usually limited to processing 100 items at a time by default. This is admittedly very low and configurable but I'm interested to know if these thousands of objects have to be processed by the glue job at the same time of if they could be processed in multiple glue job runs?

[vix35]

The stem of the problem probably was partly our fault, we were merging data in delta (delta.io) tables without running OPTIMIZE, and this process makes A LOT of little files (1 MB), we are planning to run optimize now, the way still is not clear yet, but we are thinking of a daily process (glue) which OPTIMIZE every table once per day as recommended by databricks

Still i feel the optimization you helped us to get will be helpful in the future when we have more files even with OPTIMIZE running on a daily basis, but if the data goes up the amount of files will always increase, so its bound to fail sometime in the future.

Another thing we did was modify the heavy-transform layer and the post-update lambda to instead of getting the metadata of the whole post-stage, only get the metadata of the tables which were actually changed in the execution of the stage B

As you can see in this image the "tables" list was wrong, and was picking the name of the files with the data of the tables, so we changed it to take the actual name of the tables that were processed and use it to only update the metadata of the files of those tables.

Result with the modifications