Initial Load Timeout Issue for Large Tables (>24h) with Retry Configuration Not Working as Expected

[xurtasun]

Hello Airbyte Team,

I’ve been working with Airbyte for several months, synchronizing data from a database with over 2TB of data. Some of the tables are extremely large, which has caused me a few challenges along the way. I recently upgraded to Airbyte 1.0 in hopes of resolving some of these issues, but I am still encountering a major blocker.

The Problem:

• Initial Load Timeout: My sync process for large tables exceeds the 24-hour maximum “initial load timeout.” As far as I can tell, this timeout is not configurable to extend beyond 24 hours (saw your discussion here), which poses a significant limitation for handling large datasets like mine.
• Retry Behavior: I’ve attempted to configure retries for when the sync fails. My expectation was that the retry mechanism would resume the sync process from where it left off. However, when the sync reaches the timeout and fails, the retry starts the process all over again, instead of continuing from the last point of failure. This leads to a loop of incomplete syncs, which is unmanageable for such large tables.

What I’ve Tried:

• Setting up retries after failures, but the sync restarts from the beginning rather than resuming.
• Source db (PostgreSQL)
• Destination (Snowflake)

Questions:

1. Is there any plan or current workaround to allow configuration of the initial load timeout beyond 24 hours for very large datasets?
2. Is there a way to configure the retry mechanism to pick up where the sync left off, instead of restarting from scratch?
3. Are there any other recommended strategies for syncing large tables that take more than 24 hours, considering the current limitations?

Any guidance or feedback would be greatly appreciated. I’m happy to provide more technical details or logs if needed.

Thank you for your help, and I look forward to hearing back from the team!

[Tybot204]

+1 to making this configurable. We have been using Airbyte for over a year and a half now and just recently upgraded to v1.0+ (yes we were behind quite a bit of time). When we upgraded, all of our syncs required a bulk load which we were not expecting. One sync has billions of rows and is now failing, completely blocked by this 24 hour interval. When Airbyte cancels the job, it says it will catch up incremental and then resume bulk load, but this does not appear to work. The following sync errors immediately saying the saved offset is before the replica slot's confirmed LSN.