Skip to content

Commit

Permalink
source-hubspot-native: retry failures to fetch batches with associations
Browse files Browse the repository at this point in the history
HubSpot APIs will sometimes return random 500 errors, or timeouts. The connector
often needs to string together thousands of successful API calls to work through
a delayed fetch of a time window, so these random errors causing a connector
restart results in a lot of downtime and wasted work.

This adds some retry logic to retry these requests a few times before crashing
the connector.
  • Loading branch information
williamhbaker committed Jan 9, 2025
1 parent 50f12ec commit c3fdfbb
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions source-hubspot-native/source_hubspot_native/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -487,9 +487,19 @@ async def _do_batch_fetch(batch: list[tuple[datetime, str]]) -> Iterable[tuple[d
# Enable lookup of datetimes for IDs from the result batch.
dts = {id: dt for dt, id in batch}

documents: BatchResult[CRMObject] = await fetch_batch_with_associations(
log, cls, http, with_history, object_name, [id for _, id in batch]
)
attempt = 1
while True:
try:
documents: BatchResult[CRMObject] = await fetch_batch_with_associations(
log, cls, http, with_history, object_name, [id for _, id in batch]
)
break
except Exception as e:
if attempt == 5:
raise
log.warning("failed to fetch batch with associations (will retry)", {"error": str(e), "attempt": attempt})
await asyncio.sleep(attempt * 2)
attempt += 1

return ((dts[str(doc.id)], str(doc.id), doc) for doc in documents.results)

Expand Down

0 comments on commit c3fdfbb

Please sign in to comment.