Skip to content

Commit

Permalink
source-mixpanel-native: increase funnels date window size
Browse files Browse the repository at this point in the history
This implements the fix made to `annotations` in
8284cbc to the `funnels` stream. The
connector has to send at least one API request for each funnel, and for
users with hundreds of funnels, this will a minimum take multiple hours.
If the connector uses a small date window size & makes more than a
single request per funnel, then it's possible for `funnels` to never
complete in 24 hours.

Since each funnel typically only has KB of data, we can force `funnels`
to make a single request per funnel by using a very large date window size.
  • Loading branch information
Alex-Bair committed Feb 24, 2025
1 parent 77b4cd3 commit ee7c7e1
Showing 1 changed file with 17 additions and 0 deletions.
17 changes: 17 additions & 0 deletions source-mixpanel-native/source_mixpanel_native/streams/funnels.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,23 @@ def state_checkpoint_interval(self) -> int:
# we'll have to emit state every 15 records
return 1000

def __init__(
self,
date_window_size: int = 30,
**kwargs,
):
# This stream checks from the start_date to the end_date for each funnel. Typically,
# there's KB of data between the start_date & end_date for each funnel, and using
# small date windows causes the connector to make multiple requests for each funnel.
# Since we should be able to fetch all data for a funnel in a single request,
# we use an extremely large data window to force the connector to make a single request
# for all of a funnel's records between the start & end dates.

super().__init__(
date_window_size = 100_000,
**kwargs,
)

def path(self, **kwargs) -> str:
return "funnels"

Expand Down

0 comments on commit ee7c7e1

Please sign in to comment.