source-intercom-native: add config option to use /companies/scroll endpoint
#2256
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…endpoint
Intercom has two separate endpoints for retrieving companies:
`/companies/list` (limited to 10k companies) and `/companies/scroll`
(limited to only one "scroll" happening at a time). For users that have
more than 10k companies in their Intercom account, they will need to use
the `/companies/scroll` endpoint to ensure no data is missed.
The `use_companies_list_endpoint` controls which of the two
company-related endpoints are used. Within the connector, an
`asyncio.Lock` is used to avoid attempting concurrent "scrolls" between
`companies` and `company_segments`.
The connector defaults to using `/companies/scroll`, preferring to
capture all data instead of potentially missing data if the user has >10k companies.
It may be worthwhile to remove the option to use `/companies/list` at a
later date. The main reasons I anticipate users would check the
`user_companies_list_endpoint` option are:
- they have another application using `/companies/scroll` that they
can't disable (not sure how common that is)
- they want the slight speed boost in `companies` and `company_segments`
(which seems relatively negligible with the testing I've done)
Alex-Bair
force-pushed
the
bair/intercom-native-companies-scroll
branch
from
f072c81 to
988e7bc
Compare
Alex-Bair
added a commit
to estuary/flow
that referenced
this pull request
Jan 9, 2025
Updates links & documents the `use_companies_list_endpoint` config option added in estuary/connectors#2256.
Alex-Bair
added a commit
to estuary/flow
that referenced
this pull request
Jan 9, 2025
Updates links & documents the `use_companies_list_endpoint` config option added in estuary/connectors#2256.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Intercom has two separate endpoints for retrieving companies:
/companies/list(limited to 10k companies) and/companies/scroll(limited to only one "scroll" happening at a time). For users that have more than 10k companies in their Intercom account, they will need to use the/companies/scrollendpoint to ensure no data is missed.The connector previously always used the
/companies/listendpoint, and this PR adds the ability to use the/companies/scrollendpoint instead. The newuse_companies_list_endpointconfig option controls which of the two company-related endpoints are used. Within the connector, anasyncio.Lockis used to avoid attempting concurrent "scrolls" between the two streams that want to use/companies/scroll-companiesandcompany_segments.The connector defaults to using
/companies/scroll, preferring to capture all data instead of potentially missing data if the user has >10k companies.It may be worthwhile to remove the option to use
/companies/listat a later date. The main reasons I anticipate users would check theuse_companies_list_endpointoption are:/companies/scrollthat they can't disable (not sure how common that is)companiesandcompany_segments(which seems relatively negligible with the limited testing I've done)Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
source-intercom-nativeconnector docs should be updated to reflect the newuse_companies_list_endpointconfig option.Notes for reviewers:
Tested on a local stack. Confirmed:
_scroll_companiesfunction.companiesandcompany_segmentsstreams do not concurrently try to hit/companies/scroll.use_companies_list_endpointconfig option controls whether/companies/listor/companies/scrollis used./companies/scrollbut receives a400response indicating some other application is using the endpoint.This change is