source-zendesk-support-native: add streams to achieve parity with imported connector #2407
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ntermittent snapshot test failures When testing locally, I'd intermittently get snapshot test failures because a start_time param was too recent. It turns out that Zendesk returns that error when the start_time is 60 or fewer seconds from the present. Bumping the delay up to 61 seconds avoids these errors.
The `organizations` stream uses the incremental time based export endpoint, which has the following quirks: - A 10 requests / minute rate limit - The next page is fetched using the updated_at timestamp of the last result in the current response. This means the next page will always contain at least one duplicate record (the last record of the previous page). Because of these quirks: - A super basic rate limiting strategy leveraging asyncio.Lock is used to only make 1 request every 6 seconds to incremental time export endpoints. - An error is raised if we detect 1,000+ organizations updated at the same time. If we didn't the connector would be stuck looping & making the same request endlessly since the max page size is 1,000. - Checkpoints can't be emitted at page breaks since the next page's first result overlaps with the previous page's end, and it's possible to miss records between connector restarts with that strategy.
Some Zendesk accounts don't have access to `audit_logs` or `account_attributes`, so we don't discover them if receive a specific 403 error trying to hit the associated endpoint. There may be other streams that should be conditionally discovered, but they can be handled later when discovering them becomes an issue.
Alex-Bair
changed the title
|
One thing to keep in mind: After this is merged, any captures running with this connector that have auto discovery enabled will automatically add all the new bindings and start capturing data from them. Since the native Zendesk connector is pretty new I don't think there will be too many of these cases, but we might consider turning off auto discover for any previously existing instances of this task before releasing, and then turning it back on after the new bindings have been added in a disabled state, unless we know specifically that it is desired for all the new bindings to be added to the task(s). |
|
That's probably fine @williamhbaker |
Alex-Bair
deleted the
bair/source-zendesk-support-native-additional-streams
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR's scope includes:
source-zendesk-support-nativeon (and above) stream parity with the importedsource-zendesk-support.account_attributesautomationscustom_rolesgroup_membershipsorganizationspost_comment_votespost_commentspost_votespostsschedulessla_policiesticket_formstopicsaccount_attributesandaudit_logsif the provided credentials have access to the appropriate endpoints.See individual commits for more details.
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
Documentation should be updated to reflect the new streams & mark the imported
source-zendesk-supportas deprecated.Notes for reviewers:
Tested on a local stack. Confirmed all streams capture data & complete.
This change is