Persist docs on incremental only if changed #513
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benc-db
requested review from
andrefurlan-db,
susodapop and
rcypher-databricks
as code owners
andrefurlan-db
previously approved these changes
Nov 21, 2023
|
Expanded to be general. Since this macro will always be called after we have created the table, we can always compare with the existing table's columns to see what we need to alter. This allows us to avoid the call on columns that are missing descriptions on a fresh table create, and only alter columns that have changed on later calls to incremental. |
andrefurlan-db
previously approved these changes
Nov 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Doesn't fully resolve persist_docs perf issues, but should help for the case of incremental tables. Resolves #514 for incremental and regular tables.
Description
For incremental tables, only run alter column if the description has changed. This incurs an additional describe extended (because we don't have a cache for this information), but since alter column purportedly takes 6 seconds per command, this should be a net benefit to all but the thinnest of incremental tables.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.