Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(versioning): Support entity versioning ingestion #12755

Merged
merged 7 commits into from
Mar 4, 2025
Merged

feat(versioning): Support entity versioning ingestion #12755

merged 7 commits into from
Mar 4, 2025

Conversation

asikowitz
Copy link
Collaborator

  • Create LEXICOGRAPHIC_STRING versioning scheme
  • Create versionPropertiesSideEffect that creates / updates version set latest
    • Unlink no longer creates version set

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment smoke_test Contains changes related to smoke tests labels Mar 1, 2025
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Mar 1, 2025
@codecov Codecov
Copy link

codecov bot commented Mar 1, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 84.27673% with 25 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ory/plugins/SpringStandardPluginConfiguration.java 0.00% 13 Missing ⚠️
...oning/sideeffects/VersionPropertiesSideEffect.java 92.85% 4 Missing and 3 partials ⚠️
...ub-web-react/src/app/ingest/secret/SecretsList.tsx 0.00% 1 Missing ⚠️
...om/linkedin/metadata/entity/EntityServiceImpl.java 75.00% 0 Missing and 1 partial ⚠️
...entity/versioning/EntityVersioningServiceImpl.java 85.71% 0 Missing and 1 partial ⚠️
...y/versioning/sideeffects/VersionSetSideEffect.java 94.11% 0 Missing and 1 partial ⚠️
...sioning/validation/VersionPropertiesValidator.java 94.73% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

RyanHolstien
RyanHolstien reviewed Mar 3, 2025
View reviewed changes
...in/java/com/linkedin/metadata/entity/versioning/sideeffects/VersionPropertiesSideEffect.java Outdated
retrieverContext
.getAspectRetriever()
.getLatestSystemAspect(prevLatest, VERSION_PROPERTIES_ASPECT_NAME);
if (prevLatestVersionPropertiesAspect == null) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confused on this, if the previous latest doesn't exist we don't update? Is the idea here that the VersionSetProperties got set to something that hasn't been processed yet and because that comes later than this it is considered more latest even though it could have been deleted or something else?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm your'e right I should update here

RyanHolstien
RyanHolstien reviewed Mar 3, 2025
View reviewed changes
...in/java/com/linkedin/metadata/entity/versioning/sideeffects/VersionPropertiesSideEffect.java Outdated
patchOp.setOp(PatchOperationType.ADD.getValue());
patchOp.setPath("/isLatest");
patchOp.setValue(false);
ChangeMCP updateOldLatestVersionProperties =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the reason you're protecting against empty values and returning Stream.empty, I think it makes more sense to make this part optional to execute, i.e.

Suggested change
ChangeMCP updateOldLatestVersionProperties =
if (prevLatestVersionPropertiesAspect != null) {
prevLatestVersionProperties =
RecordUtils.toRecordTemplate(
VersionProperties.class,
prevLatestVersionPropertiesAspect.getRecordTemplate().data());
if (prevLatestVersionProperties != null) {
ChangeMCP updateOldLatestVersionProperties =

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No just an oversight lol

@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Mar 3, 2025
@datahub-cyborg datahub-cyborg bot added pending-submitter-merge and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Mar 3, 2025
@asikowitz asikowitz merged commit 5576b3c into datahub-project:master Mar 4, 2025
87 checks passed
@asikowitz asikowitz deleted the support-versioning-ingestion branch March 4, 2025 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RyanHolstien RyanHolstien RyanHolstien approved these changes

Assignees
No one assigned
Labels
devops PR or Issue related to DataHub backend & deployment pending-submitter-merge product PR or Issue related to the DataHub UI/UX smoke_test Contains changes related to smoke tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@asikowitz @RyanHolstien