Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dekaf: Add in easier-to-run tests #1957

Closed

Conversation

jshearer
Copy link
Contributor

@jshearer jshearer commented Feb 20, 2025

Description:

  • Test a bunch of Dekaf's read machinery without needing a full stack up and running:
    • Connector/discovery logic like which fields are recommended etc
    • Field selection -> Avro schema conversion
    • Avro document encoding
    • Virtual fields like _meta/is_deleted
  • Move validation test infrastructure into shared validation-test crate in order to use it in dekaf Specifically, I needed to be able to convert an input fixture spec into its corresponding built output, which is exactly what validation_test::run() does.

This change is Reviewable

…in order to expose common functionality to be used by other crates

Specifically, I want to be able to convert an input fixture spec into its corresponding built output, which is exactly what `validation_test::run()` does.
@jshearer jshearer marked this pull request as ready for review February 21, 2025 15:53
We weren't correctly narrowing the default of intersected shapes. For example, if we intersected one shape like
```json
{
    "type": ["string", "null"],
    "default": "null"
}
```

with another shape like
```json
{
    "type": "string"
}
```

We'd end up with a shape of
```json
{
    "type": "string",
    "default": null
}
```

Which isn't correct, as when we try to actually emit a document for this shape that's missing the field
in question and provide its default value of null, that will subsequently fail to validate against the schema.

Fixes #1944
@jshearer jshearer closed this Feb 24, 2025
@jshearer jshearer deleted the jshearer/dekaf_testing_and_fix_default_intersection branch February 24, 2025 16:45
@jshearer jshearer restored the jshearer/dekaf_testing_and_fix_default_intersection branch February 24, 2025 16:47
@jshearer
Copy link
Contributor Author

Accidentally deleted this branch

@jshearer jshearer reopened this Feb 24, 2025
@jshearer
Copy link
Contributor Author

Closing in favor of #1965

@jshearer jshearer closed this Feb 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant