Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/materialize cratedb #2452

Merged
merged 24 commits into from
Feb 27, 2025
Merged

Feature/materialize cratedb #2452

merged 24 commits into from
Feb 27, 2025

Conversation

surister
Copy link
Contributor

@surister surister commented Feb 26, 2025 •
edited
Loading

Description:

First draft of CrateDB connector, it still has uknown issues, if merged please set it to BETA.

I did commits as atomic as I could.

Pending known issues:

  • Integer/numeric usage might be flaky: I used Numeric(22) across the board to try and get the tests pass), compatibility with other compatible numeric types should be checked.
  • 2025-02-26T10:53:59.523479Z ERROR ops: commit failed: store at index 0: ERROR: Character N is neither a decimal digit number, decimal point, nor "e" notation exponential mark. (SQLSTATE XX000) fields={}
  • Objects/JSON has to be re-checked tested.
  • Temp tables need to be manually deleted after
  • 2025-02-26T11:09:11.095578Z ERROR ops: commit failed: store at index 33: ERROR: Cannot detect the type of the value: 18446744073709551615 (SQLSTATE XX000) fields={}

This change is Reviewable

williamhbaker
williamhbaker requested changes Feb 27, 2025
View reviewed changes
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small changes needed before we can merge this beta version.

@surister
First draft of CrateDB connector
498661a
@surister
Set JSON and MULTIPLE type to OBJECT
36b432b
@surister
Set DATE and TIMESTAMPZ types as TIMESTAMP WITH TIME ZONE
f2012b5
@surister
Set type for sql.ARRAY from JSON to ARRAY(TEXT)
8f53480
@surister
Set default schema as "doc"
cc4b07e
@surister
Remove space from test tables names, e.g. 'Multiple Types' -> 'Multip…
0763f63 
…leTypes'
@surister
Remove 'COMMENT ON' statements from template definition. (Will this c…
8251bc8 
…reate an issue?)
@surister
Early return to avoid db.PingContext killing the connection with 'bad…
0ab1c67 
… connection'
@surister
Set sql.Integer to INTEGER
1c196c9
@surister
Set UUID dtype to TEXT
01e3f9d
@surister
set sql.STRING_INTEGER to DECIMAL
42ac1dc
@surister
set sql.STRING_INTEGER and sql.STRING_NUMBER to NUMERIC(20, 0)
4f5f672
@surister
use jsonConverter for sql.ARRAY
bfb7dff
@surister
set JSON to OBJECT in sql.Dialect MigratableTypes
4b45780
@surister
set time dtype to TEXT
fe285d4
@surister
set uuid dtype as TEXT
b686325
@surister
Remove the TEMPORARY from crateLoadTable definition
de1cc3b
@surister
Remove DO BEGIN and IF NOT FOUND statements from updateFence
f02a3f5
@surister
Directly use b.input for createLoadTable
6b8fe69
@surister
Rename pgDialect to crateDialect
080cc23
@surister
Create temp tables from b.target instead of loadTableColumns
6e06838
@surister
Disable NOT NULL column creation and ALTER COLUMN DROP NOT NULL
1011e32
@surister
Disable CreateSchema
336f7cc
@surister
Add materialize-cratedb to CI build matrix
ea9afaf
@surister surister force-pushed the feature/materialize-cratedb branch from a708694 to ea9afaf Compare February 27, 2025 15:32
williamhbaker
williamhbaker approved these changes Feb 27, 2025
View reviewed changes
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for a beta version

@williamhbaker williamhbaker merged commit 5b7811a into estuary:main Feb 27, 2025
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@williamhbaker williamhbaker williamhbaker approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@surister @williamhbaker