Sync from MySQL to MS SQL normalizations fails #30426
Unanswered
Marcel0643
asked this question in
Connector Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
i have some problems setting up a sync from MySQL to MS SQL.
I left everything in the Sources and Destinations Connect at default.
At the Replication i choose Incremental Append
In the Follow is the error log
`
2023-09-14 10:41:45 destination > Destination process done (exit code 0)
2023-09-14 10:41:45 destination > Skipping in-connector normalization
2023-09-14 10:41:45 INFO i.a.w.g.BufferedReplicationWorker(readFromDestination):464 - readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)
2023-09-14 10:41:45 INFO i.a.w.t.TemporalAttemptExecution(get):124 - Docker volume job log path: /tmp/workspace/15/0/logs.log
2023-09-14 10:41:45 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.29
2023-09-14 10:41:45 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START DEFAULT NORMALIZATION -----
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:41:45 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):138 - Running with normalization version: airbyte/normalization-mssql:0.4.3
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/normalization-mssql:0.4.3 exists...
2023-09-14 10:41:46 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/normalization-mssql:0.4.3 was found locally.
2023-09-14 10:41:46 INFO i.a.w.p.DockerProcessFactory(create):143 - Creating docker container = normalization-mssql-normalize-15-0-carfh with resources io.airbyte.config.ResourceRequirements@607110f6[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-09-14 10:41:46 INFO i.a.w.p.DockerProcessFactory(create):196 - Preparing command: docker run --rm --init -i -w /data/15/0/normalize --log-driver none --name normalization-mssql-normalize-15-0-carfh --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.50.29 airbyte/normalization-mssql:0.4.3 run --integration-type mssql --config destination_config.json --catalog destination_catalog.json
2023-09-14 10:41:47 normalization > Running: transform-config --config destination_config.json --integration-type mssql --out /data/15/0/normalize
2023-09-14 10:41:51 normalization > Namespace(config='destination_config.json', integration_type=<DestinationType.MSSQL: 'mssql'>, out='/data/15/0/normalize')
2023-09-14 10:41:51 normalization > transform_mssql
2023-09-14 10:41:51 normalization > Running: transform-catalog --integration-type mssql --profile-config-dir /data/15/0/normalize --catalog destination_catalog.json --out /data/15/0/normalize/models/generated/ --json-column _airbyte_data
2023-09-14 10:41:51 normalization > Processing destination_catalog.json...
2023-09-14 10:41:51 normalization > Generating airbyte_ctes/dbo/_airbyte_testtabelle_01_ab1.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > Generating airbyte_ctes/dbo/_airbyte_testtabelle_01_ab2.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > Generating airbyte_ctes/dbo/_airbyte_testtabelle_01_ab3.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > Generating airbyte_incremental/dbo/_airbyte_testtabelle_01.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > detected no config file for ssh, assuming ssh is off.
2023-09-14 10:41:57 normalization > [--event-buffer-size EVENT_BUFFER_SIZE]
2023-09-14 10:41:57 normalization > --event-buffer-size EVENT_BUFFER_SIZE
2023-09-14 10:41:57 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:41:57 normalization > DBT >=1.0.0 detected; using 10K event buffer size
2023-09-14 10:41:57 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:00 normalization > Running with dbt=1.0.0
2023-09-14 10:42:00 normalization > Partial parse save file not found. Starting full parse.
2023-09-14 10:42:02 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
2023-09-14 10:42:02 normalization > Found 4 models, 0 tests, 0 snapshots, 0 analyses, 621 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2023-09-14 10:42:02 normalization > Concurrency: 8 threads (target='prod')
2023-09-14 10:42:02 normalization > 1 of 1 START incremental model dbo._airbyte_testtabelle_01.............................................................. [RUN]
2023-09-14 10:42:03 normalization > 10:42:03 + "*****".dbo."_airbyte_testtabelle_01"._airbyte_ab_id does not exist yet. The table will be created or rebuilt with dbt.full_refresh
2023-09-14 10:42:03 normalization > Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:03 normalization > 1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
2023-09-14 10:42:03 normalization > Finished running 1 incremental model in 0.98s.
2023-09-14 10:42:03 normalization > Completed with 1 error and 0 warnings:
2023-09-14 10:42:03 normalization > ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:03 normalization > Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
2023-09-14 10:42:03 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:03 normalization > Diagnosing dbt debug to check if destination is available for dbt and well configured (1):
2023-09-14 10:42:03 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:05 normalization > 10:42:05 Running with dbt=1.0.0
2023-09-14 10:42:05 normalization > dbt version: 1.0.0
2023-09-14 10:42:05 normalization > python version: 3.9.9
2023-09-14 10:42:05 normalization > python path: /usr/local/bin/python
2023-09-14 10:42:05 normalization > os info: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
2023-09-14 10:42:05 normalization > Using profiles.yml file at /data/15/0/normalize/profiles.yml
2023-09-14 10:42:05 normalization > Using dbt_project.yml file at /data/15/0/normalize/dbt_project.yml
2023-09-14 10:42:05 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Configuration:
2023-09-14 10:42:06 normalization > profiles.yml file [OK found and valid]
2023-09-14 10:42:06 normalization > dbt_project.yml file [OK found and valid]
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Required dependencies:
2023-09-14 10:42:06 normalization > - git [OK found]
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Connection:
2023-09-14 10:42:06 normalization > server: *****
2023-09-14 10:42:06 normalization > database: *****
2023-09-14 10:42:06 normalization > schema: dbo
2023-09-14 10:42:06 normalization > port: 1433
2023-09-14 10:42:06 normalization > UID: *****
2023-09-14 10:42:06 normalization > client_id: None
2023-09-14 10:42:06 normalization > authentication: sql
2023-09-14 10:42:06 normalization > encrypt: False
2023-09-14 10:42:06 normalization > trust_cert: False
2023-09-14 10:42:06 normalization > Connection test: [OK connection ok]
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > All checks passed!
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Forward dbt output logs to diagnose/debug errors (0):
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Running with dbt=1.0.0
2023-09-14 10:42:06 normalization > Partial parse save file not found. Starting full parse.
2023-09-14 10:42:06 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
2023-09-14 10:42:06 normalization > Found 4 models, 0 tests, 0 snapshots, 0 analyses, 621 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2023-09-14 10:42:06 normalization > Concurrency: 8 threads (target='prod')
2023-09-14 10:42:06 normalization > 1 of 1 START incremental model dbo._airbyte_testtabelle_01.............................................................. [RUN]
2023-09-14 10:42:06 normalization > 10:42:03 + "*****".dbo."_airbyte_testtabelle_01"._airbyte_ab_id does not exist yet. The table will be created or rebuilt with dbt.full_refresh
2023-09-14 10:42:06 normalization > Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:06 normalization > 1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
2023-09-14 10:42:06 normalization > Finished running 1 incremental model in 0.98s.
2023-09-14 10:42:06 normalization > Completed with 1 error and 0 warnings:
2023-09-14 10:42:06 normalization > ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:06 normalization > Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
2023-09-14 10:42:06 INFO i.a.w.n.DefaultNormalizationRunner(close):207 - Terminating normalization process...
2023-09-14 10:42:06 ERROR i.a.w.g.DefaultNormalizationWorker(run):91 - Normalization failed for job 15.
io.airbyte.workers.exception.WorkerException: Normalization process did not terminate normally (exit code: 1)
at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:218) ~[io.airbyte-airbyte-commons-worker-0.50.29.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:89) ~[io.airbyte-airbyte-commons-worker-0.50.29.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-commons-worker-0.50.29.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.29.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$2(NormalizationActivityImpl.java:175) ~[io.airbyte-airbyte-workers-0.50.29.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.50.29.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:124) ~[io.airbyte-airbyte-workers-0.50.29.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-09-14 10:42:06 INFO i.a.w.g.DefaultNormalizationWorker(run):102 - Normalization executed in 20 seconds for job 15.
2023-09-14 10:42:06 INFO i.a.w.g.DefaultNormalizationWorker(run):114 - Normalization summary: io.airbyte.config.NormalizationSummary@7e70e03e[startTime=1694688105595,endTime=1694688126533,failures=[io.airbyte.config.FailureReason@5cd80151[failureOrigin=normalization,failureType=system_error,internalMessage=('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@52835da9[additionalProperties={attemptNumber=0, jobId=15, from_trace_message=true}],stacktrace=AirbyteDbtError:
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),retryable=,timestamp=1694688126531,additionalProperties={}], io.airbyte.config.FailureReason@31ab1050[failureOrigin=normalization,failureType=system_error,internalMessage=('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@4823674b[additionalProperties={attemptNumber=0, jobId=15, from_trace_message=true}],stacktrace=AirbyteDbtError:
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),retryable=,timestamp=1694688126531,additionalProperties={}]],additionalProperties={}]
2023-09-14 10:42:06 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:42:06 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END DEFAULT NORMALIZATION -----
2023-09-14 10:42:06 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:42:06 INFO i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):56 - Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=0, totalCompleteFailures=0, successivePartialFailures=1, totalPartialFailures=1)
Backoff before next attempt: 0 seconds
`
Beta Was this translation helpful? Give feedback.
All reactions