Skip to content

Commit

Permalink
support snowflake input option (#38)
Browse files Browse the repository at this point in the history
* support snowflake input option

* support output snowflake

* go generate

* fix some bug

* go generate

* run linter

* rename import name

* capitalized and snowflake input option Set schema defaults.

* change generic type
  • Loading branch information
katamotokosuke authored Feb 4, 2025
1 parent 71d90a3 commit 38f35e9
Show file tree
Hide file tree
Showing 26 changed files with 975 additions and 54 deletions.
108 changes: 108 additions & 0 deletions docs/resources/job_definition.md
Original file line number Diff line number Diff line change
Expand Up @@ -1020,6 +1020,7 @@ Optional:

- `gcs_input_option` (Attributes) Attributes about source GCS (see [below for nested schema](#nestedatt--input_option--gcs_input_option))
- `mysql_input_option` (Attributes) Attributes of source mysql (see [below for nested schema](#nestedatt--input_option--mysql_input_option))
- `snowflake_input_option` (Attributes) Attributes about source snowflake (see [below for nested schema](#nestedatt--input_option--snowflake_input_option))

<a id="nestedatt--input_option--gcs_input_option"></a>
### Nested Schema for `input_option.gcs_input_option`
Expand Down Expand Up @@ -1321,13 +1322,61 @@ Optional:



<a id="nestedatt--input_option--snowflake_input_option"></a>
### Nested Schema for `input_option.snowflake_input_option`

Required:

- `database` (String) Database name
- `input_option_columns` (Attributes List) List of columns to be retrieved and their types (see [below for nested schema](#nestedatt--input_option--snowflake_input_option--input_option_columns))
- `query` (String) Query
- `snowflake_connection_id` (Number) Id of Snowflake connection
- `warehouse` (String) Warehouse name

Optional:

- `connect_timeout` (Number) Connection timeout (sec)
- `custom_variable_settings` (Attributes List) (see [below for nested schema](#nestedatt--input_option--snowflake_input_option--custom_variable_settings))
- `fetch_rows` (Number) Number of records processed by the cursor at one time
- `schema` (String) Schema name
- `socket_timeout` (Number) Socket timeout (seconds)

<a id="nestedatt--input_option--snowflake_input_option--input_option_columns"></a>
### Nested Schema for `input_option.snowflake_input_option.input_option_columns`

Required:

- `name` (String) Column name
- `type` (String) Column type


<a id="nestedatt--input_option--snowflake_input_option--custom_variable_settings"></a>
### Nested Schema for `input_option.snowflake_input_option.custom_variable_settings`

Required:

- `name` (String) Custom variable name. It must start and end with `$`
- `type` (String) Custom variable type. The following types are supported: `string`, `timestamp`, `timestamp_runtime`

Optional:

- `direction` (String) Direction of the diff from context_time. The following directions are supported: `ago`, `later`. Required in `timestamp` and `timestamp_runtime` types
- `format` (String) Format used to replace variables. Required in `timestamp` and `timestamp_runtime` types
- `quantity` (Number) Quantity used to calculate diff from context_time. Required in `timestamp` and `timestamp_runtime` types
- `time_zone` (String) Time zone used to format the timestamp. Required in `timestamp` and `timestamp_runtime` types
- `unit` (String) Time unit used to calculate diff from context_time. The following units are supported: `hour`, `date`, `month`. Required in `timestamp` and `timestamp_runtime` types
- `value` (String) Fixed string which will replace variables at runtime. Required in `string` type




<a id="nestedatt--output_option"></a>
### Nested Schema for `output_option`

Optional:

- `bigquery_output_option` (Attributes) Attributes of destination BigQuery settings (see [below for nested schema](#nestedatt--output_option--bigquery_output_option))
- `snowflake_output_option` (Attributes) Attributes of destination Snowflake settings (see [below for nested schema](#nestedatt--output_option--snowflake_output_option))

<a id="nestedatt--output_option--bigquery_output_option"></a>
### Nested Schema for `output_option.bigquery_output_option`
Expand Down Expand Up @@ -1394,6 +1443,65 @@ Optional:



<a id="nestedatt--output_option--snowflake_output_option"></a>
### Nested Schema for `output_option.snowflake_output_option`

Required:

- `database` (String) Database name
- `schema` (String) Schema name
- `snowflake_connection_id` (Number) Snowflake connection ID
- `table` (String) Table name
- `warehouse` (String) Warehouse name

Optional:

- `batch_size` (Number) Batch size (MB)
- `custom_variable_settings` (Attributes List) (see [below for nested schema](#nestedatt--output_option--snowflake_output_option--custom_variable_settings))
- `default_time_zone` (String) Default time zone
- `delete_stage_on_error` (Boolean) Delete temporary stage on error
- `empty_field_as_null` (Boolean) Replace empty string with NULL
- `max_retry_wait` (Number) Maximum retry wait time (milliseconds)
- `mode` (String) Transfer mode
- `retry_limit` (Number) Maximum retry limit
- `retry_wait` (Number) Retry wait time (milliseconds)
- `snowflake_output_option_column_options` (Attributes List) (see [below for nested schema](#nestedatt--output_option--snowflake_output_option--snowflake_output_option_column_options))
- `snowflake_output_option_merge_keys` (List of String) Merge keys (only applicable if mode is 'merge')

<a id="nestedatt--output_option--snowflake_output_option--custom_variable_settings"></a>
### Nested Schema for `output_option.snowflake_output_option.custom_variable_settings`

Required:

- `name` (String) Custom variable name. It must start and end with `$`
- `type` (String) Custom variable type. The following types are supported: `string`, `timestamp`, `timestamp_runtime`

Optional:

- `direction` (String) Direction of the diff from context_time. The following directions are supported: `ago`, `later`. Required in `timestamp` and `timestamp_runtime` types
- `format` (String) Format used to replace variables. Required in `timestamp` and `timestamp_runtime` types
- `quantity` (Number) Quantity used to calculate diff from context_time. Required in `timestamp` and `timestamp_runtime` types
- `time_zone` (String) Time zone used to format the timestamp. Required in `timestamp` and `timestamp_runtime` types
- `unit` (String) Time unit used to calculate diff from context_time. The following units are supported: `hour`, `date`, `month`. Required in `timestamp` and `timestamp_runtime` types
- `value` (String) Fixed string which will replace variables at runtime. Required in `string` type


<a id="nestedatt--output_option--snowflake_output_option--snowflake_output_option_column_options"></a>
### Nested Schema for `output_option.snowflake_output_option.snowflake_output_option_column_options`

Required:

- `name` (String) Column name
- `type` (String) Data type

Optional:

- `timestamp_format` (String) Timestamp format
- `timezone` (String) Time zone
- `value_type` (String) Value type




<a id="nestedatt--filter_add_time"></a>
### Nested Schema for `filter_add_time`
Expand Down
23 changes: 23 additions & 0 deletions internal/client/entity/job_definition/input_option/snowflake.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package input_option

import (
"terraform-provider-trocco/internal/client/entity"
)

type SnowflakeInputOption struct {
Warehouse string `json:"warehouse"`
Database string `json:"database"`
Schema string `json:"schema"`
Query string `json:"query"`
FetchRows *int64 `json:"fetch_rows"`
ConnectTimeout *int64 `json:"connect_timeout"`
SocketTimeout *int64 `json:"socket_timeout"`
SnowflakeConnectionID int64 `json:"snowflake_connection_id"`
InputOptionColumns []SnowflakeInputOptionColumn `json:"input_option_columns"`
CustomVariableSettings *[]entity.CustomVariableSetting `json:"custom_variable_settings"`
}

type SnowflakeInputOptionColumn struct {
Name string `json:"name"`
Type string `json:"type"`
}
32 changes: 32 additions & 0 deletions internal/client/entity/job_definition/output_option/snowflake.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package output_option

import (
"terraform-provider-trocco/internal/client/entity"
)

type SnowflakeOutputOption struct {
Warehouse string `json:"warehouse"`
Database string `json:"database"`
Schema string `json:"schema"`
Table string `json:"table"`
Mode *string `json:"mode"`
EmptyFieldAsNull *bool `json:"empty_field_as_null"`
DeleteStageOnError *bool `json:"delete_stage_on_error"`
BatchSize *int64 `json:"batch_size"`
RetryLimit *int64 `json:"retry_limit"`
RetryWait *int64 `json:"retry_wait"`
MaxRetryWait *int64 `json:"max_retry_wait"`
DefaultTimeZone *string `json:"default_time_zone"`
SnowflakeConnectionId int64 `json:"snowflake_connection_id"`
SnowflakeOutputOptionColumnOptions []SnowflakeOutputOptionColumnOption `json:"snowflake_output_option_column_options"`
SnowflakeOutputOptionMergeKeys []string `json:"snowflake_output_option_merge_keys"`
CustomVariableSettings *[]entity.CustomVariableSetting `json:"custom_variable_settings"`
}

type SnowflakeOutputOptionColumnOption struct {
Name string `json:"name"`
Type string `json:"type"`
ValueType *string `json:"value_type"`
TimestampFormat *string `json:"timestamp_format"`
Timezone *string `json:"timezone"`
}
24 changes: 15 additions & 9 deletions internal/client/job_definition.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,30 +87,36 @@ type UpdateJobDefinitionInput struct {
}

type InputOption struct {
MySQLInputOption *inputOptionEntitites.MySQLInputOption `json:"mysql_input_option"`
GcsInputOption *inputOptionEntitites.GcsInputOption `json:"gcs_input_option"`
MySQLInputOption *inputOptionEntitites.MySQLInputOption `json:"mysql_input_option"`
GcsInputOption *inputOptionEntitites.GcsInputOption `json:"gcs_input_option"`
SnowflakeInputOption *inputOptionEntitites.SnowflakeInputOption `json:"snowflake_input_option"`
}

type InputOptionInput struct {
MySQLInputOption *parameter.NullableObject[input_options.MySQLInputOptionInput] `json:"mysql_input_option,omitempty"`
GcsInputOption *parameter.NullableObject[input_options.GcsInputOptionInput] `json:"gcs_input_option,omitempty"`
MySQLInputOption *parameter.NullableObject[input_options.MySQLInputOptionInput] `json:"mysql_input_option,omitempty"`
GcsInputOption *parameter.NullableObject[input_options.GcsInputOptionInput] `json:"gcs_input_option,omitempty"`
SnowflakeInputOption *parameter.NullableObject[input_options.SnowflakeInputOptionInput] `json:"snowflake_input_option,omitempty"`
}

type UpdateInputOptionInput struct {
MySQLInputOption *parameter.NullableObject[input_options.UpdateMySQLInputOptionInput] `json:"mysql_input_option,omitempty"`
GcsInputOption *parameter.NullableObject[input_options.UpdateGcsInputOptionInput] `json:"gcs_input_option,omitempty"`
MySQLInputOption *parameter.NullableObject[input_options.UpdateMySQLInputOptionInput] `json:"mysql_input_option,omitempty"`
GcsInputOption *parameter.NullableObject[input_options.UpdateGcsInputOptionInput] `json:"gcs_input_option,omitempty"`
SnowflakeInputOption *parameter.NullableObject[input_options.UpdateSnowflakeInputOptionInput] `json:"snowflake_input_option,omitempty"`
}

type OutputOption struct {
BigQueryOutputOption *outputOptionEntitites.BigQueryOutputOption `json:"bigquery_output_option"`
BigQueryOutputOption *outputOptionEntitites.BigQueryOutputOption `json:"bigquery_output_option"`
SnowflakeOutputOption *outputOptionEntitites.SnowflakeOutputOption `json:"snowflake_output_option"`
}

type OutputOptionInput struct {
BigQueryOutputOption *parameter.NullableObject[output_options.BigQueryOutputOptionInput] `json:"bigquery_output_option,omitempty"`
BigQueryOutputOption *parameter.NullableObject[output_options.BigQueryOutputOptionInput] `json:"bigquery_output_option,omitempty"`
SnowflakeOutputOption *parameter.NullableObject[output_options.SnowflakeOutputOptionInput] `json:"snowflake_output_option,omitempty"`
}

type UpdateOutputOptionInput struct {
BigQueryOutputOption *parameter.NullableObject[output_options.UpdateBigQueryOutputOptionInput] `json:"bigquery_output_option,omitempty"`
BigQueryOutputOption *parameter.NullableObject[output_options.UpdateBigQueryOutputOptionInput] `json:"bigquery_output_option,omitempty"`
SnowflakeOutputOption *parameter.NullableObject[output_options.UpdateSnowflakeOutputOptionInput] `json:"snowflake_output_option,omitempty"`
}

func (c *TroccoClient) CreateJobDefinition(in *CreateJobDefinitionInput) (*JobDefinition, error) {
Expand Down
36 changes: 36 additions & 0 deletions internal/client/parameter/job_definition/input_option/snowflake.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
package input_options

import (
"terraform-provider-trocco/internal/client/parameter"
)

type SnowflakeInputOptionInput struct {
Warehouse string `json:"warehouse"`
Database string `json:"database"`
Schema *parameter.NullableString `json:"schema"`
Query string `json:"query"`
FetchRows *parameter.NullableInt64 `json:"fetch_rows,omitempty"`
ConnectTimeout *parameter.NullableInt64 `json:"connect_timeout,omitempty"`
SocketTimeout *parameter.NullableInt64 `json:"socket_timeout,omitempty"`
SnowflakeConnectionID int64 `json:"snowflake_connection_id"`
InputOptionColumns []SnowflakeInputOptionColumn `json:"input_option_columns"`
CustomVariableSettings *[]parameter.CustomVariableSettingInput `json:"custom_variable_settings,omitempty"`
}

type UpdateSnowflakeInputOptionInput struct {
Warehouse *string `json:"warehouse,omitempty"`
Database *string `json:"database,omitempty"`
Schema *parameter.NullableString `json:"schema,omitempty"`
Query *string `json:"query,omitempty"`
FetchRows *parameter.NullableInt64 `json:"fetch_rows,omitempty"`
ConnectTimeout *parameter.NullableInt64 `json:"connect_timeout,omitempty"`
SocketTimeout *parameter.NullableInt64 `json:"socket_timeout,omitempty"`
SnowflakeConnectionID *int64 `json:"snowflake_connection_id,omitempty"`
InputOptionColumns []SnowflakeInputOptionColumn `json:"input_option_columns,omitempty"`
CustomVariableSettings *[]parameter.CustomVariableSettingInput `json:"custom_variable_settings,omitempty"`
}

type SnowflakeInputOptionColumn struct {
Name string `json:"name"`
Type string `json:"type"`
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package output_options

import (
"terraform-provider-trocco/internal/client/parameter"
)

type SnowflakeOutputOptionInput struct {
Warehouse string `json:"warehouse"`
Database string `json:"database"`
Schema string `json:"schema"`
Table string `json:"table"`
Mode *parameter.NullableString `json:"mode,omitempty"`
EmptyFieldAsNull *parameter.NullableBool `json:"empty_field_as_null,omitempty"`
DeleteStageOnError *parameter.NullableBool `json:"delete_stage_on_error,omitempty"`
BatchSize *parameter.NullableInt64 `json:"batch_size,omitempty"`
RetryLimit *parameter.NullableInt64 `json:"retry_limit,omitempty"`
RetryWait *parameter.NullableInt64 `json:"retry_wait,omitempty"`
MaxRetryWait *parameter.NullableInt64 `json:"max_retry_wait,omitempty"`
DefaultTimeZone *parameter.NullableString `json:"default_time_zone,omitempty"`
SnowflakeConnectionId int64 `json:"snowflake_connection_id"`
SnowflakeOutputOptionColumnOptions *parameter.NullableObjectList[SnowflakeOutputOptionColumnOptionInput] `json:"snowflake_output_option_column_options,omitempty"`
SnowflakeOutputOptionMergeKeys *parameter.NullableObjectList[string] `json:"snowflake_output_option_merge_keys,omitempty"`
CustomVariableSettings *[]parameter.CustomVariableSettingInput `json:"custom_variable_settings,omitempty"`
}

type UpdateSnowflakeOutputOptionInput struct {
Warehouse *string `json:"warehouse,omitempty"`
Database *string `json:"database,omitempty"`
Schema *string `json:"schema,omitempty"`
Table *string `json:"table,omitempty"`
Mode *parameter.NullableString `json:"mode,omitempty"`
EmptyFieldAsNull *parameter.NullableBool `json:"empty_field_as_null,omitempty"`
DeleteStageOnError *parameter.NullableBool `json:"delete_stage_on_error,omitempty"`
BatchSize *parameter.NullableInt64 `json:"batch_size,omitempty"`
RetryLimit *parameter.NullableInt64 `json:"retry_limit,omitempty"`
RetryWait *parameter.NullableInt64 `json:"retry_wait,omitempty"`
MaxRetryWait *parameter.NullableInt64 `json:"max_retry_wait,omitempty"`
DefaultTimeZone *parameter.NullableString `json:"default_time_zone,omitempty"`
SnowflakeConnectionId *int64 `json:"snowflake_connection_id,omitempty"`
SnowflakeOutputOptionColumnOptions *parameter.NullableObjectList[SnowflakeOutputOptionColumnOptionInput] `json:"snowflake_output_option_column_options,omitempty"`
SnowflakeOutputOptionMergeKeys *parameter.NullableObjectList[string] `json:"snowflake_output_option_merge_keys,omitempty"`
CustomVariableSettings *[]parameter.CustomVariableSettingInput `json:"custom_variable_settings,omitempty"`
}

type SnowflakeOutputOptionColumnOptionInput struct {
Name string `json:"name"`
Type string `json:"type"`
ValueType *string `json:"value_type,omitempty"`
TimestampFormat *string `json:"timestamp_format,omitempty"`
Timezone *string `json:"timezone,omitempty"`
}
Loading

0 comments on commit 38f35e9

Please sign in to comment.