Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert VARCHAR, VARBINARY type to string #1281

Closed
wants to merge 2 commits into from

Conversation

nigurr
Copy link

@nigurr nigurr commented Nov 9, 2021

Description

Currently driver returns VARCHAR columns as []byte.
When using MapScan, the specific column contains only byte information where it can be represented as string (similar issue golang/go#22544)

r := make(map[string]interface{})
    rows.MapScan(r)

For example: MySQL Java driver maps this properly to string
https://github.com/mysql/mysql-connector-j/blob/4f7120a617b9d5efb9dedda9064b9896db424a60/src/main/core-api/java/com/mysql/cj/MysqlType.java#L277-L280

Similarly other sql vendor go clients map VARCHAR type to string.

This PR fix the mapping of VARCHAR to string.

Checklist

  • Code compiles correctly
  • Created tests which fail without the change (if possible)
  • All tests passing
  • Extended the README / documentation, if necessary
  • Added myself / the copyright holder to the AUTHORS file

@nigurr nigurr changed the title Convert VARCHAR, VAR_STRING type to string Convert VARCHAR, VARBINARY type to string Nov 9, 2021
@nigurr
Copy link
Author

nigurr commented Nov 10, 2021

@methane can you please review this?

@methane
Copy link
Member

methane commented Nov 10, 2021

readRow() is very low level function. We intentionally avoid allocation as possible here.

I know that scanning into interface{} has usability issue. But it is a limitation of current design of database/sql. Driver can not help it.

@methane
Copy link
Member

methane commented Nov 10, 2021

See also: jmoiron/sqlx#225 (comment)

@nigurr
Copy link
Author

nigurr commented Nov 10, 2021

See also: jmoiron/sqlx#225 (comment)

There are two sides for this issue (correct me If I am wrong here)

  1. go default sql mapper is not exposed as interface where one can not override/use custom mapper. database/sql: value converter interface for rows.Scan golang/go#22544
  2. It relies on src and dest type to map the best possible case (that's how it works with struct but not generic map[string]interface{})
  3. At the same time, src can define the right variable definition. In this case varchar is expected to be defined as string (other MySQL language drivers map it explicitly to string)

By mapping varchar to string, the go default mapping auto defines the variable as string type, which I believe go made the best possible option to leave it to driver implementation or destination/struct definition.

There are many cases where we can't define strict struct models for data entity and often very generic in data pipelines. It would be great to know any strong reason for not mapping varchar to string instead of byte[]

@methane
Copy link
Member

methane commented Nov 10, 2021

Correct.

  • It relies on src and dest type to map the best possible case (that's how it works with struct but not generic map[string]interface{})

Correct. But database/sql doesn't support struct and map[string]interface{}. You are mixing database/sql and sqlx. database/sql provides ColumnType.ScanType. So sqlx can use it instead of interface{} when calling Rows.Scan().

At the same time, src can define the right variable definition. In this case varchar is expected to be defined as string (other MySQL language drivers map it explicitly to string)

It is not Go way, because it may cause unnecessary allocations.

By mapping varchar to string, the go default mapping auto defines the variable as string type, which I believe go made the best possible option to leave it to driver implementation or destination/struct definition.

It makes allocation unavoidable in some situations.

There are many cases where we can't define strict struct models for data entity and often very generic in data pipelines.

For such cases, database/sql and this driver provide ColumnType.

It would be great to know any strong reason for not mapping varchar to string instead of byte[]

Avoid unnecessary allocation, convertion, and memcpy.

@@ -93,9 +93,6 @@ func (mf *mysqlField) typeDatabaseName() string {
}
return "TINYBLOB"
case fieldTypeVarChar:
if mf.charSet == collations[binaryCollation] {
return "VARBINARY"
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you remove it.

@methane methane closed this May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants