Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a dict + non-dict pages mix within a column chunk #117

Merged
merged 19 commits into from
Feb 8, 2025

Conversation

gaborcsardi
Copy link
Member

@gaborcsardi gaborcsardi commented Feb 7, 2025

Closes #110.

  • convert_column_to_r_dicts
  • convert_column_to_r_dicts_na
  • convert_column_to_r_int64_dict_nomiss
  • convert_column_to_r_int64_dict_miss
  • convert_column_to_r_float_dict_nomiss
  • convert_column_to_r_float_dict_miss
  • convert_column_to_r_int96_dict_nomiss
  • convert_column_to_r_int96_dict_miss
  • convert_column_to_r_ba_string_dict_nomiss
  • convert_column_to_r_ba_decimal_dict_nomiss
  • convert_column_to_r_ba_raw_dict_nomiss
  • convert_column_to_r_ba_uuid_dict_nomiss
  • convert_column_to_r_ba_float16_dict_nomiss
  • convert_column_to_r_int32_decimal_dict_nomiss
  • convert_column_to_r_int64_decimal_dict_nomiss

@gaborcsardi
Copy link
Member Author

gaborcsardi commented Feb 8, 2025

Tests:

  • INT32
  • DOUBLE
  • BOOLEAN (cannot be a dictionary, even though we implement it)
  • INT64
  • FLOAT
  • INT96
  • BA (string)
  • BA (decimal)
  • BA (raw)
  • BA (uuid) Cannot check because pyarrow cannot write a UUID column
  • BA (float16)
  • INT32 (decimal)
  • INT64 (decimal)
  • with missing values for all

Also test mixed dict + non-dict pages.
In mixed dict + non-dict column chunks.
`as.POSIXct()` behaves pretty interestingly...
Trick is from parsedate.
[ci skip]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reader cannot handle if only part of a column chunk is dictionary encoded
1 participant