Move FileFormat and related pieces to datafusion-datasource
#14873
Conversation
AdamGS
changed the title
FileFormat and related pieces to datafusion-datasource.FileFormat and related pieces to datafusion-datasource
| /// Coerces the file schema if the table schema uses a view type. | ||
| #[cfg(not(target_arch = "wasm32"))] |
There was a problem hiding this comment.
These functions used to be public but are only used in parquet (especially transform_binary_to_string), so I think it makes sense to put them here in the long-term.
There was a problem hiding this comment.
(so they will long term become part of the datasource/parquet crate?)
There was a problem hiding this comment.
That's my plan. I expect most of this module (and json, csv and avro) to move to the new format-specific crates, hopefully without too many leftovers.
There was a problem hiding this comment.
Already ran into one small issue with csv/json when I tried to move the Decoder trait, but I think that once I make that big split it'll be easier to find a home for all the shared things, hopefully without having to change them too much.
| pub use datafusion_datasource::file_compression_type; | ||
| use datafusion_datasource::file_scan_config::FileScanConfig; | ||
| pub use datafusion_datasource::file_format::*; |
There was a problem hiding this comment.
trying to keep existing use statements working
| /// relevant methods. [FileType] is only used in logical planning and only implements | ||
| /// the subset of methods required during logical planning. | ||
| #[derive(Debug)] | ||
| pub struct DefaultFileType { |
There was a problem hiding this comment.
it is really nice to see this broken up into smaller modules
| /// Coerces the file schema if the table schema uses a view type. | ||
| #[cfg(not(target_arch = "wasm32"))] |
There was a problem hiding this comment.
(so they will long term become part of the datasource/parquet crate?)
|
Thanks again @AdamGS |
Which issue does this PR close?
Part of #14444.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?