Allow reading IPC files that use Arrow extension type

[GPSnoopy]

Problem description

This is partially related to #9112 but much narrower in scope.

As the official Apache Arrow specs state, an implementation reading an Arrow file using extension types should work by falling back to the known parent type.

This extension metadata can annotate any of the built-in Arrow logical types. The intent is that an implementation that does not support an extension type can still handle the underlying data. For example a 16-byte UUID value could be embedded in FixedSizeBinary(16), and implementations that do not have this extension type can still work with the underlying binary values and pass along the custom_metadata in subsequent Arrow protocol messages.

Currently, using LazyFrame::scan_ipc() panics at the following line:

polars/polars/polars-core/src/datatypes/field.rs

Line 158 in 9a17368

dt => panic!("Arrow datatype {dt:?} not supported by Polars. You probably need to activate that data-type feature."),

I suspect that the following

polars/polars/polars-core/src/datatypes/field.rs

Lines 146 to 155 in 9a17368

	ArrowDataType::Extension(name, _, _) if name == "POLARS_EXTENSION_TYPE" => {
	#[cfg(feature = "object")]
	{
	DataType::Object("extension")
	}
	#[cfg(not(feature = "object"))]
	{
	panic!("activate the 'object' feature to be able to load POLARS_EXTENSION_TYPE")
	}
	}

should be followed by something like

            ArrowDataType::Extension(_, _, _) => dt.to_logical_type().into(),

which would cause the function to recurse with the parent logical type when encountering an unknown extension.

[GPSnoopy]

I have created two PRs:

• In Arrow2 fixing the cast function (Convert from_type to parent logical type before attempting a cast. jorgecarleitao/arrow2#1508).
• In Polars fixing LazyFrame::scan_ipc() when the file contains extension types (Add the ability to read Arrow IPC files containing extension types. #9381).

The Arrow2 PR needs to be merged first such that I can amend the Polars PR to depend on the official Arrow2 repository again.

[GPSnoopy]

The above does not seem to work for extension types of type Boolean. Not sure why; I can't see any special path for it.

The arrow2 PR is also not perfect. When the cast logical type just matches, no conversion is performed, but the array is not set to the target data type (I do not think this is the reason why Boolean does not work though, as other types do work).

[GPSnoopy]

To be honest, I have not been able to get the extension types to work smoothly with arrow2 and polars. It is a shame, because according to the documentation, it is the proper way to expand on existing types. Instead, for the time being, I'm just adding extra metadata to the fields.

I'll close the PRs (they're still there in case anyone wants to pick up where I left) but I'm leaving this ticket open as a feature request.