Support for PyArrow's ExtensionType.

[caseyclements]

Description

Add support for PyArrow Extension Types.

Context / Motivation

Here are some details on extending pyarrow.

The following binary example comes straight from the PyArrow documentation. We recently added Polars support to PyMongoArrow, and used this to create an ObjectIdType. It's implementation is almost identical to the example.

In MongoDB, each document stored in a collection requires a unique _id field that acts as a primary key.
If an inserted document omits the _id field, the MongoDB driver automatically generates an ObjectId for the _id field.
The MongoDB documentation on ObjectIds.

Until Polars supports pyarrow.ExtensionTypes, we must cast them to their base Arrow classes.

To reproduce the issue.

import pyarrow as pa
import polars as pl
import uuid


class UuidType(pa.ExtensionType):
    """For example, we could define a custom UUID type for 128-bit numbers
    which can be represented as FixedSizeBinary type with 16 bytes
    """

    def __init__(self):
        super().__init__(pa.binary(16), "my_package.uuid")

    def __arrow_ext_serialize__(self):
        # Since we don't have a parameterized type, we don't need extra
        # metadata to be deserialized
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        # Sanity checks, not required but illustrate the method signature.
        assert storage_type == pa.binary(16)
        assert serialized == b''
        # Return an instance of this subclass given the serialized
        # metadata.
        return UuidType()


uuid_type = UuidType()
storage_array = pa.array([uuid.uuid4().bytes for _ in range(4)], pa.binary(16))
extension_arr = pa.ExtensionArray.from_storage(uuid_type, storage_array)

print(f"{pl.from_arrow(storage_array) = }")
try:
    print(f"{pl.from_arrow(extension_arr) = }")
except pl.exceptions.ComputeError as exc:
    print(f"{exc = }")

[caseyclements]

Here's the output of the example above.
See also test_polars.py in pymongoarrow.

(polars) ~/src/mongo-arrow/sandbox (main)
$ python polars_extensiontypes.py
pl_arr = shape: (4,)

Series: '' [binary]
[
	b"\xd1\x91\x0b\xae\xad&J\x99\xb0\x1f\xfe\x04E\xd0}\x99"
	b"\xf4\%\xe4\xc7\xeeMA\xb6\xd4s\xd8d\xaa8R"
	b"\xfb\xd3aa\x87\xf5M\xda\xae\x8e\xf1\xa5]\x04\xb4E"
	b"\x08G\xcc\x86\x1cW@\xf8\x8c:X\x9e#=\xa2F"
]
exc = ComputeError('cannot create series from Extension("my_package.uuid", FixedSizeBinary(16), Some(""))')

[deanm0000]

I would like to see it, not sure if this is the same as #9112. Definitely not the same as this one but highly related #9373.

I'd also like to see this in conjunction with Union Types #10827

[caseyclements]

I didn't see #9112 when I did a search of existing issues. This is a duplicate.

[caseyclements]

Closing as duplicate of #9112