Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prune stores #1791

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4b20501
chore: add deprecation warnings to v3 classes / functions
d-v-b Apr 4, 2024
eab9fd2
Resolve Mypy erorrs in `v3` branch (#1692)
DahnJ Apr 6, 2024
48c14d3
Specify hatch envs using GitHub actions matrix for v3 tests (#1728)
maxrjones Apr 8, 2024
962b537
black -> ruff format + cleanup (#1639)
Saransh-cpp Apr 8, 2024
08993b1
Remove outdated dev install docs from installation.rst and link to co…
aldenks Apr 8, 2024
bbede29
chore: remove old v3 implementation
d-v-b Apr 10, 2024
5f2c4b4
Merge branch 'v3' of github.com:zarr-developers/zarr-python into remo…
d-v-b Apr 10, 2024
244c3c6
chore: remove more version-conditional logic
d-v-b Apr 10, 2024
99455c7
Merge branch 'v3' of github.com:zarr-developers/zarr-python into remo…
d-v-b Apr 11, 2024
dcddb63
chore: prune out n5, abs, sqlite, zip, redis, mongodb, dbm, lmdb stores
d-v-b Apr 11, 2024
dd7d3a8
chore: remove v3_storage_transformers.py again
d-v-b Apr 11, 2024
6e52bfe
chore: remove v3_storage_transformers.py again
d-v-b Apr 11, 2024
1818acc
Merge branch 'v3' of github.com:zarr-developers/zarr-python into remo…
d-v-b Apr 12, 2024
f3902c4
chore: prune out n5, abs, sqlite, zip, redis, mongodb, dbm, lmdb stores
d-v-b Apr 11, 2024
ebcdd77
Merge branch 'prune_stores' of github.com:d-v-b/zarr-python into prun…
d-v-b Apr 12, 2024
14e60ea
Merge branch 'v3' into prune_stores
d-v-b Apr 19, 2024
84a882c
Revert "chore: prune out n5, abs, sqlite, zip, redis, mongodb, dbm, l…
d-v-b Apr 22, 2024
ff406dd
chore: remove n5 storage
d-v-b Apr 22, 2024
981ff7c
chore: remove redis storage
d-v-b Apr 22, 2024
3da088c
chore: remove mongo storage
d-v-b Apr 22, 2024
6e95673
chore: remove abs storage
d-v-b Apr 22, 2024
fb7629f
chore: remove lmdb storage
d-v-b Apr 22, 2024
6a8a92c
chore: remove dbm storage
d-v-b Apr 22, 2024
df179c3
chore: remove sqlite storage
d-v-b Apr 22, 2024
4da92ab
Merge branch 'prune_stores' of github.com:d-v-b/zarr-python into prun…
d-v-b Apr 22, 2024
76038d2
Merge branch 'v3' into prune_stores
d-v-b Apr 22, 2024
3513090
docs: add release notes
d-v-b Apr 22, 2024
a55f8d6
docs: edit release notes
d-v-b Apr 22, 2024
c81d1dd
Merge branch 'v3' of github.com:zarr-developers/zarr-python into prun…
d-v-b Apr 22, 2024
8573203
docs: remove n5.rst
d-v-b Apr 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/windows-testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ jobs:
azurite -l ~/blob_emulator --debug debug.log 2>&1 > stdouterr.log &
pytest -sv --timeout=300
env:
ZARR_TEST_ABS: 1
ZARR_V3_EXPERIMENTAL_API: 1
ZARR_V3_SHARDING: 1
- name: Conda info
Expand Down
1 change: 0 additions & 1 deletion docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ API reference
api/core
api/hierarchy
api/storage
api/n5
api/convenience
api/codecs
api/attrs
Expand Down
5 changes: 0 additions & 5 deletions docs/api/n5.rst

This file was deleted.

18 changes: 0 additions & 18 deletions docs/api/storage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,30 +11,12 @@ Storage (``zarr.storage``)
.. automethod:: close
.. automethod:: flush

.. autoclass:: DBMStore

.. automethod:: close
.. automethod:: flush

.. autoclass:: LMDBStore

.. automethod:: close
.. automethod:: flush

.. autoclass:: SQLiteStore

.. automethod:: close

.. autoclass:: MongoDBStore
.. autoclass:: RedisStore
.. autoclass:: LRUStoreCache

.. automethod:: invalidate
.. automethod:: invalidate_values
.. automethod:: invalidate_keys

.. autoclass:: ABSStore

.. autoclass:: FSStore

.. autoclass:: ConsolidatedMetadataStore
Expand Down
5 changes: 0 additions & 5 deletions docs/api/v3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,6 @@ You can also check ``Store type`` here (which indicates Zarr V3).
.. autoclass:: MemoryStoreV3
.. autoclass:: DirectoryStoreV3
.. autoclass:: ZipStoreV3
.. autoclass:: RedisStoreV3
.. autoclass:: MongoDBStoreV3
.. autoclass:: DBMStoreV3
.. autoclass:: LMDBStoreV3
.. autoclass:: SQLiteStoreV3
.. autoclass:: LRUStoreCacheV3
.. autoclass:: ConsolidatedMetadataStoreV3

Expand Down
12 changes: 0 additions & 12 deletions docs/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -161,18 +161,6 @@ optional dependencies to be installed), run::

$ python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst

Note that some tests also require storage services to be running
locally. To run the Azure Blob Service storage tests, run an Azure
storage emulator (e.g., azurite) and set the environment variable
``ZARR_TEST_ABS=1``. If you're using Docker to run azurite, start the service with::

docker run --rm -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --loose --blobHost 0.0.0.0

To run the Mongo DB storage tests, run a Mongo
server locally and set the environment variable ``ZARR_TEST_MONGO=1``.
To run the Redis storage tests, run a Redis server locally on port
6379 and set the environment variable ``ZARR_TEST_REDIS=1``.

All tests are automatically run via GitHub Actions for every pull
request and must pass before code can be accepted. Test coverage is
also collected automatically via the Codecov service, and total
Expand Down
7 changes: 7 additions & 0 deletions docs/release.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ Release notes
Unreleased (v3)
---------------

Refactoring
~~~~~~~~~~~

* Remove storage classes for the following backends: N5, Redis, MongoDB, ABS, LMDB, DBM, and SQLite.
The intention is that these storage classes will be developed external to ``zarr-python``.
By :user:`Davis Bennett <d-v-b>` :issue:`1791`.

Maintenance
~~~~~~~~~~~

Expand Down
75 changes: 2 additions & 73 deletions docs/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -805,60 +805,6 @@ boundaries. Note also that the ``close()`` method must be called after writing
any data to the store, otherwise essential records will not be written to the
underlying zip file.

Another storage alternative is the :class:`zarr.storage.DBMStore` class, added
in Zarr version 2.2. This class allows any DBM-style database to be used for
storing an array or group. Here is an example using a Berkeley DB B-tree
database for storage (requires `bsddb3
<https://www.jcea.es/programacion/pybsddb.htm>`_ to be installed)::

>>> import bsddb3
>>> store = zarr.DBMStore('data/example.bdb', open=bsddb3.btopen)
>>> root = zarr.group(store=store, overwrite=True)
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
>>> z[:] = 42
>>> store.close()

Also added in Zarr version 2.2 is the :class:`zarr.storage.LMDBStore` class which
enables the lightning memory-mapped database (LMDB) to be used for storing an array or
group (requires `lmdb <https://lmdb.readthedocs.io/>`_ to be installed)::

>>> store = zarr.LMDBStore('data/example.lmdb')
>>> root = zarr.group(store=store, overwrite=True)
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
>>> z[:] = 42
>>> store.close()

In Zarr version 2.3 is the :class:`zarr.storage.SQLiteStore` class which
enables the SQLite database to be used for storing an array or group (requires
Python is built with SQLite support)::

>>> store = zarr.SQLiteStore('data/example.sqldb')
>>> root = zarr.group(store=store, overwrite=True)
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
>>> z[:] = 42
>>> store.close()

Also added in Zarr version 2.3 are two storage classes for interfacing with server-client
databases. The :class:`zarr.storage.RedisStore` class interfaces `Redis <https://redis.io/>`_
(an in memory data structure store), and the :class:`zarr.storage.MongoDB` class interfaces
with `MongoDB <https://www.mongodb.com/>`_ (an object oriented NoSQL database). These stores
respectively require the `redis-py <https://redis-py.readthedocs.io>`_ and
`pymongo <https://api.mongodb.com/python/current/>`_ packages to be installed.

For compatibility with the `N5 <https://github.com/saalfeldlab/n5>`_ data format, Zarr also provides
an N5 backend (this is currently an experimental feature). Similar to the zip storage class, an
:class:`zarr.n5.N5Store` can be instantiated directly::

>>> store = zarr.N5Store('data/example.n5')
>>> root = zarr.group(store=store)
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
>>> z[:] = 42

For convenience, the N5 backend will automatically be chosen when the filename
ends with `.n5`::

>>> root = zarr.open('data/example.n5', mode='w')

Distributed/cloud storage
~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -902,23 +848,6 @@ Here is an example using S3Map to read an array created previously::
>>> z[:].tobytes()
b'Hello from the cloud!'

Zarr now also has a builtin storage backend for Azure Blob Storage.
The class is :class:`zarr.storage.ABSStore` (requires
`azure-storage-blob <https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python>`_
to be installed)::

>>> import azure.storage.blob
>>> container_client = azure.storage.blob.ContainerClient(...) # doctest: +SKIP
>>> store = zarr.ABSStore(client=container_client, prefix='zarr-testing') # doctest: +SKIP
>>> root = zarr.group(store=store, overwrite=True) # doctest: +SKIP
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4') # doctest: +SKIP
>>> z[:] = 42 # doctest: +SKIP

When using an actual storage account, provide ``account_name`` and
``account_key`` arguments to :class:`zarr.storage.ABSStore`, the
above client is just testing against the emulator. Please also note
that this is an experimental feature.

Note that retrieving data from a remote service via the network can be significantly
slower than retrieving data from a local file system, and will depend on network latency
and bandwidth between the client and server systems. If you are experiencing poor
Expand Down Expand Up @@ -1590,8 +1519,8 @@ storage.
Note that if an array or group is backed by an in-memory store like a ``dict`` or
:class:`zarr.storage.MemoryStore`, then when it is pickled all of the store data will be
included in the pickled data. However, if an array or group is backed by a persistent
store like a :class:`zarr.storage.DirectoryStore`, :class:`zarr.storage.ZipStore` or
:class:`zarr.storage.DBMStore` then the store data **are not** pickled. The only thing
store like a :class:`zarr.storage.DirectoryStore` or :class:`zarr.storage.ZipStore`
then the store data **are not** pickled. The only thing
that is pickled is the necessary parameters to allow the store to re-open any
underlying files or databases upon being unpickled.

Expand Down
Loading
Loading