Skip to content

Commit

Permalink
minor improvements
Browse files Browse the repository at this point in the history
  • Loading branch information
jhamman committed Dec 18, 2024
1 parent 3678df6 commit de9c8aa
Show file tree
Hide file tree
Showing 8 changed files with 60 additions and 64 deletions.
6 changes: 4 additions & 2 deletions docs/developers/index.rst
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@

Developer Guide
---------------

.. toctree::
:maxdepth: 1
:hidden:

release
contributing
release
roadmap
11 changes: 6 additions & 5 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@ Zarr-Python
quickstart
user-guide/index
api/index
developers/contributing
developers/release
developers/index

**Version**: |version|

Expand All @@ -24,7 +23,7 @@ Zarr-Python
`Zulip Chat <https://ossci.zulipchat.com/>`_ |
`Zarr specifications <https://zarr-specs.readthedocs.io>`_

Zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification. Highlights include:
Zarr is a storage format for chunked, compressed, N-dimensional arrays based on an open-source specification. Highlights include:

* Create N-dimensional arrays with any NumPy dtype.
* Chunk arrays along any dimension.
Expand All @@ -45,7 +44,7 @@ Zarr-Python Documentation
Quick Start
^^^^^^^^^^^

New to Zarr? Check out the quick start guide. It contains an
New to Zarr? Check out the quick start guide. It contains a brief
introduction to Zarr's main concepts and links to additional tutorials.

+++
Expand Down Expand Up @@ -101,7 +100,9 @@ Zarr-Python Documentation
Contributor's Guide
^^^^^^^^^^^^^^^^^^^

Want to contribute to Zarr? We welcome contributions in the form of bug reports, bug fixes, documentation, enhancement proposals and more. The contributing guidelines will guide you through the process of improving Zarr.
Want to contribute to Zarr? We welcome contributions in the form of bug reports,
bug fixes, documentation, enhancement proposals and more. The contributing
guidelines will guide you through the process of improving Zarr.

+++

Expand Down
2 changes: 0 additions & 2 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
.. _quickstart:

.. ipython::
:suppress:

Expand Down
2 changes: 1 addition & 1 deletion docs/user-guide/arrays.rst
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ disk then load back into memory later, the functions
:func:`zarr.convenience.save` and :func:`zarr.convenience.load` may be
useful. E.g.:

.. ipython:: python
.. ipython::
:suppress:

In [144]: rm -r data/example.zarr
Expand Down
45 changes: 22 additions & 23 deletions docs/user-guide/consolidated_metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,44 +19,43 @@ metadata reads get child Group or Array nodes will *not* require reads from the
In Python, the consolidated metadata is available on the ``.consolidated_metadata``
attribute of the ``GroupMetadata`` object.

.. code-block:: python
.. TODO: remove :okwarning: after warnings are removed
>>> import zarr
>>> store = zarr.storage.MemoryStore({}, mode="w")
>>> group = zarr.open_group(store=store)
>>> group.create_array(shape=(1,), name="a")
>>> group.create_array(shape=(2, 2), name="b")
>>> group.create_array(shape=(3, 3, 3), name="c")
>>> zarr.consolidate_metadata(store)
.. ipython:: python
:okwarning:
import zarr
store = zarr.storage.MemoryStore()
group = zarr.open_group(store=store)
group.create_array(shape=(1,), name="a")
group.create_array(shape=(2, 2), name="b")
group.create_array(shape=(3, 3, 3), name="c")
zarr.consolidate_metadata(store)
If we open that group, the Group's metadata has a :class:`zarr.ConsolidatedMetadata`
that can be used.

.. code-block:: python
.. ipython:: python
>>> consolidated = zarr.open_group(store=store)
>>> consolidated.metadata.consolidated_metadata.metadata
{'b': ArrayV3Metadata(shape=(2, 2), fill_value=np.float64(0.0), ...),
'a': ArrayV3Metadata(shape=(1,), fill_value=np.float64(0.0), ...),
'c': ArrayV3Metadata(shape=(3, 3, 3), fill_value=np.float64(0.0), ...)}
consolidated = zarr.open_group(store=store)
consolidated.metadata.consolidated_metadata.metadata
Operations on the group to get children automatically use the consolidated metadata.

.. code-block:: python
.. ipython:: python
>>> consolidated["a"] # no read / HTTP request to the Store is required
<Array memory://.../a shape=(1,) dtype=float64>
consolidated["a"] # no read / HTTP request to the Store is required
With nested groups, the consolidated metadata is available on the children, recursively.

... code-block:: python
.. ipython:: python
:okwarning:
>>> child = group.create_group("child", attributes={"kind": "child"})
>>> grandchild = child.create_group("child", attributes={"kind": "grandchild"})
>>> consolidated = zarr.consolidate_metadata(store)
child = group.create_group("child", attributes={"kind": "child"})
grandchild = child.create_group("child", attributes={"kind": "grandchild"})
consolidated = zarr.consolidate_metadata(store)
>>> consolidated["child"].metadata.consolidated_metadata
ConsolidatedMetadata(metadata={'child': GroupMetadata(attributes={'kind': 'grandchild'}, zarr_format=3, )}, ...)
consolidated["child"].metadata.consolidated_metadata
Synchronization and Concurrency
-------------------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/user-guide/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ Advanced Topics
.. toctree::
:maxdepth: 1

consolidated_metadata
async
consolidated_metadata
extending
55 changes: 25 additions & 30 deletions docs/user-guide/storage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,19 @@ Implicit Store Creation
In most cases, it is not required to create a ``Store`` object explicitly. Passing a string
to Zarr's top level API will result in the store being created automatically.

.. code-block:: python
.. ipython:: python
>>> import zarr
>>> zarr.open("data/foo/bar", mode="r") # implicitly creates a read-only LocalStore
<Group file://data/foo/bar>
>>> zarr.open("s3://foo/bar", mode="r") # implicitly creates a read-only FsspecStore
<Group s3://foo/bar>
>>> data = {}
>>> zarr.open(data, mode="w") # implicitly creates a MemoryStore
<Group memory://4791444288>
import zarr
# implicitly create a writable LocalStore
zarr.open_group("data/foo/bar", mode="w")
# implicitly create a read-only FsspecStore
zarr.open_group("s3://noaa-nwm-retro-v2-zarr-pds", mode="r")
# implicitly creates a MemoryStore
data = {}
zarr.open_group(data, mode="w")
Explicit Store Creation
-----------------------
Expand All @@ -42,25 +45,21 @@ Local Store
The :class:`zarr.storage.LocalStore` stores data in a nested set of directories on a local
filesystem.

.. code-block:: python
.. ipython:: python
>>> import zarr
>>> store = zarr.storage.LocalStore("data/foo/bar", read_only=True)
>>> zarr.open(store=store)
<Group file://data/foo/bar>
store = zarr.storage.LocalStore("data/foo/bar", read_only=True)
zarr.open(store=store, mode='r')
Zip Store
~~~~~~~~~

The :class:`zarr.storage.ZipStore` stores the contents of a Zarr hierarchy in a single
Zip file. The `Zip Store specification_` is currently in draft form.

.. code-block:: python
.. ipython:: python
>>> import zarr
>>> store = zarr.storage.ZipStore("data.zip", mode="w")
>>> zarr.open(store=store, shape=(2,))
<Array zip://data.zip shape=(2,) dtype=float64
store = zarr.storage.ZipStore("data.zip", mode="w")
zarr.open(store=store, shape=(2,))
Remote Store
~~~~~~~~~~~~
Expand All @@ -71,26 +70,22 @@ such as cloud object storage (e.g. AWS S3, Google Cloud Storage, Azure Blob Stor
:class:`zarr.storage.FsspecStore` is backed by `Fsspec_` and can support any Fsspec backend
that implements the `AbstractFileSystem` API,

.. code-block:: python
.. ipython:: python
>>> import zarr
>>> store = zarr.storage.FsspecStore.from_url("gs://foo/bar", read_only=True)
>>> zarr.open(store=store)
<Array <FsspecStore(GCSFileSystem, foo/bar)> shape=(10, 20) dtype=float32>
store = zarr.storage.FsspecStore.from_url("s3://noaa-nwm-retro-v2-zarr-pds", read_only=True)
zarr.open_group(store=store, mode='r')
Memory Store
~~~~~~~~~~~~

The :class:`zarr.storage.FsspecStore` a in-memory store that allows for serialization of
Zarr data (metadata and chunks) to a dictionary.

.. code-block:: python
.. ipython:: python
>>> import zarr
>>> data = {}
>>> store = zarr.storage.MemoryStore(data)
>>> zarr.open(store=store, shape=(2, ))
<Array memory://4943638848 shape=(2,) dtype=float64>
data = {}
store = zarr.storage.MemoryStore(data)
zarr.open(store=store, shape=(2, ))
Developing custom stores
------------------------
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ docs = [
'ipython',
'pickleshare',
'rich',
's3fs',
]

[project.urls]
Expand Down

0 comments on commit de9c8aa

Please sign in to comment.