Skip to content

Commit

Permalink
doc: remove parts of parallel from execution
Browse files Browse the repository at this point in the history
  • Loading branch information
ianhi committed Mar 1, 2025
1 parent 543103a commit ea18b8b
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions docs/docs/icechunk-python/parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ def write_timestamp(*, itime: int, session: Session) -> None:

Now execute the writes.

```python exec="on" session="parallel" source="material-block"
<!-- ```python exec="on" session="parallel" source="material-block" -->
```python
from concurrent.futures import ThreadPoolExecutor, wait
from icechunk.distributed import merge_sessions

Expand All @@ -74,7 +75,8 @@ print(session.commit("finished writes"))

Verify that the writes worked as expected:

```python exec="on" session="parallel" source="material-block"
<!-- ```python exec="on" session="parallel" source="material-block" -->
```python
ondisk = xr.open_zarr(repo.readonly_session("main").store, consolidated=False)
xr.testing.assert_identical(ds, ondisk)
```
Expand Down Expand Up @@ -102,7 +104,7 @@ There are three key points to keep in mind:

First we modify `write_task` to return the `Session`:

```python exec="on" session="parallel" source="material-block"
```python
from icechunk import Session

def write_timestamp(*, itime: int, session: Session) -> Session:
Expand All @@ -116,7 +118,7 @@ def write_timestamp(*, itime: int, session: Session) -> Session:
Now we issue write tasks within the [`session.allow_pickling()`](./reference/md#icechunk.Session.allow_pickling) context, gather the Sessions from individual tasks,
merge them, and make a successful commit.

```python exec="on" session="parallel" source="material-block"
```python
from concurrent.futures import ProcessPoolExecutor
from icechunk.distributed import merge_sessions

Expand Down

0 comments on commit ea18b8b

Please sign in to comment.