From 6baff6488fafacdc5915666530dddf0fd4deeba2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Mon, 27 Nov 2023 18:43:08 +0100
Subject: [PATCH 01/21] Fix script names in documentation

This fixes a previous omission and updates usage help snippets to
match documentation headings (and actual script names).
---
 docs/source/admin.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/docs/source/admin.rst b/docs/source/admin.rst
index ad2605d..c0e5094 100644
--- a/docs/source/admin.rst
+++ b/docs/source/admin.rst
@@ -112,8 +112,8 @@ files, in JSON format, in the study directory:
 
 .. code-block:: console
 
-  $ icf-utils getmeta_studyvisit -h
-  usage: getmeta_studyvisit [-h] [-o PATH] --id STUDY-ID VISIT-ID
+  $ icf-utils deposit_visit_metadata -h
+  usage: deposit_visit_metadata [-h] [-o PATH] --id STUDY-ID VISIT-ID
 
 ``deposit_visit_dataset``
 """""""""""""""""""""""""
@@ -136,8 +136,8 @@ represents the actual dataset as a compressed archive.
 
 .. code-block:: console
 
-   $ icf-utils dataladify_studyvisit_from_meta -h
-   usage: dataladify_studyvisit_from_meta [-h] [-o PATH] --id STUDY-ID VISIT-ID
+   $ icf-utils deposit_visit_dataset -h
+   usage: deposit_visit_dataset [-h] --id STUDY-ID VISIT-ID [-o PATH] [--store-url URL]
 
 ``catalogify_studyvisit_from_meta``
 """""""""""""""""""""""""""""""""""
@@ -149,4 +149,4 @@ folder in the study directory.
 .. code-block:: console
 
   $ icf-utils dataladify_studyvisit_from_meta --help
-  usage: dataladify_studyvisit_from_meta [-h] [-o PATH] --id STUDY-ID VISIT-ID
+  usage: catalogify_studyvisit_from_meta [-h] [-o PATH] --id STUDY-ID VISIT-ID

From a723682c7ed023bcc1d072e23966e4b2ed47bb76 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Wed, 29 Nov 2023 13:59:24 +0100
Subject: [PATCH 02/21] Update docs requirements

Seeing that a build on Sphinx 7.0.1 succeeded, but a build on Sphinx
7.2.6 failed with Furo 2023.5.20 theme, I decided to bump furo version
and narrow down Sphinx to a minor release.

Furo changelog: https://pradyunsg.me/furo/changelog/

If you are reading this commit message and considering updating or
loosening the dependencies, then you probably should do it without
issues. I suppose even now an unpinned dependency would work; it's
probably on rare occasions when the released version of the theme
needs to catch up to Sphinx development, which likely caused us to
introduce pinning in first place. In either case, it would seem that
pinning Furo to a specific version, but Sphinx only to major release
is not a good idea.
---
 docs/requirements.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/requirements.txt b/docs/requirements.txt
index bed3e43..ed6ac1f 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,2 +1,2 @@
-Sphinx >= 7.0, < 8.0
-furo==2023.5.20
+Sphinx == 7.2.*
+furo == 2023.9.10

From 79fee18fbd2cb111435fb0829760e0e42722be5c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Wed, 29 Nov 2023 16:43:40 +0100
Subject: [PATCH 03/21] Draft the user docs for generating DataLad datasets

---
 docs/source/user/datalad-generate.rst | 64 +++++++++++++++++++++++++++
 docs/source/user/index.rst            |  1 +
 2 files changed, 65 insertions(+)
 create mode 100644 docs/source/user/datalad-generate.rst

diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
new file mode 100644
index 0000000..37f459d
--- /dev/null
+++ b/docs/source/user/datalad-generate.rst
@@ -0,0 +1,64 @@
+.. _dl-generate:
+
+Generate DataLad datasets
+-------------------------
+
+The ICF archive for a given project contains DICOM files packaged in
+tar archives (DICOM tarballs). In this section we describe creating
+DataLad datasets, which index content and location of these tarballs,
+for DataLad-based access on institute-local infrastructure.
+
+In principle, such datasets are *lightweight*, meaning that they only
+index the content that can be retrieved from the ICF archive (all
+access restrictions apply). Using DataLad can simplify local access,
+allow raw data versioning, and enable logical transformations of the
+DICOM folder structure - see :ref:`dl-advanced` for examples of the
+latter.
+
+Obtain the tarball
+^^^^^^^^^^^^^^^^^^
+
+First, create an empty directory to be the local dataset store. The
+last path component must be the ``project-ID`` used by the ICF store,
+because following commands use project and visit IDs to determine
+paths.
+
+.. code-block:: bash
+
+   mkdir -p local_dicomstore/<project-ID>
+
+Download the visit tarball:
+
+.. code-block:: bash
+
+   cd local_dicomstore/<project-ID>
+   datalad download ...
+   cd ../..
+
+Deposit visit metadata alongside tarball
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   singularity run -B $STORE_DIR icf.sif deposit_visit_metadata --store-dir $STORE_DIR --id <project-ID> <visit ID>
+
+Deposit dataset alongside tarball
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+DataLad dataset is created based on the metadata extracted in the
+previous step.  Additionally, you need to provide the base URL of the
+ICF store, ``<ICF STORE URL>`` (this base URL should not contain study
+or visit ID). The URL, combined with respective IDs, will be
+registered in the dataset as the source of the DICOM tarball, and used
+for retrieval by dataset clones.
+
+.. code-block:: bash
+
+   singularity run -B $STORE_DIR icf.sif deposit_visit_dataset --store-dir $STORE_DIR --store-url <ICF STORE URL>
+
+This will produce two files, ...
+
+Remove the tarball
+^^^^^^^^^^^^^^^^^^
+
+The DICOM tarball can be safely removed.
diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst
index fcee949..cc60f64 100644
--- a/docs/source/user/index.rst
+++ b/docs/source/user/index.rst
@@ -15,5 +15,6 @@ Please contact `ICF personnel`_ to get access and for any authentication-related
    :caption: Contents:
 
    browser
+   datalad-generate
    datalad
    datalad-advanced

From 9064bcb601c3a07ef3ee7af5db3d23abdb6fbe56 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Tue, 12 Dec 2023 18:54:31 +0100
Subject: [PATCH 04/21] Fill in most blanks in DataLad dataset generation

---
 docs/source/user/datalad-generate.rst | 40 ++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 4 deletions(-)

diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index 37f459d..035cbb7 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -35,12 +35,31 @@ Download the visit tarball:
    datalad download ...
    cd ../..
 
+For the following examples, the *absolute path* to the local dicom
+store will be represented by ``$STORE_DIR``:
+
+.. code-block:: bash
+
+   export STORE_DIR=$PWD/local_dicomstore
+
+
 Deposit visit metadata alongside tarball
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Information required to create a DataLad dataset needs to be extracted
+from the tarball:
+
 .. code-block:: bash
 
-   singularity run -B $STORE_DIR icf.sif deposit_visit_metadata --store-dir $STORE_DIR --id <project-ID> <visit ID>
+   singularity run -B $STORE_DIR icf.sif deposit_visit_metadata \
+     --store-dir $STORE_DIR --id <project-ID> <visit ID>
+
+This will generate two files, ``<visit ID>_metadata_dicoms.json`` and
+``<visit ID>_metadata_tarball.json``, and place them alongside the
+tarball. The former contains metadata describing individual files
+within the tarball (relative path, MD5 checksum, size, and a small
+subset of DICOM headers describing acquisition type), and the latter
+describes the tarball itself.
 
 Deposit dataset alongside tarball
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -54,11 +73,24 @@ for retrieval by dataset clones.
 
 .. code-block:: bash
 
-   singularity run -B $STORE_DIR icf.sif deposit_visit_dataset --store-dir $STORE_DIR --store-url <ICF STORE URL>
+   singularity run -B $STORE_DIR icf.sif deposit_visit_dataset \
+     --store-dir $STORE_DIR --store-url <ICF STORE URL>
 
-This will produce two files, ...
+This will produce two files, ``<visit ID>_XDLA--refs`` and ``<visit
+ID>_XDLA--repo-export`` (text file and zip archive
+respectively). Together, they are a representation of a (lightweight)
+DataLad dataset, and contain the information necessary to retrieve the
+data content with DataLad (but do not contain the data content
+itself).
 
 Remove the tarball
 ^^^^^^^^^^^^^^^^^^
 
-The DICOM tarball can be safely removed.
+Finally, the DICOM tarball can be safely removed.
+
+.. code-block:: bash
+
+   rm local_dicomstore/<project-ID>/<visit ID>_dicom.tar
+
+The local dicom store can be used as a DataLad entry point for
+obtaining the dicom files.

From 82026ff5328d7016fa112b6af63a2f560202bd24 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 17:49:25 +0100
Subject: [PATCH 05/21] docs: rename datalad-based access

---
 docs/source/user/{datalad.rst => datalad-access.rst} | 0
 docs/source/user/index.rst                           | 2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename docs/source/user/{datalad.rst => datalad-access.rst} (100%)

diff --git a/docs/source/user/datalad.rst b/docs/source/user/datalad-access.rst
similarity index 100%
rename from docs/source/user/datalad.rst
rename to docs/source/user/datalad-access.rst
diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst
index cc60f64..3625e02 100644
--- a/docs/source/user/index.rst
+++ b/docs/source/user/index.rst
@@ -16,5 +16,5 @@ Please contact `ICF personnel`_ to get access and for any authentication-related
 
    browser
    datalad-generate
-   datalad
+   datalad-access
    datalad-advanced

From e4bc0f5f40ce4c2f973027705204a189c3dcdbd5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 18:23:51 +0100
Subject: [PATCH 06/21] Rewrite datalad-based access

This changes the store base URL used in the examples from
https://data.inm-icf.de to file:///data/group/..., reflecting the fact
that the DataLad datasets are not provided by the ICF -- but they can
be generated on local infrastructure instead.

Title is changed (making it more similar to the previous section), and
some explanations are slightly tweaked.
---
 docs/source/user/datalad-access.rst | 42 ++++++++++++++++++-----------
 1 file changed, 26 insertions(+), 16 deletions(-)

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index a4a3001..f03343d 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -1,5 +1,7 @@
-DataLad-based access
---------------------
+.. _dl-access:
+
+Access data with DataLad
+------------------------
 
 Software requirements
 ^^^^^^^^^^^^^^^^^^^^^
@@ -44,41 +46,43 @@ to DataLad.
 Clone & get
 ^^^^^^^^^^^
 
-A visit dataset can be cloned with DataLad from a URL containing the
-following components:
+If a visit dataset has been prepared (procedure described in
+:ref:`dl-generate`) and saved in an accessible location, it can be
+cloned with DataLad from a URL containing the following components:
 
-* store base URL (e.g., ``https://data.inm-icf.de``)
+* a set of configuration parameters, always constant
+* store base URL (e.g., ``file:///data/group/groupname/local_dicom_store``) [2]_
 * study ID (e.g., ``my-study``)
 * visit ID (e.g., ``P000123``)
-* a set of additional parameters, always constant
+* a file name suffix / template, ``_annex{{annex_key}}`` (verbatim), always constant
 
 The pattern for the URL is::
 
-    'datalad-annex::?type=external&externaltype=uncurl&url=<store base URL>/<study ID>/<visit ID>_{{annex_key}}&encryption=none'
+    'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=<store base URL>/<study ID>/<visit ID>_{{annex_key}}'
 
 Given the exemplary values above, the pattern would expand to
 
 .. code-block::
 
-    'datalad-annex::?type=external&externaltype=uncurl&url=https://data.inm-icf.de/my-study/P000123_{{annex_key}}&encryption=none'
+    'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///data/group/groupname/local_dicom_store/my-study/P000123_{{annex_key}}'
 
-.. note:: The URL is arguably a bit clunky. A convenience short cut can be provided via configuration item ``datalad.clone.url-substitute.<label>`` and a substitution rule based on regular expressions. For example, clone URLs can be shortened to require only an identifier (here, ``https://data.inm-icf.de``), study ID, and visit ID (``inm-icf/<study-ID>/<visit-ID>``) with the following configuration:
+.. note:: The URL is arguably a bit clunky. A convenience short cut can be provided via configuration item ``datalad.clone.url-substitute.<label>`` and a substitution rule based on regular expressions. For example, clone URLs can be shortened to require only an identifier (here, ``file:///data/group/groupname/local_dicom_store``), study ID, and visit ID (``inm-icf/<study-ID>/<visit-ID>``) with the following configuration:
 
    .. code-block::
 
-      git config --global datalad.clone.url-substitute.inm-icf ',^https://data.inm-icf.de/([^/]+)/(.*)$,datalad-annex::?type=external&externaltype=uncurl&url=https://data.inm-icf.de/\1/\2_{{annex_key}}&encryption=none'
+      git config --global datalad.clone.url-substitute.inm-icf ',^file:///data/group/groupname/local_dicom_store/([^/]+)/(.*)$,datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///data/group/groupname/local_dicom_store/\1/\2_{{annex_key}}'
 
-   This configuration allows DataLad to take any URL of the form ``https://data.inm-icf.de/<study-ID>/<visit-ID>`` and assemble the required ``datalad-annex::...`` URL on its own, and a clone call shortens into ``datalad clone https://data.inm-icf.de/my-study/P000123``.
+   This configuration allows DataLad to take any URL of the form ``file:///data/group/groupname/local_dicom_store/<study-ID>/<visit-ID>`` and assemble the required ``datalad-annex::...`` URL on its own, and a clone call shortens into ``datalad clone file:///data/group/groupname/local_dicom_store/my-study/P000123``.
    You are free to adjust this configuration custom to your needs and preferences.
    Further documentation on it can be found in the `DataLad Docs`_.
 
 .. _DataLad Docs: http://docs.datalad.org/en/stable/design/url_substitution.html
 
 Cloning will retrieve a lightweight dataset, which does not (yet)
-contain file content. File content can be retrieved with `datalad
-get`. DataLad will handle download and unpacking of the tar file.
-Take a look at the section :ref:`dl-advanced` to learn about
-useful convenience features DataLad adds on top of this.
+contain file content. File content can be retrieved with ``datalad
+get``. DataLad will handle download and unpacking of the tar file.
+Take a look at the section :ref:`dl-advanced` to learn about useful
+convenience features DataLad adds on top of this.
 
 Catalog-based clone URLs
 ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -93,4 +97,10 @@ clicking the "Download with DataLad" button on each individual visit ID.
 .. [1] To install software with pip, run a call such as the one below
        in your favourite `virtual environment <https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/>`_::
 
-              python -m pip install datalad-next
+         python -m pip install datalad-next
+
+.. [2] Examples use ``file://`` URLs, given that the datasets are most
+       likely to be generated on institute-local infrastructure. Other
+       protocoles (e.g. ``https://`` or ``ssh://``) can be substituted
+       depending on the particular setup, without affecting the URL
+       structure.

From a1e32b81d486ecdda5ad8852c4beb1e1afc68451 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 19:05:09 +0100
Subject: [PATCH 07/21] Rename admin docs to reference

Apart from generating dicom tarballs, the icf tooling can be used by
either icf or its users to dataladify the datasets. Therefore, it
makes sense to rename the current "admin docs" (which mostly explain
containerized execution and included scripts) as
"reference". Following commits can break it down a bit more.
---
 docs/source/index.rst                    | 2 +-
 docs/source/{admin.rst => reference.rst} | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename docs/source/{admin.rst => reference.rst} (100%)

diff --git a/docs/source/index.rst b/docs/source/index.rst
index b40546f..bee0fac 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -16,7 +16,7 @@ individuals.
    :caption: Contents:
 
    user/index
-   admin
+   reference
    developer
 
 
diff --git a/docs/source/admin.rst b/docs/source/reference.rst
similarity index 100%
rename from docs/source/admin.rst
rename to docs/source/reference.rst

From 10ad36bda08c84211cfa4673524122139232c2b1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 19:31:25 +0100
Subject: [PATCH 08/21] Split reference and admin docs into separate pages

This reflects the fact that dataladification scripts are no longer
meant to be used by the ICF admins only - in fact, they are most
likely meant to be used by ICF users.
---
 docs/source/admin.rst     | 26 ++++++++++++++++++++++++++
 docs/source/index.rst     |  1 +
 docs/source/reference.rst | 33 ++++++++-------------------------
 3 files changed, 35 insertions(+), 25 deletions(-)
 create mode 100644 docs/source/admin.rst

diff --git a/docs/source/admin.rst b/docs/source/admin.rst
new file mode 100644
index 0000000..febff3e
--- /dev/null
+++ b/docs/source/admin.rst
@@ -0,0 +1,26 @@
+Administrator docs
+==================
+
+Archival workflow
+-----------------
+
+The main part of visit archival is the creation a TAR file.
+
+The DataLad dataset can be generated and placed alongside the tarballs
+without affecting them. Placement in the study folder guarantees the
+same access permissions (authenticated https). The datasets are
+generated based on file metadata -- the TAR archive remains the only
+data source -- so storage overhead is minimal.
+
+Four scripts, executed in the given order, capture the archival
+process.
+
+- ``make_studyvisit_archive``
+- ``deposit_visit_metadata`` (optional)
+- ``deposit_visit_dataset`` (optional)
+- ``catalogify_studyvisit_from_meta`` (optional)
+
+Creation of the TAR file needs to be done by the ICF. The remaining
+three steps can be done by the ICF (with results deposited alongside
+the TAR file), or by the ICF users who can access the data (on their
+own infrastructure), and for this reason are marked as optional.
diff --git a/docs/source/index.rst b/docs/source/index.rst
index bee0fac..e349fcc 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -17,6 +17,7 @@ individuals.
 
    user/index
    reference
+   admin
    developer
 
 
diff --git a/docs/source/reference.rst b/docs/source/reference.rst
index c0e5094..6c88ff9 100644
--- a/docs/source/reference.rst
+++ b/docs/source/reference.rst
@@ -1,5 +1,5 @@
-Administrator docs
-==================
+Reference
+=========
 
 The INM-ICF Utilities `Github repository`_ provides a set of
 executable Python scripts which automate generation of deposits in the
@@ -11,11 +11,8 @@ dependencies are packaged as a `Singularity`_ v3 container
 .. _singularity: https://docs.sylabs.io/guides/main/user-guide/
 .. _download: https://ci.appveyor.com/api/projects/mih/inm-icf-utilities/artifacts/icf.sif
 
-Archive generation
-------------------
-
 Containerized execution
-^^^^^^^^^^^^^^^^^^^^^^^
+-----------------------
 
 With the Singilarity image, ``icf.sif``, all scripts are made directly
 available, either through ``singularity run``:
@@ -46,25 +43,11 @@ available under an ``icf-utils`` command.
 
    $ sudo install -t /usr/bin icf-utils
 
-Archival workflow
-^^^^^^^^^^^^^^^^^
-
-The main part of visit archival is the creation a TAR file.
-
-The DataLad dataset can be generated and placed alongside the tarballs
-without affecting them. Placement in the study folder guarantees the
-same access permissions (authenticated https). The datasets are
-generated based on file metadata -- the TAR archive remains the only
-data source -- so storage overhead is minimal.
-
-Four scripts, executed in the given order, capture the archival
-process.
-
 Script listing
-^^^^^^^^^^^^^^
+--------------
 
 ``make_studyvisit_archive``
-"""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This utility generates a TAR archive from a directory containing DICOM files.
 
@@ -87,7 +70,7 @@ for any non-DICOM file is set to the latest timestamp across all DICOM files.
    usage: make_studyvisit_archive [-h] [-o PATH] --id STUDY-ID VISIT-ID <input-dir>
 
 ``deposit_visit_metadata``
-""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This command locates the DICOM tarball for a particular visit in a
 study (given by their respective identifiers) in the data store, and
@@ -116,7 +99,7 @@ files, in JSON format, in the study directory:
   usage: deposit_visit_metadata [-h] [-o PATH] --id STUDY-ID VISIT-ID
 
 ``deposit_visit_dataset``
-"""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This command reads the metadata deposit from
 ``deposit_visit_metadata`` for a visit in a study (given by their
@@ -140,7 +123,7 @@ represents the actual dataset as a compressed archive.
    usage: deposit_visit_dataset [-h] --id STUDY-ID VISIT-ID [-o PATH] [--store-url URL]
 
 ``catalogify_studyvisit_from_meta``
-"""""""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This command creates or updates a DataLad catalog -- a user-facing
 html rendering of dataset contents. It is placed in the ``catalog``

From d3c80904def27f77378b08880a53ff5e9fa65b5b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 19:43:33 +0100
Subject: [PATCH 09/21] Break up reference into pages (container, scripts)

---
 docs/source/index.rst                         |  2 +-
 docs/source/reference/container.rst           | 40 +++++++++++++++++
 docs/source/reference/index.rst               | 19 ++++++++
 .../{reference.rst => reference/scripts.rst}  | 45 +------------------
 4 files changed, 61 insertions(+), 45 deletions(-)
 create mode 100644 docs/source/reference/container.rst
 create mode 100644 docs/source/reference/index.rst
 rename docs/source/{reference.rst => reference/scripts.rst} (71%)

diff --git a/docs/source/index.rst b/docs/source/index.rst
index e349fcc..5746e48 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -16,7 +16,7 @@ individuals.
    :caption: Contents:
 
    user/index
-   reference
+   reference/index
    admin
    developer
 
diff --git a/docs/source/reference/container.rst b/docs/source/reference/container.rst
new file mode 100644
index 0000000..4617734
--- /dev/null
+++ b/docs/source/reference/container.rst
@@ -0,0 +1,40 @@
+.. _container:
+
+Containerized execution
+-----------------------
+
+To simplify deployment, ICF utilities scripts and all their
+dependencies are packaged as a `Singularity`_ v3 container
+(`download`_).
+
+.. _singularity: https://docs.sylabs.io/guides/main/user-guide/
+.. _download: https://ci.appveyor.com/api/projects/mih/inm-icf-utilities/artifacts/icf.sif
+
+With the Singilarity image, ``icf.sif``, all scripts are made directly
+available, either through ``singularity run``:
+
+.. code-block:: console
+
+   $ singularity run <singularity options> icf.sif <script name> <script options>
+
+or by making the image file executable.
+
+The singularity image can also be installed as if it was a system
+package. For this, fill in the placeholders in the following script,
+and save it as ``icf-utils``:
+
+.. code-block:: sh
+
+   #!/bin/sh
+   set -e -u
+   singularity run -B <absolute-path-to-data> <absolute-path-to-icf.sif-file> "$@" > icf-utils
+
+The ``-B`` defines a bind path, making it accessible from within the
+container.
+
+Afterwards, install it under ``/usr/bin`` to make all functionality
+available under an ``icf-utils`` command.
+
+.. code-block::
+
+   $ sudo install -t /usr/bin icf-utils
diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst
new file mode 100644
index 0000000..185600f
--- /dev/null
+++ b/docs/source/reference/index.rst
@@ -0,0 +1,19 @@
+Reference
+=========
+
+The INM-ICF Utilities `Github repository`_ provides a set of
+executable Python scripts which automate generation of deposits in the
+ICF archive. To simplify deployment, these scripts and all their
+dependencies are packaged as a `Singularity`_ v3 container
+(`download`_).
+
+.. _github repository: https://github.com/psychoinformatics-de/inm-icf-utilities
+.. _singularity: https://docs.sylabs.io/guides/main/user-guide/
+.. _download: https://ci.appveyor.com/api/projects/mih/inm-icf-utilities/artifacts/icf.sif
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   container
+   scripts
diff --git a/docs/source/reference.rst b/docs/source/reference/scripts.rst
similarity index 71%
rename from docs/source/reference.rst
rename to docs/source/reference/scripts.rst
index 6c88ff9..0abff7a 100644
--- a/docs/source/reference.rst
+++ b/docs/source/reference/scripts.rst
@@ -1,47 +1,4 @@
-Reference
-=========
-
-The INM-ICF Utilities `Github repository`_ provides a set of
-executable Python scripts which automate generation of deposits in the
-ICF archive. To simplify deployment, these scripts and all their
-dependencies are packaged as a `Singularity`_ v3 container
-(`download`_).
-
-.. _github repository: https://github.com/psychoinformatics-de/inm-icf-utilities
-.. _singularity: https://docs.sylabs.io/guides/main/user-guide/
-.. _download: https://ci.appveyor.com/api/projects/mih/inm-icf-utilities/artifacts/icf.sif
-
-Containerized execution
------------------------
-
-With the Singilarity image, ``icf.sif``, all scripts are made directly
-available, either through ``singularity run``:
-
-.. code-block:: console
-
-   $ singularity run <singularity options> icf.sif <script name> <script options>
-
-or by making the image file executable.
-
-The singularity image can also be installed as if it was a system
-package. For this, fill in the placeholders in the following script,
-and save it as ``icf-utils``:
-
-.. code-block:: sh
-
-   #!/bin/sh
-   set -e -u
-   singularity run -B <absolute-path-to-data> <absolute-path-to-icf.sif-file> "$@" > icf-utils
-
-The ``-B`` defines a bind path, making it accessible from within the
-container.
-
-Afterwards, install it under ``/usr/bin`` to make all functionality
-available under an ``icf-utils`` command.
-
-.. code-block::
-
-   $ sudo install -t /usr/bin icf-utils
+.. _scripts:
 
 Script listing
 --------------

From ad1026dd6710900a431d1d24d984a76826ad27d0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 19:47:55 +0100
Subject: [PATCH 10/21] Tweak admin docs

Emphasise the DataLad part being optional, add link to scripts and
container pages for detailed information.
---
 docs/source/admin.rst | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/docs/source/admin.rst b/docs/source/admin.rst
index febff3e..922f7be 100644
--- a/docs/source/admin.rst
+++ b/docs/source/admin.rst
@@ -6,14 +6,15 @@ Archival workflow
 
 The main part of visit archival is the creation a TAR file.
 
-The DataLad dataset can be generated and placed alongside the tarballs
-without affecting them. Placement in the study folder guarantees the
-same access permissions (authenticated https). The datasets are
-generated based on file metadata -- the TAR archive remains the only
-data source -- so storage overhead is minimal.
+Optionally, the DataLad dataset can be generated and placed alongside
+the tarballs without affecting them. Placement in the study folder
+guarantees the same access permissions (authenticated https). The
+datasets are generated based on file metadata -- the TAR archive
+remains the only data source -- so storage overhead is minimal.
 
 Four scripts, executed in the given order, capture the archival
-process.
+process. See :ref:`scripts` for usage details and :ref:`container` for
+recommended deployment of the tools.
 
 - ``make_studyvisit_archive``
 - ``deposit_visit_metadata`` (optional)

From 17aed3579967be49bbaed0096f45e31d39111aa7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Fri, 15 Dec 2023 20:02:53 +0100
Subject: [PATCH 11/21] Tweak datalad download command

---
 docs/source/user/datalad-generate.rst | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index 035cbb7..4807cb0 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -27,13 +27,11 @@ paths.
 
    mkdir -p local_dicomstore/<project-ID>
 
-Download the visit tarball:
+Download the visit tarball, keeping the same relative path:
 
 .. code-block:: bash
 
-   cd local_dicomstore/<project-ID>
-   datalad download ...
-   cd ../..
+   datalad download "https://data.inm-icf.de/<project-ID>/<visit-ID>_dicom.tar <project-ID>/<visit-ID>_dicom.tar"
 
 For the following examples, the *absolute path* to the local dicom
 store will be represented by ``$STORE_DIR``:

From 27fb1651ebc7b3bfdd660e4fbe2affe59dfb3017 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Mon, 18 Dec 2023 18:01:36 +0100
Subject: [PATCH 12/21] docs: fix script name in reference

---
 docs/source/reference/scripts.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/source/reference/scripts.rst b/docs/source/reference/scripts.rst
index 0abff7a..c0b04c9 100644
--- a/docs/source/reference/scripts.rst
+++ b/docs/source/reference/scripts.rst
@@ -88,5 +88,5 @@ folder in the study directory.
 
 .. code-block:: console
 
-  $ icf-utils dataladify_studyvisit_from_meta --help
+  $ icf-utils catalogify_studyvisit_from_meta --help
   usage: catalogify_studyvisit_from_meta [-h] [-o PATH] --id STUDY-ID VISIT-ID

From b793615586a64791118244d27ef8953c2abc89e4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Mon, 18 Dec 2023 19:07:26 +0100
Subject: [PATCH 13/21] docs: create dedicated page for datalad credentials

In the current docs organisation we need to refer to credentials in
two pages: generating and accessing datasets. It makes sense to
elevate the credentials info (including an admonition about DataLad
access being equivalent to https access) to a separate page which can
be referenced in 1-2 sentences.
---
 docs/source/user/datalad-access.rst      | 32 ++++--------------------
 docs/source/user/datalad-credentials.rst | 28 +++++++++++++++++++++
 docs/source/user/datalad-generate.rst    |  5 ++++
 docs/source/user/index.rst               |  1 +
 4 files changed, 39 insertions(+), 27 deletions(-)
 create mode 100644 docs/source/user/datalad-credentials.rst

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index f03343d..56491da 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -3,8 +3,8 @@
 Access data with DataLad
 ------------------------
 
-Software requirements
-^^^^^^^^^^^^^^^^^^^^^
+Requirements
+^^^^^^^^^^^^
 
 Accessing the ICF store requires `DataLad`_ with `Datalad-Next`_
 extension installed.
@@ -17,31 +17,9 @@ system in the `DataLad Handbook`_.
 .. _datalad handbook: https://handbook.datalad.org/intro/installation.html
 .. _pip: https://pip.pypa.io/en/stable/
 
-Credentials
-^^^^^^^^^^^
-
-The ICF store is not publicly available, and ICF administrators will provide user names and passwords on a per-study basis.
-DataLad will store or retrieve these credentials using your
-operating system's keyring service. In general, the first time you use
-DataLad to access a project directory, you will be prompted for your
-credentials. If content retrieval succeeds, the credential will be
-saved, and reused the next time you access a URL from the same realm.
-
-If you have access to multiple projects, you can have different sets
-of credentials. You can use the `datalad credentials`_ command from
-DataLad Next to manage (e.g. query, set or remove) credentials known
-to DataLad.
-
-.. admonition:: DataLad usage in the context of GDPR
-
-   DataLad is a client-side software. Usage of DataLad with ICF store
-   is technically equivalent to downloading tar archives with ``wget``
-   or with a web browser click-to-download: in either case, data
-   access happens over https, and the authorisation is performed by
-   the ICF server, not by the clients.
-
-.. _datalad credentials: http://docs.datalad.org/projects/next/en/latest/generated/man/datalad-credentials.html
-
+Obtaining data hosted in the ICF store requires access credentials for
+a given study, issued by the ICF. DataLad acts only as a client
+software. See :ref:`dl-credentials` for details.
 
 Clone & get
 ^^^^^^^^^^^
diff --git a/docs/source/user/datalad-credentials.rst b/docs/source/user/datalad-credentials.rst
new file mode 100644
index 0000000..ab16e6c
--- /dev/null
+++ b/docs/source/user/datalad-credentials.rst
@@ -0,0 +1,28 @@
+.. _dl-credentials:
+
+Manage DataLad credentials
+--------------------------
+
+The ICF store is not publicly available, and ICF administrators will
+provide user names and passwords on a per-study basis.  DataLad will
+store or retrieve these credentials using your operating system's
+keyring service. In general, the first time you use DataLad to access
+a project directory, you will be prompted for your credentials. If
+content retrieval succeeds, you will have a possibility of saving the
+credential, to be reused the next time you access a URL from the same
+realm.
+
+If you have access to multiple projects, you can have different sets
+of credentials. You can use the `datalad credentials`_ command from
+DataLad Next to manage (e.g. query, set or remove) credentials known
+to DataLad.
+
+.. admonition:: DataLad usage in the context of GDPR
+
+   DataLad is a client-side software. Usage of DataLad with ICF store
+   is technically equivalent to downloading tar archives with ``wget``
+   or with a web browser click-to-download: in either case, data
+   access happens over https, and the authorisation is performed by
+   the ICF server, not by the clients.
+
+.. _datalad credentials: http://docs.datalad.org/projects/next/en/latest/generated/man/datalad-credentials.html
diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index 4807cb0..6c67ad4 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -33,6 +33,11 @@ Download the visit tarball, keeping the same relative path:
 
    datalad download "https://data.inm-icf.de/<project-ID>/<visit-ID>_dicom.tar <project-ID>/<visit-ID>_dicom.tar"
 
+Using ``datalad download`` for downloading the file has the benefit of
+using DataLad's credential management. If this is the first time you
+use DataLad to access the project directory, you will be asked to
+provide your ICF credentials. See :ref:`dl-credentials` for details.
+
 For the following examples, the *absolute path* to the local dicom
 store will be represented by ``$STORE_DIR``:
 
diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst
index 3625e02..03b0a42 100644
--- a/docs/source/user/index.rst
+++ b/docs/source/user/index.rst
@@ -15,6 +15,7 @@ Please contact `ICF personnel`_ to get access and for any authentication-related
    :caption: Contents:
 
    browser
+   datalad-credentials
    datalad-generate
    datalad-access
    datalad-advanced

From 8eef757903516f6a5a7103e57ebf96bad1c51f2d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Mon, 18 Dec 2023 20:14:51 +0100
Subject: [PATCH 14/21] docs: update section on generating DataLad datasets

Adds software requirements and catalog generation, and a few lesser
changes.
---
 docs/source/user/datalad-generate.rst | 58 +++++++++++++++++++++++++--
 1 file changed, 54 insertions(+), 4 deletions(-)

diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index 6c67ad4..a2e30ce 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -11,9 +11,26 @@ for DataLad-based access on institute-local infrastructure.
 In principle, such datasets are *lightweight*, meaning that they only
 index the content that can be retrieved from the ICF archive (all
 access restrictions apply). Using DataLad can simplify local access,
-allow raw data versioning, and enable logical transformations of the
-DICOM folder structure - see :ref:`dl-advanced` for examples of the
-latter.
+allow raw data versioning, integrate with existing workflows, and
+enable logical transformations of the DICOM folder structure - see
+:ref:`dl-advanced` for examples of the latter.
+
+Software requirements
+^^^^^^^^^^^^^^^^^^^^^
+
+Accessing the ICF store requires `DataLad`_ with `Datalad-Next`_
+extension installed.  You can find instructions for installing DataLad
+on your operating system in the `DataLad Handbook`_.  `Datalad-Next`_
+can be installed with `pip`_ [1]_.
+
+.. _datalad: https://www.datalad.org/
+.. _datalad-next: https://docs.datalad.org/projects/next
+.. _datalad handbook: https://handbook.datalad.org/intro/installation.html
+.. _pip: https://pip.pypa.io/en/stable/
+
+Dataset generation workflow (all steps after the initial download)
+uses the INM-ICF tools, which are packaged as a Singularity container;
+see :ref:`container`.
 
 Obtain the tarball
 ^^^^^^^^^^^^^^^^^^
@@ -86,6 +103,23 @@ DataLad dataset, and contain the information necessary to retrieve the
 data content with DataLad (but do not contain the data content
 itself).
 
+Create a catalog view (optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A catalog page (html+JS rendering of dataset contents generated with
+`DataLad catalog`_) can be created for the visit dataset. This is
+mostly useful when providing (internal) https access to the datasets.
+
+The following command will create the catalog (or update its content)
+and place it in the ``catalog`` folder in the study directory.
+
+.. _DataLad catalog: https://docs.datalad.org/projects/catalog
+
+.. code-block:: bash
+
+   singularity run -B $STORE_DIR icf.sif catalogify_studyvisit_from_meta \
+     --store-dir $STORE_DIR --id <project-ID> <visit ID>
+
 Remove the tarball
 ^^^^^^^^^^^^^^^^^^
 
@@ -95,5 +129,21 @@ Finally, the DICOM tarball can be safely removed.
 
    rm local_dicomstore/<project-ID>/<visit ID>_dicom.tar
 
+Metadata files can be removed, too, leaving only the dataset
+representation in ``*XDLRA*`` files.
+
+.. code-block:: bash
+
+   rm local_dicomstore/<project-ID>/<visit ID>_metadata_*.json
+
+
 The local dicom store can be used as a DataLad entry point for
-obtaining the dicom files.
+obtaining the dicom files, with ICF store serving as the data source
+for dataset clones; see :ref:`dl-access`.
+
+.. rubric:: Footnotes
+
+.. [1] To install software with pip, run a call such as the one below
+       in your favourite `virtual environment <https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/>`_::
+
+         python -m pip install datalad-next

From c60429c50eb8b6c134cf308f96daf70f09216617 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Mon, 18 Dec 2023 20:31:19 +0100
Subject: [PATCH 15/21] docs: increase cohesion, adjust catalog mentions

This unifies some spelling (local_dicom_store, uppercase DICOM), and
adds or tweaks some opening/closing paragraphs.

Catalog mention for browser-based access gets prefixed with an "if",
because it is unlikely that it will be generated for the ICF
datasets. I leave the paragraph nonetheless, to signal the
possibility.

Catalog mention for datalad based access is removed, because the
catalogify script hardcodes the (clone) URL to the ICF store URL, and
we are now assuming that the catalog generation happens on local
infrastructure.
---
 docs/source/user/browser.rst          |  4 ++--
 docs/source/user/datalad-access.rst   | 18 ++++++++----------
 docs/source/user/datalad-generate.rst | 16 ++++++++--------
 3 files changed, 18 insertions(+), 20 deletions(-)

diff --git a/docs/source/user/browser.rst b/docs/source/user/browser.rst
index c776c59..dd40b42 100644
--- a/docs/source/user/browser.rst
+++ b/docs/source/user/browser.rst
@@ -24,10 +24,10 @@ following:
 Catalog-based browsing
 ======================
 
-By entering the ``datalad_catalog`` directory, users will be able to
+If a catalog has been generated for a given study, users will be able to
 browse through the directory tree with additional annotations
 of available metadata, and search for acquisitions based on keywords
-or name.
+or name, by entering the ``datalad_catalog`` directory.
 
 Downloads
 =========
diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index 56491da..318d568 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -3,6 +3,11 @@
 Access data with DataLad
 ------------------------
 
+This section describes accessing the ICF data by cloning DataLad
+datasets which have already been created and made available, most
+likely on local infrastructure. Dataset generation is described in
+the previous section, :ref:`dl-generate`.
+
 Requirements
 ^^^^^^^^^^^^
 
@@ -24,9 +29,9 @@ software. See :ref:`dl-credentials` for details.
 Clone & get
 ^^^^^^^^^^^
 
-If a visit dataset has been prepared (procedure described in
-:ref:`dl-generate`) and saved in an accessible location, it can be
-cloned with DataLad from a URL containing the following components:
+If a visit dataset has been prepared and placed in an accessible
+location, it can be cloned with DataLad from a URL containing the
+following components:
 
 * a set of configuration parameters, always constant
 * store base URL (e.g., ``file:///data/group/groupname/local_dicom_store``) [2]_
@@ -62,13 +67,6 @@ get``. DataLad will handle download and unpacking of the tar file.
 Take a look at the section :ref:`dl-advanced` to learn about useful
 convenience features DataLad adds on top of this.
 
-Catalog-based clone URLs
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Instead of crafting clone URLs by hand, the ``datalad_catalog``
-directory in the data store displays a copy-paste URL for cloning when
-clicking the "Download with DataLad" button on each individual visit ID.
-
 
 .. rubric:: Footnotes
 
diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index a2e30ce..e1f7822 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -42,7 +42,7 @@ paths.
 
 .. code-block:: bash
 
-   mkdir -p local_dicomstore/<project-ID>
+   mkdir -p local_dicom_store/<project-ID>
 
 Download the visit tarball, keeping the same relative path:
 
@@ -55,12 +55,12 @@ using DataLad's credential management. If this is the first time you
 use DataLad to access the project directory, you will be asked to
 provide your ICF credentials. See :ref:`dl-credentials` for details.
 
-For the following examples, the *absolute path* to the local dicom
+For the following examples, the *absolute path* to the local DICOM
 store will be represented by ``$STORE_DIR``:
 
 .. code-block:: bash
 
-   export STORE_DIR=$PWD/local_dicomstore
+   export STORE_DIR=$PWD/local_dicom_store
 
 
 Deposit visit metadata alongside tarball
@@ -127,19 +127,19 @@ Finally, the DICOM tarball can be safely removed.
 
 .. code-block:: bash
 
-   rm local_dicomstore/<project-ID>/<visit ID>_dicom.tar
+   rm $STORE_DIR/<project-ID>/<visit ID>_dicom.tar
 
 Metadata files can be removed, too, leaving only the dataset
 representation in ``*XDLRA*`` files.
 
 .. code-block:: bash
 
-   rm local_dicomstore/<project-ID>/<visit ID>_metadata_*.json
+   rm $STORE_DIR/<project-ID>/<visit ID>_metadata_*.json
 
 
-The local dicom store can be used as a DataLad entry point for
-obtaining the dicom files, with ICF store serving as the data source
-for dataset clones; see :ref:`dl-access`.
+The local store can be used as a DataLad entry point for obtaining the
+DICOM files from the ICF store (which would serve as the data source
+for dataset clones); see :ref:`dl-access`.
 
 .. rubric:: Footnotes
 

From 61451854150c5d67c0533680462f4a921f9624b3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Tue, 19 Dec 2023 12:00:07 +0100
Subject: [PATCH 16/21] docs: be more precise about requirements

If datasets generated with ICF tooling are on local infrastructure,
then cloning datasets and accessing ICF are two separate things, but
both require DataLad-Next.
---
 docs/source/user/datalad-access.rst | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index 318d568..46cd7cc 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -11,11 +11,11 @@ the previous section, :ref:`dl-generate`.
 Requirements
 ^^^^^^^^^^^^
 
-Accessing the ICF store requires `DataLad`_ with `Datalad-Next`_
-extension installed.
+Cloning datasets generated with the ICF tooling and accessing the ICF
+store requires `DataLad`_ with `Datalad-Next`_ extension installed.
 You can find instructions for installing DataLad on your operating
-system in the `DataLad Handbook`_.
-`Datalad-Next`_ can be installed with `pip`_ [1]_.
+system in the `DataLad Handbook`_.  `Datalad-Next`_ can be installed
+with `pip`_ [1]_.
 
 .. _datalad: https://www.datalad.org/
 .. _datalad-next: https://docs.datalad.org/projects/next

From 7bebdd2a6fbc7a71efb6506fb92531e79f9b4739 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Tue, 19 Dec 2023 12:37:22 +0100
Subject: [PATCH 17/21] docs: refactor datalad requirements

The DataLad access requirements are moved to a separate section, which
is referenced in short paragraph by dataset generation and access
sections.
---
 docs/source/user/datalad-access.rst       | 31 +++++---------------
 docs/source/user/datalad-generate.rst     | 28 ++++--------------
 docs/source/user/datalad-requirements.rst | 35 +++++++++++++++++++++++
 docs/source/user/index.rst                |  1 +
 4 files changed, 48 insertions(+), 47 deletions(-)
 create mode 100644 docs/source/user/datalad-requirements.rst

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index 46cd7cc..2072185 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -8,23 +8,11 @@ datasets which have already been created and made available, most
 likely on local infrastructure. Dataset generation is described in
 the previous section, :ref:`dl-generate`.
 
-Requirements
-^^^^^^^^^^^^
-
-Cloning datasets generated with the ICF tooling and accessing the ICF
-store requires `DataLad`_ with `Datalad-Next`_ extension installed.
-You can find instructions for installing DataLad on your operating
-system in the `DataLad Handbook`_.  `Datalad-Next`_ can be installed
-with `pip`_ [1]_.
-
-.. _datalad: https://www.datalad.org/
-.. _datalad-next: https://docs.datalad.org/projects/next
-.. _datalad handbook: https://handbook.datalad.org/intro/installation.html
-.. _pip: https://pip.pypa.io/en/stable/
-
-Obtaining data hosted in the ICF store requires access credentials for
-a given study, issued by the ICF. DataLad acts only as a client
-software. See :ref:`dl-credentials` for details.
+This workflow uses DataLad with DataLad-Next extension (see
+:ref:`dl-requirements`). DataLad datasets index data in their original
+(ICF) location. Obtaining data hosted in the ICF store requires access
+credentials for a given study, issued by the ICF. DataLad acts only as
+a client software. See :ref:`dl-credentials` for details.
 
 Clone & get
 ^^^^^^^^^^^
@@ -34,7 +22,7 @@ location, it can be cloned with DataLad from a URL containing the
 following components:
 
 * a set of configuration parameters, always constant
-* store base URL (e.g., ``file:///data/group/groupname/local_dicom_store``) [2]_
+* store base URL (e.g., ``file:///data/group/groupname/local_dicom_store``) [1]_
 * study ID (e.g., ``my-study``)
 * visit ID (e.g., ``P000123``)
 * a file name suffix / template, ``_annex{{annex_key}}`` (verbatim), always constant
@@ -70,12 +58,7 @@ convenience features DataLad adds on top of this.
 
 .. rubric:: Footnotes
 
-.. [1] To install software with pip, run a call such as the one below
-       in your favourite `virtual environment <https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/>`_::
-
-         python -m pip install datalad-next
-
-.. [2] Examples use ``file://`` URLs, given that the datasets are most
+.. [1] Examples use ``file://`` URLs, given that the datasets are most
        likely to be generated on institute-local infrastructure. Other
        protocoles (e.g. ``https://`` or ``ssh://``) can be substituted
        depending on the particular setup, without affecting the URL
diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index e1f7822..ae27e5d 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -15,22 +15,11 @@ allow raw data versioning, integrate with existing workflows, and
 enable logical transformations of the DICOM folder structure - see
 :ref:`dl-advanced` for examples of the latter.
 
-Software requirements
-^^^^^^^^^^^^^^^^^^^^^
-
-Accessing the ICF store requires `DataLad`_ with `Datalad-Next`_
-extension installed.  You can find instructions for installing DataLad
-on your operating system in the `DataLad Handbook`_.  `Datalad-Next`_
-can be installed with `pip`_ [1]_.
-
-.. _datalad: https://www.datalad.org/
-.. _datalad-next: https://docs.datalad.org/projects/next
-.. _datalad handbook: https://handbook.datalad.org/intro/installation.html
-.. _pip: https://pip.pypa.io/en/stable/
-
-Dataset generation workflow (all steps after the initial download)
-uses the INM-ICF tools, which are packaged as a Singularity container;
-see :ref:`container`.
+The workflow described below uses DataLad with DataLad-Next extension
+for initial DICOM download and the INM-ICF tools packaged as a
+Singularity container for subsequent steps (see
+:ref:`dl-requirements`). ICF access credentials are required (see
+:ref:`dl-credentials`).
 
 Obtain the tarball
 ^^^^^^^^^^^^^^^^^^
@@ -140,10 +129,3 @@ representation in ``*XDLRA*`` files.
 The local store can be used as a DataLad entry point for obtaining the
 DICOM files from the ICF store (which would serve as the data source
 for dataset clones); see :ref:`dl-access`.
-
-.. rubric:: Footnotes
-
-.. [1] To install software with pip, run a call such as the one below
-       in your favourite `virtual environment <https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/>`_::
-
-         python -m pip install datalad-next
diff --git a/docs/source/user/datalad-requirements.rst b/docs/source/user/datalad-requirements.rst
new file mode 100644
index 0000000..493ee1e
--- /dev/null
+++ b/docs/source/user/datalad-requirements.rst
@@ -0,0 +1,35 @@
+.. _dl-requirements:
+
+DataLad requirements
+--------------------
+
+Accessing the ICF store contents and cloning datasets generated with
+the ICF tooling requires `DataLad`_ with `Datalad-Next`_ extension
+installed.  You can find instructions for installing DataLad on your
+operating system in the `DataLad Handbook`_.  `Datalad-Next`_ can be
+installed with `pip`_ [1]_.
+
+Generating DataLad datasets based on the DICOMS in the ICF store
+additionally requires the INM-ICF tools, which are packaged as a
+`Singularity`_ container; see :ref:`container`. The tools are not
+required for accessing already existing DataLad datasets.
+
+Obtaining data hosted in the ICF store requires access credentials for
+a given study, issued by the ICF. DataLad acts only as a client
+software. See :ref:`dl-credentials` for details.
+
+.. rubric:: Footnotes
+
+.. [1] To install software with pip, run a call such as the one below
+       in your favourite `virtual environment`_:
+
+       .. code-block:: bash
+
+          python -m pip install datalad-next
+
+.. _datalad: https://www.datalad.org/
+.. _datalad-next: https://docs.datalad.org/projects/next
+.. _datalad handbook: https://handbook.datalad.org/intro/installation.html
+.. _pip: https://pip.pypa.io/en/stable/
+.. _virtual environment: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/
+.. _singularity: https://docs.sylabs.io/guides/main/user-guide/
diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst
index 03b0a42..fadaf52 100644
--- a/docs/source/user/index.rst
+++ b/docs/source/user/index.rst
@@ -15,6 +15,7 @@ Please contact `ICF personnel`_ to get access and for any authentication-related
    :caption: Contents:
 
    browser
+   datalad-requirements
    datalad-credentials
    datalad-generate
    datalad-access

From 2d2e21ef4897db7f9e76a5742ba37d657b86a7ea Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Wed, 10 Jan 2024 12:40:19 +0100
Subject: [PATCH 18/21] Apply suggestions from code review

Co-authored-by: Adina Wagner <adina.wagner@t-online.de>
---
 docs/source/user/datalad-access.rst   | 6 ++++++
 docs/source/user/datalad-generate.rst | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index 2072185..c9ded59 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -37,6 +37,12 @@ Given the exemplary values above, the pattern would expand to
 
     'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///data/group/groupname/local_dicom_store/my-study/P000123_{{annex_key}}'
 
+A full ``datalad clone`` command could then look like this:
+
+.. code-block::
+    datalad clone 'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///tmp/local_dicom_store/dl-Z03/P000624_{{annex_key}}'  my_clone
+    
+
 .. note:: The URL is arguably a bit clunky. A convenience short cut can be provided via configuration item ``datalad.clone.url-substitute.<label>`` and a substitution rule based on regular expressions. For example, clone URLs can be shortened to require only an identifier (here, ``file:///data/group/groupname/local_dicom_store``), study ID, and visit ID (``inm-icf/<study-ID>/<visit-ID>``) with the following configuration:
 
    .. code-block::
diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index ae27e5d..8c83f8a 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -37,7 +37,7 @@ Download the visit tarball, keeping the same relative path:
 
 .. code-block:: bash
 
-   datalad download "https://data.inm-icf.de/<project-ID>/<visit-ID>_dicom.tar <project-ID>/<visit-ID>_dicom.tar"
+   datalad download "https://data.inm-icf.de/<project-ID>/<visit-ID>_dicom.tar  local_dicom_store/<project-ID>/<visit-ID>_dicom.tar"
 
 Using ``datalad download`` for downloading the file has the benefit of
 using DataLad's credential management. If this is the first time you

From 3e8f484606dcb8c4196ef2bb53548261ca3387c3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Wed, 10 Jan 2024 14:28:41 +0100
Subject: [PATCH 19/21] Apply further suggestions from code review

Apart from small formatting tweaks, this commit adds the following
changes to the dataladification workflow description:

- clarify dataset creation: the workflow creates dataset
  representations that can be cloned, not checked-out datasets
- link to the singularity container description before using
  singularity run for the first time
- add missing argument placeholders
- mention that the catalog needs to be served

Suggested-by: Adina Wagner <adina.wagner@t-online.de>
---
 docs/source/user/datalad-access.rst   |  3 ++-
 docs/source/user/datalad-generate.rst | 37 +++++++++++++++++----------
 2 files changed, 26 insertions(+), 14 deletions(-)

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index c9ded59..e571047 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -31,7 +31,7 @@ The pattern for the URL is::
 
     'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=<store base URL>/<study ID>/<visit ID>_{{annex_key}}'
 
-Given the exemplary values above, the pattern would expand to
+Given the exemplary values above, the pattern would expand to:
 
 .. code-block::
 
@@ -40,6 +40,7 @@ Given the exemplary values above, the pattern would expand to
 A full ``datalad clone`` command could then look like this:
 
 .. code-block::
+
     datalad clone 'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///tmp/local_dicom_store/dl-Z03/P000624_{{annex_key}}'  my_clone
     
 
diff --git a/docs/source/user/datalad-generate.rst b/docs/source/user/datalad-generate.rst
index 8c83f8a..a56661d 100644
--- a/docs/source/user/datalad-generate.rst
+++ b/docs/source/user/datalad-generate.rst
@@ -37,21 +37,27 @@ Download the visit tarball, keeping the same relative path:
 
 .. code-block:: bash
 
-   datalad download "https://data.inm-icf.de/<project-ID>/<visit-ID>_dicom.tar  local_dicom_store/<project-ID>/<visit-ID>_dicom.tar"
+   datalad download "https://data.inm-icf.de/<project-ID>/<visit-ID>_dicom.tar local_dicom_store/<project-ID>/<visit-ID>_dicom.tar"
+
+The local copy of the tarball is required to index its contents. It
+can be removed afterwards -- datasets will use the ICF store as the
+content source.
 
 Using ``datalad download`` for downloading the file has the benefit of
 using DataLad's credential management. If this is the first time you
 use DataLad to access the project directory, you will be asked to
 provide your ICF credentials. See :ref:`dl-credentials` for details.
 
-For the following examples, the *absolute path* to the local DICOM
-store will be represented by ``$STORE_DIR``:
+For the following steps, the ICF utility scripts packaged as a
+Singularity container will be used, and executed with ``singularity
+run`` (see :ref:`container` for download and usage details). The
+*absolute path* to the local DICOM store will be represented by
+``$STORE_DIR``:
 
 .. code-block:: bash
 
    export STORE_DIR=$PWD/local_dicom_store
 
-
 Deposit visit metadata alongside tarball
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -70,20 +76,21 @@ within the tarball (relative path, MD5 checksum, size, and a small
 subset of DICOM headers describing acquisition type), and the latter
 describes the tarball itself.
 
-Deposit dataset alongside tarball
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Deposit dataset representation alongside tarball
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-DataLad dataset is created based on the metadata extracted in the
-previous step.  Additionally, you need to provide the base URL of the
-ICF store, ``<ICF STORE URL>`` (this base URL should not contain study
-or visit ID). The URL, combined with respective IDs, will be
-registered in the dataset as the source of the DICOM tarball, and used
-for retrieval by dataset clones.
+The next step is to create a lightweight, clone-able representation of
+a dataset in the local dataset store. This step relies on the metadata
+extracted with the previous command. Additionally, the base URL of the
+ICF store needs to be provided (here represented by ``<ICF STORE
+URL>``, this base URL should not contain study or visit ID). The URL,
+combined with respective IDs, will be registered in the dataset as the
+source of the DICOM tarball, and used for retrieval by dataset clones.
 
 .. code-block:: bash
 
    singularity run -B $STORE_DIR icf.sif deposit_visit_dataset \
-     --store-dir $STORE_DIR --store-url <ICF STORE URL>
+     --store-dir $STORE_DIR --store-url <ICF STORE URL> --id <project-ID> <visit ID>
 
 This will produce two files, ``<visit ID>_XDLA--refs`` and ``<visit
 ID>_XDLA--repo-export`` (text file and zip archive
@@ -109,6 +116,10 @@ and place it in the ``catalog`` folder in the study directory.
    singularity run -B $STORE_DIR icf.sif catalogify_studyvisit_from_meta \
      --store-dir $STORE_DIR --id <project-ID> <visit ID>
 
+This catalog needs to be subsequently served; a simple (possibly
+local) http server is enough. See the generated README file in the
+``catalog`` folder for details.
+
 Remove the tarball
 ^^^^^^^^^^^^^^^^^^
 

From 9d1a83962a296c9299d337c3e86cd5709d2b0605 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Wed, 10 Jan 2024 16:25:31 +0100
Subject: [PATCH 20/21] Apply one more suggestion from code review

This adds a note to the datalad access documentation, advising users
that cloning from an incorrect datalad-annex URL will not cause an
error, but instead produce an empty repository.

Suggested-by: Adina Wagner <adina.wagner@t-online.de>
---
 docs/source/user/datalad-access.rst | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index e571047..4c01d71 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -42,7 +42,19 @@ A full ``datalad clone`` command could then look like this:
 .. code-block::
 
     datalad clone 'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///tmp/local_dicom_store/dl-Z03/P000624_{{annex_key}}'  my_clone
-    
+
+.. note::
+
+   The clone command will not fail if the ``datalad-annex::`` URL
+   points to a nonexisting target. If you see the following warning:
+
+   .. code-block:: none
+
+      [WARNING] You appear to have cloned an empty repository.
+      [WARNING] Cloned /path/to/my_clone but could not find a branch with commits
+
+   it is likely that the provided URL is mistyped or otherwise not correct.
+
 
 .. note:: The URL is arguably a bit clunky. A convenience short cut can be provided via configuration item ``datalad.clone.url-substitute.<label>`` and a substitution rule based on regular expressions. For example, clone URLs can be shortened to require only an identifier (here, ``file:///data/group/groupname/local_dicom_store``), study ID, and visit ID (``inm-icf/<study-ID>/<visit-ID>``) with the following configuration:
 
@@ -54,6 +66,7 @@ A full ``datalad clone`` command could then look like this:
    You are free to adjust this configuration custom to your needs and preferences.
    Further documentation on it can be found in the `DataLad Docs`_.
 
+
 .. _DataLad Docs: http://docs.datalad.org/en/stable/design/url_substitution.html
 
 Cloning will retrieve a lightweight dataset, which does not (yet)

From dd028cc1d8cc9b4a12470ce62e4064eca7a3d55d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20Szczepanik?= <m.szczepanik@fz-juelich.de>
Date: Wed, 10 Jan 2024 18:34:05 +0100
Subject: [PATCH 21/21] docs: tweak example

Makes example identifiers consistent with previous examples.
---
 docs/source/user/datalad-access.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/source/user/datalad-access.rst b/docs/source/user/datalad-access.rst
index 4c01d71..5a89022 100644
--- a/docs/source/user/datalad-access.rst
+++ b/docs/source/user/datalad-access.rst
@@ -41,7 +41,7 @@ A full ``datalad clone`` command could then look like this:
 
 .. code-block::
 
-    datalad clone 'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///tmp/local_dicom_store/dl-Z03/P000624_{{annex_key}}'  my_clone
+    datalad clone 'datalad-annex::?type=external&externaltype=uncurl&encryption=none&url=file:///tmp/local_dicom_store/my-study/P000123_{{annex_key}}'  my_clone
 
 .. note::