Skip to content

Commit 014f95e

Browse files
nischildnischild
authored
Merge branch 'openPMD:dev' into dev
2 parents 6e553a3 + 1c7d7ff commit 014f95e

File tree

150 files changed

+8736
-3466
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

150 files changed

+8736
-3466
lines changed

‎.circleci/config.yml

+1
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ jobs:
1919
sudo .github/workflows/dependencies/install_spack
2020
python3 -m pip install -U pip
2121
python3 -m pip install -U packaging setuptools wheel
22+
python3 -m pip install -U six
2223
python3 -m pip install -U numpy
2324
python3 -m pip install -U mpi4py
2425
python3 -m pip install -U pandas

‎.github/workflows/clang-format/clang-format.sh

+2-2
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22

33
if (( $# > 0 )); then
44
# received arguments, format those files
5-
clang-format-13 -i "$@"
5+
clang-format-18 -i "$@"
66
else
77
# received no arguments, find files on our own
88
find include/ src/ test/ examples/ \
99
-regextype egrep \
1010
-type f -regex '.*\.(hpp|cpp|hpp\.in)$' \
11-
| xargs clang-format-13 -i
11+
| xargs clang-format-18 -i
1212
fi

‎.github/workflows/linux.yml

+30-10
Original file line numberDiff line numberDiff line change
@@ -128,26 +128,38 @@ jobs:
128128
- name: Install
129129
run: |
130130
sudo apt update
131-
sudo apt install clang-14 cmake gfortran libhdf5-dev python3.11 python3.11-dev wget
132-
wget https://bootstrap.pypa.io/get-pip.py
133-
python3.11 get-pip.py
134-
python3.11 -m pip install numpy pandas
131+
sudo apt install clang-14 cmake gfortran libhdf5-dev python3.11 python3.11-dev wget python3.11-venv
135132
sudo .github/workflows/dependencies/install_spack
133+
python3.11 -m venv /opt/python_venv
134+
. /opt/python_venv/bin/activate
135+
python -m pip install numpy pandas
136+
git clone -b v4.0.3 https://github.com/ToruNiina/toml11
137+
cmake -S toml11 -B build_toml11 \
138+
-DCMAKE_INSTALL_PREFIX=toml11_install \
139+
-DCMAKE_CXX_STANDARD_REQUIRED=OFF \
140+
-DCMAKE_CXX_STANDARD=11
141+
cmake --build build_toml11 -j 2 --target install
136142
- name: Build
137143
env: {CC: clang-14, CXX: clang++-14, CXXFLAGS: -Werror}
138144
run: |
145+
# Build Spack packages against system Python
146+
# and activate the virtual environment just for the openPMD build.
147+
# Spack does not play nice with venv.
139148
eval $(spack env activate --sh .github/ci/spack-envs/clang14_py311_nompi_h5_ad2/)
140149
spack install
150+
. /opt/python_venv/bin/activate
141151
142152
share/openPMD/download_samples.sh build
153+
export CMAKE_PREFIX_PATH="$(realpath toml11_install):$CMAKE_PREFIX_PATH"
143154
cmake -S . -B build \
144155
-DopenPMD_USE_PYTHON=ON \
145156
-DopenPMD_USE_MPI=OFF \
146157
-DopenPMD_USE_HDF5=ON \
147158
-DopenPMD_USE_ADIOS2=ON \
148-
-DopenPMD_USE_INVASIVE_TESTS=ON \
149-
-DCMAKE_VERBOSE_MAKEFILE=ON \
150-
-DPython_EXECUTABLE=$(which python3.11)
159+
-DopenPMD_USE_INVASIVE_TESTS=ON \
160+
-DopenPMD_USE_INTERNAL_TOML11=OFF \
161+
-DCMAKE_VERBOSE_MAKEFILE=ON \
162+
-DPython_EXECUTABLE="$(which python)"
151163
cmake --build build --parallel 2
152164
ctest --test-dir build --output-on-failure
153165
@@ -271,17 +283,22 @@ jobs:
271283
run: |
272284
apk update
273285
apk add hdf5-dev
274-
python3.10 -m pip install numpy
286+
# Use a virtual environment in order to avoid compatibility issues
287+
# between the various Python installations in this image.
288+
python3.10 -m venv /opt/python_env
289+
. /opt/python_env/bin/activate
290+
python -m pip install numpy
275291
- name: Build
276292
env: {CXXFLAGS: -Werror}
277293
run: |
294+
. /opt/python_env/bin/activate
278295
share/openPMD/download_samples.sh build
279296
cmake -S . -B build \
280297
-DopenPMD_USE_PYTHON=ON \
281298
-DopenPMD_USE_MPI=OFF \
282299
-DopenPMD_USE_HDF5=ON \
283300
-DopenPMD_USE_INVASIVE_TESTS=ON \
284-
-DPython_EXECUTABLE=$(which python3.10)
301+
-DPython_EXECUTABLE=$(which python)
285302
cmake --build build --parallel 2
286303
cd build
287304
ctest --output-on-failure
@@ -312,11 +329,14 @@ jobs:
312329
313330
share/openPMD/download_samples.sh build
314331
cmake -S . -B build \
332+
-DCMAKE_CXX_FLAGS="-Wno-error=stringop-overread" \
333+
-DCMAKE_C_FLAGS="-Wno-error=stringop-overread" \
315334
-DopenPMD_USE_PYTHON=ON \
316335
-DopenPMD_USE_MPI=ON \
317336
-DopenPMD_USE_HDF5=ON \
318337
-DopenPMD_USE_ADIOS2=ON \
319-
-DopenPMD_USE_INVASIVE_TESTS=ON
338+
-DopenPMD_USE_INVASIVE_TESTS=ON \
339+
-DMPIEXEC_EXECUTABLE=".github/workflows/mpirun_workaround.sh"
320340
cmake --build build --parallel 2
321341
cd build
322342
ctest --output-on-failure

‎.github/workflows/macos.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jobs:
1212
# appleclang10_py37_h5_ad2_libcpp
1313
# appleclang11_nopy_nompi_h5_ad2
1414

15-
appleclang14_py_mpi_h5_ad2:
15+
appleclang15_py_mpi_h5_ad2:
1616
runs-on: macos-latest
1717
if: github.event.pull_request.draft == false
1818
steps:
@@ -45,8 +45,8 @@ jobs:
4545
cmake --build build --parallel 3
4646
ctest --test-dir build --verbose
4747
48-
appleclang13_py:
49-
runs-on: macos-11
48+
appleclang14_py:
49+
runs-on: macos-12
5050
if: github.event.pull_request.draft == false
5151
steps:
5252
- uses: actions/checkout@v3

‎.github/workflows/mpirun_workaround.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@
1414
# This script provides a workaround by putting the called sub-command into
1515
# a script in a temporary file.
1616

17-
mpiexec -n 1 ls --all \
17+
ls="$(which ls)"
18+
mpiexec "$ls" -m \
1819
&& echo "MPIRUN WORKING AGAIN, PLEASE REMOVE WORKAROUND" >&2 \
1920
&& exit 1 \
2021
|| true

‎.github/workflows/source.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
runs-on: ubuntu-22.04
3737
steps:
3838
- uses: actions/checkout@v3
39-
- uses: s-weigand/setup-conda@v1.2.1
39+
- uses: s-weigand/setup-conda@v1.2.2
4040
with:
4141
update-conda: true
4242
conda-channels: conda-forge

‎.pre-commit-config.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ exclude: '^share/openPMD/thirdParty'
1818
# See https://pre-commit.com/hooks.html for more hooks
1919
repos:
2020
- repo: https://github.com/pre-commit/pre-commit-hooks
21-
rev: v4.5.0
21+
rev: v4.6.0
2222
hooks:
2323
- id: trailing-whitespace
2424
args: [--markdown-linebreak-ext=md]
@@ -49,7 +49,7 @@ repos:
4949

5050
# Changes tabs to spaces
5151
- repo: https://github.com/Lucas-C/pre-commit-hooks
52-
rev: v1.5.4
52+
rev: v1.5.5
5353
hooks:
5454
- id: remove-tabs
5555

@@ -66,7 +66,7 @@ repos:
6666
# clang-format v13
6767
# to run manually, use .github/workflows/clang-format/clang-format.sh
6868
- repo: https://github.com/pre-commit/mirrors-clang-format
69-
rev: v17.0.6
69+
rev: v18.1.5
7070
hooks:
7171
- id: clang-format
7272
# By default, the clang-format hook configures:

‎CHANGELOG.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Changes to "0.15.0"
1717
Features
1818
""""""""
1919

20-
- pybind11: require version 2.11.1+ #1220 #1322
20+
- pybind11: require version 2.12.0+ #1220 #1322 #1637
2121

2222
Bug Fixes
2323
"""""""""

‎CMakeLists.txt

+16-6
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,15 @@ if(openPMD_USE_INTERNAL_TOML11)
278278
add_subdirectory("${openPMD_SOURCE_DIR}/share/openPMD/thirdParty/toml11")
279279
message(STATUS "toml11: Using INTERNAL version '3.7.1'")
280280
else()
281-
find_package(toml11 3.7.1 CONFIG REQUIRED)
281+
# toml11 4.0 was a breaking change. This is reflected in the library's CMake
282+
# logic: version 4.0 is not accepted by a call to find_package(toml11 3.7).
283+
# Since we support both incompatible versions, we use two find_package()
284+
# calls. Search for version 4 first in order to prefer that
285+
# in (the unlikely) case that both versions are installed.
286+
find_package(toml11 4.0 CONFIG QUIET)
287+
if(NOT toml11_FOUND)
288+
find_package(toml11 3.7.1 CONFIG REQUIRED)
289+
endif()
282290
message(STATUS "toml11: Found version '${toml11_VERSION}'")
283291
endif()
284292
add_library(openPMD::thirdparty::toml11 INTERFACE IMPORTED)
@@ -408,9 +416,9 @@ if(openPMD_USE_PYTHON STREQUAL AUTO)
408416
if(openPMD_USE_INTERNAL_PYBIND11)
409417
add_subdirectory("${openPMD_SOURCE_DIR}/share/openPMD/thirdParty/pybind11")
410418
set(openPMD_HAVE_PYTHON TRUE)
411-
message(STATUS "pybind11: Using INTERNAL version 2.11.1")
419+
message(STATUS "pybind11: Using INTERNAL version 2.12.0")
412420
else()
413-
find_package(pybind11 2.11.1 CONFIG)
421+
find_package(pybind11 2.12.0 CONFIG)
414422
if(pybind11_FOUND)
415423
set(openPMD_HAVE_PYTHON TRUE)
416424
message(STATUS "pybind11: Found version '${pybind11_VERSION}'")
@@ -426,9 +434,9 @@ elseif(openPMD_USE_PYTHON)
426434
if(openPMD_USE_INTERNAL_PYBIND11)
427435
add_subdirectory("${openPMD_SOURCE_DIR}/share/openPMD/thirdParty/pybind11")
428436
set(openPMD_HAVE_PYTHON TRUE)
429-
message(STATUS "pybind11: Using INTERNAL version 2.11.1")
437+
message(STATUS "pybind11: Using INTERNAL version 2.12.0")
430438
else()
431-
find_package(pybind11 2.11.1 REQUIRED CONFIG)
439+
find_package(pybind11 2.12.0 REQUIRED CONFIG)
432440
set(openPMD_HAVE_PYTHON TRUE)
433441
message(STATUS "pybind11: Found version '${pybind11_VERSION}'")
434442
endif()
@@ -460,6 +468,7 @@ set(CORE_SOURCE
460468
src/auxiliary/Date.cpp
461469
src/auxiliary/Filesystem.cpp
462470
src/auxiliary/JSON.cpp
471+
src/auxiliary/Mpi.cpp
463472
src/backend/Attributable.cpp
464473
src/backend/BaseRecordComponent.cpp
465474
src/backend/MeshRecordComponent.cpp
@@ -482,6 +491,7 @@ set(IO_SOURCE
482491
src/IO/JSON/JSONIOHandlerImpl.cpp
483492
src/IO/JSON/JSONFilePosition.cpp
484493
src/IO/ADIOS/ADIOS2IOHandler.cpp
494+
src/IO/ADIOS/ADIOS2File.cpp
485495
src/IO/ADIOS/ADIOS2Auxiliary.cpp
486496
src/IO/InvalidatableFile.cpp)
487497

@@ -1369,7 +1379,7 @@ if(openPMD_BUILD_TESTING)
13691379
--outfile \
13701380
../samples/git-sample/thetaMode/data_%T.bp && \
13711381
\
1372-
${Python_EXECUTABLE} \
1382+
${MPI_TEST_EXE} ${Python_EXECUTABLE} \
13731383
${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe \
13741384
--infile ../samples/git-sample/thetaMode/data_%T.bp \
13751385
--outfile ../samples/git-sample/thetaMode/data%T.json \

‎NEWS.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Please transition to ADIOS2.
1313
For reading legacy ADIOS1 BP3 files, either use an older version of openPMD-api or the BP3 backend in ADIOS2.
1414
Note that ADIOS2 does not support compression in BP3 files.
1515

16-
pybind11 2.11.1 is now the minimally supported version for Python support.
16+
pybind11 2.12.0 is now the minimally supported version for Python support.
1717

1818

1919
0.15.0

‎README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ Required:
101101

102102
Shipped internally in `share/openPMD/thirdParty/`:
103103
* [Catch2](https://github.com/catchorg/Catch2) 2.13.10+ ([BSL-1.0](https://github.com/catchorg/Catch2/blob/master/LICENSE.txt))
104-
* [pybind11](https://github.com/pybind/pybind11) 2.11.1+ ([new BSD](https://github.com/pybind/pybind11/blob/master/LICENSE))
104+
* [pybind11](https://github.com/pybind/pybind11) 2.12.0+ ([new BSD](https://github.com/pybind/pybind11/blob/master/LICENSE))
105105
* [NLohmann-JSON](https://github.com/nlohmann/json) 3.9.1+ ([MIT](https://github.com/nlohmann/json/blob/develop/LICENSE.MIT))
106106
* [toml11](https://github.com/ToruNiina/toml11) 3.7.1+ ([MIT](https://github.com/ToruNiina/toml11/blob/master/LICENSE))
107107

@@ -116,7 +116,7 @@ while those can be built either with or without:
116116
Optional language bindings:
117117
* Python:
118118
* Python 3.8 - 3.12
119-
* pybind11 2.11.1+
119+
* pybind11 2.12.0+
120120
* numpy 1.15+
121121
* mpi4py 2.1+ (optional, for MPI)
122122
* pandas 1.0+ (optional, for dataframes)
@@ -267,7 +267,7 @@ The following options allow to switch to external installs:
267267
| CMake Option | Values | Library | Version |
268268
|---------------------------------|------------|---------------|----------|
269269
| `openPMD_USE_INTERNAL_CATCH` | **ON**/OFF | Catch2 | 2.13.10+ |
270-
| `openPMD_USE_INTERNAL_PYBIND11` | **ON**/OFF | pybind11 | 2.11.1+ |
270+
| `openPMD_USE_INTERNAL_PYBIND11` | **ON**/OFF | pybind11 | 2.12.0+ |
271271
| `openPMD_USE_INTERNAL_JSON` | **ON**/OFF | NLohmann-JSON | 3.9.1+ |
272272
| `openPMD_USE_INTERNAL_TOML11` | **ON**/OFF | toml11 | 3.7.1+ |
273273

‎docs/source/analysis/pandas.rst

+1-10
Original file line numberDiff line numberDiff line change
@@ -45,16 +45,7 @@ One can also combine all iterations in a single dataframe like this:
4545

4646
.. code-block:: python
4747
48-
import pandas as pd
49-
50-
df = pd.concat(
51-
(
52-
s.iterations[i].particles["electrons"].to_df().assign(iteration=i)
53-
for i in s.iterations
54-
),
55-
axis=0,
56-
ignore_index=True,
57-
)
48+
df = s.to_df("electrons")
5849
5950
# like before but with a new column "iteration" and all particles
6051
print(df)

‎docs/source/analysis/rapids.rst

+1-8
Original file line numberDiff line numberDiff line change
@@ -51,14 +51,7 @@ One can also combine all iterations in a single dataframe like this:
5151

5252
.. code-block:: python
5353
54-
cdf = cudf.concat(
55-
(
56-
cudf.from_pandas(s.iterations[i].particles["electrons"].to_df().assign(iteration=i))
57-
for i in s.iterations
58-
),
59-
axis=0,
60-
ignore_index=True,
61-
)
54+
cdf = s.to_cudf("electrons")
6255
6356
# like before but with a new column "iteration" and all particles
6457
print(cdf)

‎docs/source/backends/adios2.rst

+5
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,10 @@ Exceptions to this are the BP3 and SST engines which require their endings ``.bp
4848

4949
For file engines, we currently leverage the default ADIOS2 transport parameters, i.e. ``POSIX`` on Unix systems and ``FStream`` on Windows.
5050

51+
.. tip::
52+
53+
Use the ``adios2.engine.treat_unsupported_engine_as`` :ref:`JSON/TOML parameter <backendconfig-adios2>` for experimentally interacting with an unsupported ADIOS2 engine.
54+
5155
Steps
5256
-----
5357

@@ -81,6 +85,7 @@ environment variable default description
8185
``OPENPMD_ADIOS2_HAVE_METADATA_FILE`` ``1`` Online creation of the adios journal file (``1``: yes, ``0``: no).
8286
``OPENPMD_ADIOS2_NUM_SUBSTREAMS`` ``0`` Number of files to be created, 0 indicates maximum number possible.
8387
``OPENPMD_ADIOS2_ENGINE`` ``File`` `ADIOS2 engine <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_
88+
``OPENPMD_ADIOS2_PRETEND_ENGINE`` *empty* Pretend that an (unknown) ADIOS2 engine is in fact another one (also see the ``adios2.pretend_engine`` :ref:`parameter <backendconfig-adios2>`).
8489
``OPENPMD2_ADIOS2_USE_GROUP_TABLE`` ``0`` Use group table (see below)
8590
``OPENPMD_ADIOS2_STATS_LEVEL`` ``0`` whether to generate statistics for variables in ADIOS2. (``1``: yes, ``0``: no).
8691
``OPENPMD_ADIOS2_ASYNC_WRITE`` ``0`` ADIOS2 BP5 engine: 1 means setting "AsyncWrite" in ADIOS2 to "on". Flushes will go to the buffer by default (see ``preferred_flush_target``).

‎docs/source/backends/hdf5.rst

+14
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,15 @@ I/O Method
1515

1616
HDF5 internally either writes serially, via ``POSIX`` on Unix systems, or parallel to a single logical file via MPI-I/O.
1717

18+
Virtual File Drivers
19+
********************
20+
21+
Rudimentary support for HDF5 VFDs (`virtual file driver <https://www.hdfgroup.org/wp-content/uploads/2021/10/HDF5-VFD-Plugins-HUG.pdf>`_) is available (currently only the *subfiling* VFD).
22+
Note that the subfiling VFD needs to be enabled explicitly when configuring HDF5 and threaded MPI must be used.
23+
24+
Virtual file drivers are configured via JSON/TOML.
25+
Refer to the page on :ref:`JSON/TOML configuration <backendconfig-hdf5>` for further details.
26+
1827

1928
Backend-Specific Controls
2029
-------------------------
@@ -45,6 +54,10 @@ Although we choose the default to be non-collective (independent) for ease of us
4554
For independent parallel I/O, potentially prefer using a modern version of the MPICH implementation (especially, use ROMIO instead of OpenMPI's ompio implementation).
4655
Please refer to the `HDF5 manual, function H5Pset_dxpl_mpio <https://support.hdfgroup.org/HDF5/doc/RM/H5P/H5Pset_dxpl_mpio.htm>`_ for more details.
4756

57+
.. tip::
58+
59+
Instead of using an environment variable, independent/collective data transfer can also be configured at the API level via :ref:`JSON/TOML <backendconfig-hdf5>`.
60+
4861
``OPENPMD_HDF5_ALIGNMENT``: this sets the alignment in Bytes for writes via the ``H5Pset_alignment`` function.
4962
According to the `HDF5 documentation <https://support.hdfgroup.org/HDF5/doc/RM/H5P/H5Pset_alignment.htm>`_:
5063
*For MPI IO and other parallel systems, choose an alignment which is a multiple of the disk block size.*
@@ -56,6 +69,7 @@ Any file object greater than or equal in size to threshold bytes will be aligned
5669

5770
``OPENPMD_HDF5_CHUNKS``: this sets defaults for data chunking via `H5Pset_chunk <https://support.hdfgroup.org/HDF5/doc/RM/H5P/H5Pset_chunk.htm>`__.
5871
Chunking generally improves performance and only needs to be disabled in corner-cases, e.g. when heavily relying on independent, parallel I/O that non-collectively declares data records.
72+
The chunk size can alternatively (or additionally) be specified explicitly per dataset, by specifying a dataset-specific chunk size in the JSON/TOML configuration of ``resetDataset()``/``reset_dataset()``.
5973

6074
``OPENPMD_HDF5_COLLECTIVE_METADATA``: this is an option to enable collective MPI calls for HDF5 metadata operations via `H5Pset_all_coll_metadata_ops <https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAllCollMetadataOps>`__ and `H5Pset_coll_metadata_write <https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetCollMetadataWrite>`__.
6175
By default, this optimization is enabled as it has proven to provide performance improvements.

0 commit comments

Comments
 (0)