Add benchmark script #81

sirmarcel · 2024-10-15T13:43:43Z

Fixes #17

As discussed a while ago with @PicoCentauri , this adds a benchmark script, designed to emulate a "typical" training workload. The idea is that we agree to run this before major merges, to make sure we don't cause performance regressions.

The script runs Ewald and PME, with and without Metatensor, for CsCl crystals of various sizes. The results are saved as .yaml, with some diagnonstic information. An example output can be seen in #80. The script already worked quite nicely to pointint that something was wrong in that case...

We could consider making some additional repository for the .yaml and make a nice plot of performance, hopefully, increasing over time.

📚 Documentation preview 📚: https://torch-pme--81.org.readthedocs.build/en/81/

sirmarcel · 2024-10-15T13:51:24Z

@kvhuguenin the times are reported, as far as i'm able to determine, per sample. however, the script allows for asynchronous dispatch, so in principle there could even be some parallelisation benefits.... maybe i should turn that off. let's have a look.

PicoCentauri · 2024-10-15T14:11:57Z

Very nice @sirmarcel and thanks. This will be very useful.

Regarding storing. I had the idea of creating an orphan branch maybe called benchmark and put the files there together with a plotting script and the figure.

PicoCentauri · 2024-10-15T14:15:18Z

benchmark/README.rst

+
+.. code-block:: bash
+
+    pip install .[metatensor]


I think also vesin, ase etc are needed.

benchmark/README.rst

benchmark/run.py

PicoCentauri · 2024-10-15T14:53:55Z

benchmark/run.py

+version = {
+    "torch": str(torch.__version__),
+    "torch-pme-commit": torch_pme_commit,
+    "torch-pme-status": torch_pme_status,
+}


maybe also add the torchpme.__version__. Even though useless before we release but useful in the future.

benchmark/run.py

PicoCentauri · 2024-10-15T14:57:42Z

Maybe we should also run this code somewhere in CI. Not for benchmark but just that we know we don't break anything. maybe just a subprocess call in der testsuite...

EDIT. Maybe one can eve do a tox -e benchmark. This will create an fresh env automatically, but maybe is problematic on HPC systems where you can't easily install packages...

ceriottm · 2024-10-15T19:22:41Z

Would be nice to make this a bit more customizable - setting the devices to run on, the sizes to try, etc. We can leave all this as defaults, but e.g. if one is using this in development it makes sense to only run what's being optimized for.

benchmark/run.py

PicoCentauri · 2024-12-16T09:57:10Z

What is the status here @sirmarcel ?

sirmarcel · 2024-12-16T17:45:07Z

Couldn't keep up with the API changes. Do you think this is worth updating and keeping current?

PicoCentauri · 2024-12-16T18:33:40Z

I think we kind of stabilized now. I am not aware of any API breaking changes soon. Right @ceriottm ?

Co-authored-by: Philip Loche <philip.loche@posteo.de>

* Refactor Calculator class to include units conversion This commit refactors the Calculator class in the base.py file to include a new parameter, units, which allows for conversion of potential values to different units. The units parameter accepts values such as "Gaussian", "eV", "kcal/mol", and "kJ/mol". The commit also adds a unit conversion factor dictionary and checks if the provided units are valid. The potential_sr and potential_lr calculations in the _compute_rspace and _compute_kspace methods respectively are multiplied by the unit conversion factor. --------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

--------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

An example for a Calculator that evaluates LODE features using some of the advanced torch-pme features

--------- Co-authored-by: Michele Ceriotti <michele.ceriotti@gmail.com> Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

* Restructure for better docs * Fixed "broken" image and removed reference to Meshlode * Add an mini-gallery to all doc pages * remove :py: prefix --------- Co-authored-by: Michele Ceriotti <michele.ceriotti@gmail.com> Co-authored-by: E-Rum <rev99@internet.ru>

Co-authored-by: Kevin Kazuki Huguenin-Dumittan <kvhuguenin@gmail.com>

--------- Co-authored-by: Michele Ceriotti <michele.ceriotti@gmail.com> Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

* Validate consistent dtype between positions and neighbor_distances * unify `_validate_parameters` * use torch.get_default_device() * remove excessive use of double precision * update README example --------- Co-authored-by: GardevoirX <xqj.2001@outlook.com>

…MCalculator` (#159)

)

--------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

* Add CalculatorDipole and PotentialDipole classes with tests * Refactor PotentialDipole class to inherit from torch.nn.Module * Refactor potential calculations in CalculatorDipole and PotentialDipole classes for improved performance and clarity * Add lr_from_dist calculation to PotentialDipole class * Implement lr_from_k_sq, self_contribution and background_correction for PotentialDipole class * Finish calculator_dipole.py * Correct energy calculation in CalculatorDipole and fix smearing factor in PotentialDipole * Refactor device handling in CalculatorDipole and PotentialDipole classes; improve performance of potential calculations and enhance test coverage for magnetostatics * Enhance PotentialDipole class by allowing device parameter to accept string type; fix tensor dimension in f_cutoff method and update test assertions for clarity * Fix a bug in k_space calculator; add epsilon parameter to PotentialDipole and update background_correction method * Add dipoles test frames and enhance tests for magnetostatic Ewald calculations * Replace vesin.torch, as it doesn’t support Windows OS. * Refactor CalculatorDipole and PotentialDipole classes to remove device and dtype parameters; update tests accordingly for improved clarity and consistency. * Update __init__.py to include CalculatorDipole and PotentialDipole in exports; enhance compute_distances function to optionally return distance vectors. * restructure some files * Enhance docstrings for CalculatorDipole and PotentialDipole classes; provide detailed parameter descriptions and usage examples for improved clarity. * proofread docs --------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

We now also run P3M.

sirmarcel · 2025-02-17T15:17:42Z

Something went terribly wrong here. Closing this and re-opening.

Fixes #17 A continuation of #81 -- see discussion there.

Add benchmark script

63f99ce

sirmarcel mentioned this pull request Oct 15, 2024

Ewald calculator is extremely slow due to for loop #80

Closed

PicoCentauri reviewed Oct 15, 2024

View reviewed changes

PicoCentauri reviewed Oct 16, 2024

View reviewed changes

benchmark/run.py Outdated Show resolved Hide resolved

benchmark/run.py Show resolved Hide resolved

sirmarcel and others added 18 commits January 24, 2025 11:29

Update benchmark/run.py

41fc721

Co-authored-by: Philip Loche <philip.loche@posteo.de>

Apply suggestions from code review

65775f4

Co-authored-by: Philip Loche <philip.loche@posteo.de>

rephrase prefactor docs text

8262a3b

tuning of PME parameters (#75)

ba9afa2

--------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

remoce cscl examples from main repo (#82)

0808877

Fix the initial guess of tune_ewald (#83)

04a2d86

--------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

Remove for loop from EwaldCalculator (#84)

17bb754

Spline potential (#85)

55a224e

--------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

refactor update functions (#91)

d7380f7

LODE example (#90)

1a57962

An example for a Calculator that evaluates LODE features using some of the advanced torch-pme features

Check on the positivity of sum_squared_charges during tuning (#93)

f565965

Support tuning with fixed parameters (#94)

bba8c1d

Add CombinedPotential class (#86)

5fad9fe

--------- Co-authored-by: Michele Ceriotti <michele.ceriotti@gmail.com> Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

update to metatensor-torch 0.6 (#95)

1a22b1c

Update Madelung constants for higher accuracy (#97)

7872ed9

Co-authored-by: Kevin Kazuki Huguenin-Dumittan <kvhuguenin@gmail.com>

Add basic doc linting (#96)

887d5d0

E-Rum and others added 25 commits February 17, 2025 16:14

Fix docs

1440be9

Another fix docs

de75e53

fix docs

60addd1

Fix some more typos in atomistic example

9f8a008

Revamped tuning algorithm (#130)

2bbd818

--------- Co-authored-by: Michele Ceriotti <michele.ceriotti@gmail.com> Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

Update release instructions

1c05c81

prepare v0.2.0 release

a9e4fdd

Enhanced cleanup (#155)

2376ad5

* Validate consistent dtype between positions and neighbor_distances * unify `_validate_parameters` * use torch.get_default_device() * remove excessive use of double precision * update README example --------- Co-authored-by: GardevoirX <xqj.2001@outlook.com>

test readme

fb56cae

Make timing test less strict

e617e5e

better docs for cell, charges, positions (#160)

ad58a5b

Fix device and dtype not being specified in the __init__ of `P3…

9b18ba1

…MCalculator` (#159)

Fix tests for the case when CUDA is available on the system (#161)

75e5ebf

Add device consistency between string and torch.device (#163)

8347c12

sort pyproject.toml (#164)

9d419bf

Fix formatting in README.rst for consistency in class descriptions (#165

e0fc607

)

Make dtype and device implicit in classes (#166)

80fbd14

--------- Co-authored-by: Philip Loche <ploche@physik.fu-berlin.de>

Update dependency pinning (#168)

f171d75

move some text and images from paper to docs

c298942

fix duplicate examples in metatensor

884ca07

use python -m consistently in CI

15c99df

Update tox to lint/format the benchmark script

e324ad0

Update benchmark script and readme

883419f

We now also run P3M.

Update run.py

82e4421

sirmarcel closed this Feb 17, 2025

sirmarcel added a commit that referenced this pull request Feb 17, 2025

Add benchmark scripts

e8fe852

Fixes #17 A continuation of #81 -- see discussion there.

sirmarcel mentioned this pull request Feb 17, 2025

Add benchmark scripts #173

Open

4 tasks

sirmarcel deleted the benchmark branch February 17, 2025 15:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add benchmark script #81

Add benchmark script #81

sirmarcel commented Oct 15, 2024 •

edited by PicoCentauri

Loading

sirmarcel commented Oct 15, 2024

PicoCentauri commented Oct 15, 2024

PicoCentauri Oct 15, 2024 •

edited

Loading

PicoCentauri Oct 15, 2024

PicoCentauri commented Oct 15, 2024 •

edited

Loading

ceriottm commented Oct 15, 2024

PicoCentauri commented Dec 16, 2024

sirmarcel commented Dec 16, 2024

PicoCentauri commented Dec 16, 2024

sirmarcel commented Feb 17, 2025

Add benchmark script #81

Add benchmark script #81

Conversation

sirmarcel commented Oct 15, 2024 • edited by PicoCentauri Loading

sirmarcel commented Oct 15, 2024

PicoCentauri commented Oct 15, 2024

PicoCentauri Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

PicoCentauri Oct 15, 2024

Choose a reason for hiding this comment

PicoCentauri commented Oct 15, 2024 • edited Loading

ceriottm commented Oct 15, 2024

PicoCentauri commented Dec 16, 2024

sirmarcel commented Dec 16, 2024

PicoCentauri commented Dec 16, 2024

sirmarcel commented Feb 17, 2025

sirmarcel commented Oct 15, 2024 •

edited by PicoCentauri

Loading

PicoCentauri Oct 15, 2024 •

edited

Loading

PicoCentauri commented Oct 15, 2024 •

edited

Loading