Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic benchmarking workflow, create benchmark/ folder with bench script #299

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

starsfordummies
Copy link
Contributor

@starsfordummies starsfordummies commented Jan 20, 2025

Ideally this should create some benchmark data in dev/bench/data.js in the gh-pages branch, as well as an accompanying index.html - I have an updated index.html as well but I don't know how to make it such that the pr puts it in the gh-pages folder...

the html file to put in gh-pages should be https://github.com/starsfordummies/testing_livebench/blob/gh-pages/dev/bench/index3.html

More info on the thing in https://github.com/marketplace/actions/continuous-benchmark

@mofeing
Copy link
Member

mofeing commented Jan 20, 2025

yeah, don't worry. we can later add it to the gh-pages (although i think we need an automated way).

let's start by adding some benchmarks. @starsfordummies @jofrevalles @Todorbsc what do you think about benchmarking the following?

  • contraction of 2 Tensors (matmul) for Float64 & ComplexF64. at least 3 different sizes (small, mid & large).
  • contraction of 2 Tensors (more complex pattern) with the same variations.
  • MPS overlap for different number of sites (L=10,=30,=60) and maximum bond dimensions
    • for L=10, use 2,32
    • for L=30, use 2,128,1024
    • for L=60, use 128,1024
  • MPS canonization from NonCanonical to Canonical, from NonCanonical to MixedCanonical and from MixedCanonical to MixedCanonical in another site
    • yet to decide the benchmark space

we can benchmark more stuff in the future, but let's start with the following which are key to us right now

@starsfordummies i'm changing this to a draft PR because it's not yet ready to merge

@mofeing mofeing marked this pull request as draft January 20, 2025 15:22
@jofrevalles
Copy link
Member

Hey @mofeing @starsfordummies @Todorbsc, I will design some benchmarks for this and let you know. Is this okay?

@mofeing
Copy link
Member

mofeing commented Jan 22, 2025

@jofrevalles that sounds great! i would say that feel free to push to this PR so we can see the result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants