perf: Add prover jobs as one multi-insert for Witness Generator #3587
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sanekmelnikov
changed the title
sanekmelnikov
changed the title
EmilLuta
reviewed
Feb 11, 2025
EmilLuta
previously approved these changes
Feb 11, 2025
EmilLuta
previously approved these changes
Feb 11, 2025
EmilLuta
approved these changes
Feb 11, 2025
Artemka374
pushed a commit
that referenced
this pull request
Feb 12, 2025
## What ❔ This PR modifies the logic of inserting prover jobs to the database - switching from adding each job sequentially to performing once multi-insert for all rows. ## Why ❔ Currently [insert_prover_jobs](https://github.com/matter-labs/zksync-era/blob/prover-v18.0.0/prover/crates/lib/prover_dal/src/fri_prover_dal.rs#L32-L57) iterates over jobs one by one and makes one network call per job insert. This is inefficient and can result into big latency for cases when Witness Generator and DB are located in different regions. BWG is especially affected since there can be 10k+ jobs added [here](https://github.com/matter-labs/zksync-era/blob/prover-v18.0.0/prover/crates/bin/witness_generator/src/rounds/basic_circuits/artifacts.rs#L71-L111) We expect a significant gain in performance for the non collocated BWG and DB setup. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
Artemka374
pushed a commit
that referenced
this pull request
Feb 13, 2025
## What ❔ This PR modifies the logic of inserting prover jobs to the database - switching from adding each job sequentially to performing once multi-insert for all rows. ## Why ❔ Currently [insert_prover_jobs](https://github.com/matter-labs/zksync-era/blob/prover-v18.0.0/prover/crates/lib/prover_dal/src/fri_prover_dal.rs#L32-L57) iterates over jobs one by one and makes one network call per job insert. This is inefficient and can result into big latency for cases when Witness Generator and DB are located in different regions. BWG is especially affected since there can be 10k+ jobs added [here](https://github.com/matter-labs/zksync-era/blob/prover-v18.0.0/prover/crates/bin/witness_generator/src/rounds/basic_circuits/artifacts.rs#L71-L111) We expect a significant gain in performance for the non collocated BWG and DB setup. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
This PR modifies the logic of inserting prover jobs to the database - switching from adding each job sequentially to performing once multi-insert for all rows.
Why ❔
Currently insert_prover_jobs iterates over jobs one by one and makes one network call per job insert.
This is inefficient and can result into big latency for cases when Witness Generator and DB are located in different regions. BWG is especially affected since there can be 10k+ jobs added here
We expect a significant gain in performance for the non collocated BWG and DB setup.
Checklist
zkstack dev fmtandzkstack dev lint.