Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add reorgs support + indexer restructuring #53

Merged
merged 25 commits into from
Feb 21, 2024

Conversation

PJColombo
Copy link
Member

@PJColombo PJColombo commented Jan 30, 2024

It resolves #19
It resolves #46
It resolves #49

It depends on the following API's PR

This PR contains the following refactoring:

  • Renames slots_processor to synchroniser, which now encapsulates the slot checkpoint logic that was originally part of the indexer's logic. It also adds support for reverse syncing.
  • Refactors the indexer's main logic by defining two parallel tasks, each with its own synchroniser instance:
    1. A historical syncing task that processes slots in reverse, from the last indexed one to either the genesis slot or a slot specified by the user.
    2. An event stream syncing task that listens to block head events and indexes them.

It also adds support for block reorgs.

0xGabi and others added 13 commits January 26, 2024 02:34
…rate the indexer's checkpoint slicing logic within the synchronizer
…on issue

There's a potential race condition issue when handling chain reorganizations by subscribing to block reorg events. This situation arises when new blocks from the newly established canonical chain are indexed prior to the processing of the reorg event. As a result, these blocks might be incorrectly labeled, leading to inconsistencies in our indexing process.
@PJColombo PJColombo force-pushed the support-dual-indexing branch from 3f75e47 to 7f12bd7 Compare January 31, 2024 11:16
@PJColombo PJColombo force-pushed the support-dual-indexing branch from b1a3d44 to 17f5d0c Compare January 31, 2024 14:01
Copy link
Member

@0xGabi 0xGabi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have conducted an initial review of the changes. I still need to cover some important sections on the synchronizer. Anyways I wanted to provide an initial round of feedback.

The refactor of the code has made it more robust with the addition of the synchronizer, which does a great job dividing the sync in two steps. Well done!

@PJColombo PJColombo force-pushed the support-dual-indexing branch from 462d998 to 3dc2c9a Compare February 2, 2024 13:18
@0xGabi 0xGabi mentioned this pull request Feb 4, 2024
@PJColombo PJColombo force-pushed the support-dual-indexing branch 3 times, most recently from 135abfc to 605c242 Compare February 12, 2024 12:22
@PJColombo PJColombo force-pushed the support-dual-indexing branch from 605c242 to e926869 Compare February 12, 2024 15:16
@PJColombo PJColombo force-pushed the support-dual-indexing branch from e38af2f to a82000a Compare February 12, 2024 16:01
@PJColombo PJColombo changed the title feat: add reorgs support + major indexer restructuring feat: add reorgs support + indexer restructuring Feb 12, 2024
Copy link
Member

@0xGabi 0xGabi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic job! I've noticed considerable improvements since my last review. I'm particularly impressed by how you've managed to split the synchronization into two separate processes, as we discussed. The addition of real-time sync significantly enhances the usability of Blobscan, allowing users to engage with it immediately without having to wait for the entire chain to synchronize.

Furthermore, the updates to the error messages are a substantial improvement. They now more effectively explain which dependency is causing an issue, which is super helpful.

I have very few additional suggestions to offer. Well done!

@PJColombo PJColombo merged commit d72630c into master Feb 21, 2024
4 checks passed
@PJColombo PJColombo deleted the support-dual-indexing branch February 21, 2024 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants