Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented SQL query plan explainer, added necessary indexes #728

Open
wants to merge 18 commits into
base: next
Choose a base branch
from

Conversation

polydez
Copy link
Contributor

@polydez polydez commented Mar 6, 2025

Resolves #712 and #57

In this PR we implemented SQL query plan explainer which writes to log something like a query plan below for each running query:

>> EXPLAIN QUERY PLAN -- Selects new notes matching the tags and account IDs search criteria.
SELECT
    block_num,
    batch_index,
    note_index,
    note_id,
    note_type,
    sender,
    tag,
    aux,
    execution_hint,
    merkle_path
FROM
    notes
WHERE
    -- find the next block which contains at least one note with a matching tag or sender
    block_num = (
        SELECT
            block_num
        FROM
            notes
        WHERE
            (tag IN rarray(NULL) OR sender IN rarray(NULL)) AND
            block_num > 0
        ORDER BY
            block_num ASC
    LIMIT 1) AND
    -- filter the block's notes and return only the ones matching the requested tags or senders
    (tag IN rarray(NULL) OR sender IN rarray(NULL))


QUERY PLAN
├── SEARCH notes USING INDEX sqlite_autoindex_notes_1 (block_num=?)
├── SCALAR SUBQUERY 3
│   ├── MULTI-INDEX OR
│   │   ├── INDEX 1
│   │   │   ├── LIST SUBQUERY 1
│   │   │   │   └── SCAN rarray VIRTUAL TABLE INDEX 1:
│   │   │   └── SEARCH notes USING INDEX idx_notes_tag (tag=? AND block_num>?)
│   │   └── INDEX 2
│   │       ├── LIST SUBQUERY 2
│   │       │   └── SCAN rarray VIRTUAL TABLE INDEX 1:
│   │       └── SEARCH notes USING INDEX idx_notes_sender (sender=? AND block_num>?)
│   └── USE TEMP B-TREE FOR ORDER BY
├── LIST SUBQUERY 4
│   └── SCAN rarray VIRTUAL TABLE INDEX 1:
└── LIST SUBQUERY 5
    └── SCAN rarray VIRTUAL TABLE INDEX 1:

This works for both tests and running node, if feature explain-query-plans is enabled.

In order to achieve this, we implemented our own SQLite connection pool which constructs special wrapper for each connection, returning special wrapper for each cached prepared statement. The latter is able to explain a query once it is being run. The same was done for transactions as well. This solution also helped us to get rid of deadpool-sqlite (all necessary functionality from it we have now). And configuring of connection now done in SQLite connection pool instead of hooking connection creation, it looks more elegant to me.

Also implemented some simple checking for problems in query plans during test run (test will fail, if any problem found in any query plan for queries run in tests).

We also added all necessary indices and foreign keys.

@polydez polydez marked this pull request as ready for review March 6, 2025 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant