feat: Add `tree` / pretty explain mode #14677

irenjj · 2025-02-15T03:54:01Z

Which issue does this PR close?

Closes Implement Nicer / DuckDB style explain plans #9371
Follow on: [EPIC] Complete SQL EXPLAIN Tree Rendering #14914

Rationale for this change

What changes are included in this PR?

Prettify explain display format like DuckDB.

In this pr, implemented TreeRender structure, and DataSourceExec info:

> explain select * from t1  order by i;

+---------------+--------------------------------+
| plan_type     | plan                           |
+---------------+--------------------------------+
| logical_plan  | Sort: t1.i ASC NULLS LAST      |
|               |   TableScan: t1 projection=[i] |
| physical_plan | ┌───────────────────────────┐  |
|               | │          SortExec         │  |
|               | └─────────────┬─────────────┘  |
|               | ┌─────────────┴─────────────┐  |
|               | │       DataSourceExec      │  |
|               | │    --------------------   │  |
|               | │    partition_sizes: [0]   │  |
|               | │       partitions: 1       │  |
|               | └───────────────────────────┘  |
|               |                                |
+---------------+--------------------------------+
2 row(s) fetched. 
Elapsed 0.005 seconds.

> explain select * from t1 inner join t2 on t1.i=t2.i;

+---------------+------------------------------------------------------------+
| plan_type     | plan                                                       |
+---------------+------------------------------------------------------------+
| logical_plan  | Inner Join: t1.i = t2.i                                    |
|               |   TableScan: t1 projection=[i]                             |
|               |   TableScan: t2 projection=[i]                             |
| physical_plan | ┌───────────────────────────┐                              |
|               | │    CoalesceBatchesExec    │                              |
|               | └─────────────┬─────────────┘                              |
|               | ┌─────────────┴─────────────┐                              |
|               | │        HashJoinExec       ├──────────────┐               |
|               | └─────────────┬─────────────┘              │               |
|               | ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      ││       DataSourceExec      │ |
|               | │    --------------------   ││    --------------------   │ |
|               | │    partition_sizes: [0]   ││       partitions: 1       │ |
|               | │       partitions: 1       ││    partition_sizes: [0]   │ |
|               | └───────────────────────────┘└───────────────────────────┘ |
|               |                                                            |
+---------------+------------------------------------------------------------+
2 row(s) fetched.

Are these changes tested?

Are there any user-facing changes?

alamb · 2025-02-15T10:13:49Z

😍

I have been dreaming about this @irenjj -- can't wait to check it out

xudong963 · 2025-02-15T15:21:24Z

It's my lovest todo:

Thank you for making it happen!

irenjj · 2025-02-23T13:30:52Z

PTAL @alamb @xudong963 👀

xudong963

Happy to see this happen.

Maybe it's better to open a tracking issue to record next todos, such as rich other physical operator's information to display. Then we can make our new explain perfect in parallel after the PR is merged.

irenjj · 2025-02-24T01:07:55Z

Happy to see this happen.

Maybe it's better to open a tracking issue to record next todos, such as rich other physical operator's information to display. Then we can make our new explain perfect in parallel after the PR is merged.

I will do it later today.❤️

alamb · 2025-02-24T14:26:58Z

Looks awesome -- I have put this on my review queue. So exciting

2010YOUY01

This is very cool 💯

I suggest to add more tests to make sure the tree rendering is correct for complex plans:

Very long metric entry like partition_sizes: [20, 20, 20, 30, 20, 20, 20...]
Complex query: perhaps pick some from TPCH and make sure the tree structure is rendered correct.

Then we can write instructions and open tickets to support pretty explain for other executor nodes. I think now the default explain info is very detailed debug-style messages, perhaps we can make this pretty explain more concise (by looking at what metrics DuckDB includes for a certain plan node 😄 )

2010YOUY01 · 2025-02-25T09:02:10Z

datafusion/datasource/src/source.rs

@@ -43,6 +44,7 @@ pub trait DataSource: Send + Sync {
    ) -> datafusion_common::Result<SendableRecordBatchStream>;
    fn as_any(&self) -> &dyn Any;
    fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter) -> fmt::Result;
+    fn collect_info(&self) -> HashMap<String, String>;


Why do we need a new interface to collect info for pretty explain? Looks like it's possible to implement inside fmt_as().
DuckDB explains also have many non key-value patterns like the following example, we will want the formater to be more flexible.

┌─────────────┴─────────────┐ │ READ_PARQUET │ │ ──────────────────── │ │ Function: │ │ READ_PARQUET │ │ │ │ Projections: │ │ l_orderkey │ │ l_partkey │ │ l_suppkey │ │ l_linenumber │ │ l_quantity │ │ l_extendedprice │ │ │ │ ~6046496 Rows │ └───────────────────────────┘

I agree with this -- it would be much better to avoid a new interface and instead update the existing fmt_as API

alamb

Thank you @irenjj

I took the liberty of merging up from main to resolve a conflict and fixing the CI failure

I am still going through this PR but so far I think it looks great

alamb · 2025-02-25T19:09:02Z

datafusion/common/src/config.rs

@@ -704,6 +704,10 @@ config_namespace! {

        /// When set to true, the explain statement will print schema information
        pub show_schema: bool, default = false
+
+        /// Display format of explain. Default is "indent".
+        /// When set to "pretty", it will print the plan in a tree-rendered format.


rather than "pretty" that prints out a tree I suggest we change the option name to "tree"

datafusion/physical-plan/src/memory.rs

rluvaton · 2025-02-25T19:25:28Z

datafusion/physical-plan/src/render_tree.rs

+
+use crate::ExecutionPlan;
+
+#[allow(dead_code)]


Is it still deadcode? it doesn't look like it

It will warn while cargo build:

warning: fields `x` and `y` are never read --> datafusion/physical-plan/src/render_tree.rs:29:9 | 28 | pub struct Coordinate { | ---------- fields in this struct 29 | pub x: usize, | ^ 30 | pub y: usize, | ^ | = note: `#[warn(dead_code)]` on by default

the Coordinate is inspired from duckdb's implementation, I'll figure out if it's necessary to introduce such a structure.

datafusion/physical-plan/src/render_tree.rs

alamb · 2025-02-27T19:42:40Z

I hope to look at this later today

Looks like the follow on ticket is filed.

[EPIC] Complete SQL EXPLAIN Tree Rendering #14914

alamb

I think this looks great @irenjj

Thanks so much 🙏

I merged up from main and will touch up the ticket you filed

[EPIC] Complete SQL EXPLAIN Tree Rendering #14914

alamb · 2025-03-03T21:27:27Z

This is next on my list of things to merge

What i would like ot do is merge this PR in and then file follow on subtasks in

[EPIC] Complete SQL EXPLAIN Tree Rendering #14914

I am sorry I have been slammed trying to get the DataFusion 46 release out

alamb

I merged up from main to resolve some conflicts in this PR

alamb · 2025-03-04T12:26:15Z

🚀 thank you again so much @irenjj -- this is so cool. A great step towards better explain plans.

I am working on enhancing the follow on ticket you filed in #14914 ❤️

feat: pretty explain

6ad7a62

github-actions bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate common Related to common crate labels Feb 15, 2025

irenjj added 2 commits February 17, 2025 19:49

fix compile errors

d53d876

render_tree

d143783

irenjj force-pushed the pretty_explain branch from ad83ec7 to d143783 Compare February 23, 2025 09:20

irenjj added 5 commits February 23, 2025 20:00

Merge remote-tracking branch 'upstream/main' into pretty_explain

47c1383

fix compile error

8d2ecf5

fmt

d905f46

add License

d2ec193

fix test

e0b27ef

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Feb 23, 2025

irenjj marked this pull request as ready for review February 23, 2025 13:30

xudong963 reviewed Feb 23, 2025

View reviewed changes

This was referenced Feb 24, 2025

Weekly Plan (Andrew Lamb) Feb 24, 2025 #14850

Closed

Add value spans to clickbench tests, add docs and From impl lovasoa/sqlparser-rs#1

Closed

2010YOUY01 reviewed Feb 25, 2025

View reviewed changes

alamb added 2 commits February 25, 2025 14:06

Merge remote-tracking branch 'apache/main' into pretty_explain

aec1d91

Update config docs

1aec472

github-actions bot added the documentation Improvements or additions to documentation label Feb 25, 2025

alamb reviewed Feb 25, 2025

View reviewed changes

rluvaton reviewed Feb 25, 2025

View reviewed changes

datafusion/physical-plan/src/memory.rs Outdated Show resolved Hide resolved

rluvaton reviewed Feb 25, 2025

View reviewed changes

alamb reviewed Feb 25, 2025

View reviewed changes

datafusion/physical-plan/src/render_tree.rs Show resolved Hide resolved

irenjj added 4 commits February 26, 2025 20:26

update config md

d58873c

remove collect_info interface

daefaf2

add comments

3dde06f

fix test

599e43f

github-actions bot added the optimizer Optimizer rules label Feb 26, 2025

fix doc

4907cf6

irenjj mentioned this pull request Feb 27, 2025

[EPIC] Complete SQL EXPLAIN Tree Rendering #14914

Open

36 tasks

Merge branch 'main' into pretty_explain

85bd86a

alamb changed the title ~~feat: tree / pretty explain~~ feat: Add tree / pretty explain mode Feb 28, 2025

Merge remote-tracking branch 'apache/main' into pretty_explain

e050a70

alamb approved these changes Feb 28, 2025

View reviewed changes

alamb mentioned this pull request Mar 3, 2025

Weekly Plan (Andrew Lamb) March 3, 2025 #14978

Open

12 tasks

Merge remote-tracking branch 'apache/main' into pretty_explain

12c1d89

github-actions bot added datasource Changes to the datasource crate ffi Changes to the ffi crate and removed physical-expr Changes to the physical-expr crates labels Mar 3, 2025

alamb mentioned this pull request Mar 3, 2025

Implement Nicer / DuckDB style explain plans #9371

Closed

Merge remote-tracking branch 'apache/main' into pretty_explain

3a9c6f3

alamb reviewed Mar 4, 2025

View reviewed changes

alamb merged commit 6c95b56 into apache:main Mar 4, 2025
26 checks passed

alamb mentioned this pull request Mar 4, 2025

Implement tree explain for FilterExec #15000

Closed

irenjj deleted the pretty_explain branch March 4, 2025 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add `tree` / pretty explain mode #14677

feat: Add `tree` / pretty explain mode #14677

irenjj commented Feb 15, 2025 •

edited by alamb

Loading

alamb commented Feb 15, 2025

xudong963 commented Feb 15, 2025

irenjj commented Feb 23, 2025

xudong963 left a comment

irenjj commented Feb 24, 2025 •

edited

Loading

alamb commented Feb 24, 2025

2010YOUY01 left a comment

2010YOUY01 Feb 25, 2025

alamb Feb 25, 2025

alamb left a comment

alamb Feb 25, 2025

rluvaton Feb 25, 2025

irenjj Feb 26, 2025

alamb commented Feb 27, 2025

alamb left a comment

alamb commented Mar 3, 2025

alamb left a comment

alamb commented Mar 4, 2025


		use crate::ExecutionPlan;

		#[allow(dead_code)]

feat: Add tree / pretty explain mode #14677

feat: Add tree / pretty explain mode #14677

Conversation

irenjj commented Feb 15, 2025 • edited by alamb Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb commented Feb 15, 2025

xudong963 commented Feb 15, 2025

irenjj commented Feb 23, 2025

xudong963 left a comment

Choose a reason for hiding this comment

irenjj commented Feb 24, 2025 • edited Loading

alamb commented Feb 24, 2025

2010YOUY01 left a comment

Choose a reason for hiding this comment

2010YOUY01 Feb 25, 2025

Choose a reason for hiding this comment

alamb Feb 25, 2025

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb Feb 25, 2025

Choose a reason for hiding this comment

rluvaton Feb 25, 2025

Choose a reason for hiding this comment

irenjj Feb 26, 2025

Choose a reason for hiding this comment

alamb commented Feb 27, 2025

alamb left a comment

Choose a reason for hiding this comment

alamb commented Mar 3, 2025

alamb left a comment

Choose a reason for hiding this comment

alamb commented Mar 4, 2025

feat: Add `tree` / pretty explain mode #14677

feat: Add `tree` / pretty explain mode #14677

irenjj commented Feb 15, 2025 •

edited by alamb

Loading

irenjj commented Feb 24, 2025 •

edited

Loading