[EPIC] Complete SQL EXPLAIN Tree Rendering

[irenjj]

Is your feature request related to a problem or challenge?

#14677 introduces basic tree rendering for SQL EXPLAIN output but requires follow-up work to improve completeness, test coverage, and code quality. Below are the key tasks and enhancements needed:

Here is an example of running the new explains:

set datafusion.explain.format = 'tree';
create table foo(x int, y int) as values (1,2), (3,4);
explain select * from foo where x = 4;

The output looks like

+---------------+------------------------------------+
| plan_type     | plan                               |
+---------------+------------------------------------+
| logical_plan  | Filter: foo.x = Int32(4)           |
|               |   TableScan: foo projection=[x, y] |
| physical_plan | ┌───────────────────────────┐      |
|               | │    CoalesceBatchesExec    │      |
|               | └─────────────┬─────────────┘      |
|               | ┌─────────────┴─────────────┐      |
|               | │         FilterExec        │      |
|               | └─────────────┬─────────────┘      |
|               | ┌─────────────┴─────────────┐      |
|               | │       DataSourceExec      │      |
|               | │    --------------------   │      |
|               | │    partition_sizes: [1]   │      |
|               | │       partitions: 1       │      |
|               | └───────────────────────────┘      |
|               |                                    |
+---------------+------------------------------------+

Items to complete

Information Completion for all ExecutionPlans

The current implementation lacks detailed information for many physical plans.

Add rendering logic for missing operators (In every physical plan's fmt_as function).

• Implement tree explain for FilterExec #15000
• Implement tree explain for DataSourceExec #15029
• Implement tree explain for ProjectionExec #15023
• Implement tree explain for AggregateExec #15024
• Implement tree explain for LocalLimitExec #15025
• Implement tree explain for GlobalLimitExec #15026
• Implement tree explain for PartialSortExec
• Implement tree explain for SortExec
• Implement tree explain for RepartitionExec
• Implement tree explain for BoundedWindowAggExec
• Implement tree explain for WindowAggExec
• Implement tree explain for SortPreservingMergeExec
• Implement tree explain for HashJoinExec
• Implement tree explain for SymmetricHashJoinExec
• Implement tree explain for CrossJoinExec
• Implement tree explain for NestedLoopJoinExec
• Implement tree explain for SortMergeJoinExec
• Implement tree explain for StreamingTableExec
• Implement tree explain for ValuesExec
• Implement tree explain for WorkTableExec
• Implement tree explain for InterleaveExec
• Implement tree explain for LazyMemoryExec
• Implement tree explain for ForeignExecutionPlan
• Implement tree explain for JsonSink
• Implement tree explain for CsvSink
• Implement tree explain for ArrowFileSink
• (TODO file more)

New Features:

• Support different EXPLAIN formats via SQL #15021
• Do not print logical plans in tree explain format #15020

Process / Documentation

• Document the tree explain in the user documentation
• Add upgrade guide in 47 for TreeRender in DisplayFormatType`
• Use tree explain by default

Test Coverage Expansion

• Add tests for plans with long metrics (e.g., partition_sizes: [20, 20, 20, 30, ...]) to validate line-wrapping and truncation logic.
• Tests for TPCH queries (e.g., Q5, Q9) to test rendering of deeply nested plans.
• Validate alignment and indentation for multi-level operators.

Code Cleanup & Improvements

TODOs

• Improve explain tree formatting for longer lines / word wrap #15019

Make variables (e.g., indentation size, truncation thresholds) configurable:

// TODO: Make these variables configurable.  

Dead Code Removal

Remove or utilize the unused Coordinate struct:

#[allow(dead_code)]  
pub struct Coordinate { ... }  
// TODO: It's never used.  

[alamb]

I made a first PR to show how to add tree explain

• Implement tree explain for FilterExec #15001

Once we get that one in I plan to file a bunch of other tickets for the remaining ExplainPlan nodes

[milenkovicm]

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

[alamb]

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

There is already a graphviz version (that kind of looks crappy).

Since the output is text, perhaps there is some way to render text as a svg 🤔

[waynexia]

For the setting side, how about using the existing EXPLAIN [ANALYZE] FORMAT <format> grammar? sqlparser can parse and generate a AnalyzeFormat, which the planner can use. But our current implementation just drops this format field:

datafusion/datafusion/sql/src/statement.rs

Lines 213 to 222 in ed517ef

	Statement::Explain {
	verbose,
	statement,
	analyze,
	format: _,
	describe_alias: _,
	..
	} => {
	self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement))
	}

We can also try to support GRAPHVIZ format in this way, it should be easier to be rendered to svg.

[alamb]

For the setting side, how about using the existing EXPLAIN [ANALYZE] FORMAT <format> grammar? sqlparser can parse and generate a AnalyzeFormat, which the planner can use. But our current implementation just drops this format field:

datafusion/datafusion/sql/src/statement.rs

Lines 213 to 222 in ed517ef

Statement::Explain {
verbose,
statement,
analyze,
format: _,
describe_alias: _,
..
} => {
self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement))
}
We can also try to support GRAPHVIZ format in this way, it should be easier to be rendered to svg.

This is a great idea @waynexia

It seems like AnalyzeFormat currently only supports Text, JSON, and GRAPHVIZ 🤔

https://docs.rs/sqlparser/latest/sqlparser/ast/enum.AnalyzeFormat.html

It would be a good way to show graphviz plans

[alamb]

For the setting side, how about using the existing EXPLAIN [ANALYZE] FORMAT grammar? sqlparser can parse and generate a AnalyzeFormat, which the planner can use. But our current implementation just drops this format field:

I filed the following ticket for this

• Support different EXPLAIN formats via SQL #15021

I also filed a ticket to avoid printing logical plans when in tree mode

• Do not print logical plans in tree explain format #15020

[alamb]

Here is another PR to improve DataSource:

• Implement tree explain for DataSourceExec #15029

[milenkovicm]

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

There is already a graphviz version (that kind of looks crappy).

Since the output is text, perhaps there is some way to render text as a svg 🤔

ballista uses graphviz-rust which generates those sub-optimal images (and needs local graph viz installation). spark uses

• https://github.com/dagrejs/graphlib-dot

a javascript framework which works much better for them. Maybe we should keep dot representation if we already have it

[alamb]

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

There is already a graphviz version (that kind of looks crappy).
Since the output is text, perhaps there is some way to render text as a svg 🤔

ballista uses graphviz-rust which generates those sub-optimal images (and needs local graph viz installation). spark uses

• https://github.com/dagrejs/graphlib-dot

a javascript framework which works much better for them. Maybe we should keep dot representation if we already have it

I think that if we just made it easier to use the existing formats, that would be a big win.