Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add tree / pretty explain mode #14677

Merged
merged 23 commits into from
Mar 4, 2025
Merged

feat: Add tree / pretty explain mode #14677

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6ad7a62
feat: pretty explain
irenjj Feb 15, 2025
d53d876
fix compile errors
irenjj Feb 15, 2025
d143783
render_tree
irenjj Feb 23, 2025
47c1383
Merge remote-tracking branch 'upstream/main' into pretty_explain
irenjj Feb 23, 2025
8d2ecf5
fix compile error
irenjj Feb 23, 2025
d905f46
fmt
irenjj Feb 23, 2025
d2ec193
add License
irenjj Feb 23, 2025
e0b27ef
fix test
irenjj Feb 23, 2025
aec1d91
Merge remote-tracking branch 'apache/main' into pretty_explain
alamb Feb 25, 2025
1aec472
Update config docs
alamb Feb 25, 2025
48fd985
Add .slt test
alamb Feb 25, 2025
5f2e4df
Add test for MemoryExec
alamb Feb 25, 2025
b351878
Use BTreeMap to ensure consistent order
alamb Feb 25, 2025
aba177d
credit & rename pretty to tree
irenjj Feb 26, 2025
d58873c
update config md
irenjj Feb 26, 2025
daefaf2
remove collect_info interface
irenjj Feb 26, 2025
3dde06f
add comments
irenjj Feb 26, 2025
599e43f
fix test
irenjj Feb 26, 2025
4907cf6
fix doc
irenjj Feb 27, 2025
85bd86a
Merge branch 'main' into pretty_explain
irenjj Feb 28, 2025
e050a70
Merge remote-tracking branch 'apache/main' into pretty_explain
alamb Feb 28, 2025
12c1d89
Merge remote-tracking branch 'apache/main' into pretty_explain
alamb Mar 3, 2025
3a9c6f3
Merge remote-tracking branch 'apache/main' into pretty_explain
alamb Mar 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions datafusion-examples/examples/planner_api.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

use datafusion::error::Result;
use datafusion::logical_expr::{LogicalPlan, PlanType};
use datafusion::physical_plan::displayable;
use datafusion::physical_plan::{displayable, DisplayFormatType};
use datafusion::physical_planner::DefaultPhysicalPlanner;
use datafusion::prelude::*;

Expand Down Expand Up @@ -78,7 +78,11 @@ async fn to_physical_plan_in_one_api_demo(
println!(
"Physical plan direct from logical plan:\n\n{}\n\n",
displayable(physical_plan.as_ref())
.to_stringified(false, PlanType::InitialPhysicalPlan)
.to_stringified(
false,
PlanType::InitialPhysicalPlan,
DisplayFormatType::Default
)
.plan
);

Expand Down Expand Up @@ -120,7 +124,11 @@ async fn to_physical_plan_step_by_step_demo(
println!(
"Final physical plan:\n\n{}\n\n",
displayable(physical_plan.as_ref())
.to_stringified(false, PlanType::InitialPhysicalPlan)
.to_stringified(
false,
PlanType::InitialPhysicalPlan,
DisplayFormatType::Default
)
.plan
);

Expand All @@ -135,7 +143,11 @@ async fn to_physical_plan_step_by_step_demo(
println!(
"Optimized physical plan:\n\n{}\n\n",
displayable(physical_plan.as_ref())
.to_stringified(false, PlanType::InitialPhysicalPlan)
.to_stringified(
false,
PlanType::InitialPhysicalPlan,
DisplayFormatType::Default
)
.plan
);

Expand Down
4 changes: 4 additions & 0 deletions datafusion/common/src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -704,6 +704,10 @@ config_namespace! {

/// When set to true, the explain statement will print schema information
pub show_schema: bool, default = false

/// Display format of explain. Default is "indent".
/// When set to "pretty", it will print the plan in a tree-rendered format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than "pretty" that prints out a tree I suggest we change the option name to "tree"

pub format: String, default = "indent".to_string()
}
}

Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/src/datasource/file_format/arrow.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,9 @@ impl Debug for ArrowFileSink {
impl DisplayAs for ArrowFileSink {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "ArrowFileSink(file_groups=",)?;
FileGroupDisplay(&self.config.file_groups).fmt_as(t, f)?;
write!(f, ")")
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/src/datasource/file_format/csv.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -667,7 +667,9 @@ impl Debug for CsvSink {
impl DisplayAs for CsvSink {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "CsvSink(file_groups=",)?;
FileGroupDisplay(&self.config.file_groups).fmt_as(t, f)?;
write!(f, ")")
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/src/datasource/file_format/json.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,9 @@ impl Debug for JsonSink {
impl DisplayAs for JsonSink {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "JsonSink(file_groups=",)?;
FileGroupDisplay(&self.config.file_groups).fmt_as(t, f)?;
write!(f, ")")
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/src/datasource/file_format/parquet.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -696,7 +696,9 @@ impl Debug for ParquetSink {
impl DisplayAs for ParquetSink {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "ParquetSink(file_groups=",)?;
FileGroupDisplay(&self.config.file_groups).fmt_as(t, f)?;
write!(f, ")")
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/src/datasource/memory.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,9 @@ impl Debug for MemSink {
impl DisplayAs for MemSink {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
let partition_count = self.batches.len();
write!(f, "MemoryTable (partitions={partition_count})")
}
Expand Down
6 changes: 6 additions & 0 deletions datafusion/core/src/datasource/physical_plan/file_scan_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ use crate::datasource::file_format::file_compression_type::FileCompressionType;
use crate::datasource::{listing::PartitionedFile, object_store::ObjectStoreUrl};
use crate::{error::Result, scalar::ScalarValue};
use std::any::Any;
use std::collections::HashMap;
use std::fmt::Formatter;
use std::{fmt, sync::Arc};

Expand Down Expand Up @@ -270,6 +271,11 @@ impl DataSource for FileScanConfig {
.build() as _
}))
}

fn collect_info(&self) -> HashMap<String, String> {
// TODO: collect explain info
HashMap::new()
}
}

impl FileScanConfig {
Expand Down
4 changes: 2 additions & 2 deletions datafusion/core/src/datasource/physical_plan/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ impl DisplayAs for FileGroupsDisplay<'_> {
let groups = if n_groups == 1 { "group" } else { "groups" };
write!(f, "{{{n_groups} {groups}: [")?;
match t {
DisplayFormatType::Default => {
DisplayFormatType::Default | DisplayFormatType::TreeRender => {
// To avoid showing too many partitions
let max_groups = 5;
fmt_up_to_n_elements(self.0, max_groups, f, |group, f| {
Expand Down Expand Up @@ -155,7 +155,7 @@ impl DisplayAs for FileGroupDisplay<'_> {
fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter) -> FmtResult {
write!(f, "[")?;
match t {
DisplayFormatType::Default => {
DisplayFormatType::Default | DisplayFormatType::TreeRender => {
// To avoid showing too many files
let max_files = 5;
fmt_up_to_n_elements(self.0, max_files, f, |pf, f| {
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/src/datasource/physical_plan/parquet/source.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -559,7 +559,9 @@ impl FileSource for ParquetSource {

fn fmt_extra(&self, t: DisplayFormatType, f: &mut Formatter) -> std::fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
let predicate_string = self
.predicate()
.map(|p| format!(", predicate={p}"))
Expand Down
29 changes: 25 additions & 4 deletions datafusion/core/src/physical_planner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

use std::borrow::Cow;
use std::collections::HashMap;
use std::str::FromStr;
use std::sync::Arc;

use crate::datasource::file_format::file_type_to_format;
Expand Down Expand Up @@ -87,6 +88,7 @@ use datafusion_physical_optimizer::PhysicalOptimizerRule;
use datafusion_physical_plan::execution_plan::InvariantLevel;
use datafusion_physical_plan::placeholder_row::PlaceholderRowExec;
use datafusion_physical_plan::unnest::ListUnnest;
use datafusion_physical_plan::DisplayFormatType;

use crate::schema_equivalence::schema_satisfied_by;
use async_trait::async_trait;
Expand Down Expand Up @@ -1723,6 +1725,7 @@ impl DefaultPhysicalPlanner {
let mut stringified_plans = vec![];

let config = &session_state.config_options().explain;
let explain_format = DisplayFormatType::from_str(&config.format)?;

if !config.physical_plan_only {
stringified_plans.clone_from(&e.stringified_plans);
Expand All @@ -1742,7 +1745,11 @@ impl DefaultPhysicalPlanner {
displayable(input.as_ref())
.set_show_statistics(config.show_statistics)
.set_show_schema(config.show_schema)
.to_stringified(e.verbose, InitialPhysicalPlan),
.to_stringified(
e.verbose,
InitialPhysicalPlan,
explain_format,
),
);

// Show statistics + schema in verbose output even if not
Expand All @@ -1755,6 +1762,7 @@ impl DefaultPhysicalPlanner {
.to_stringified(
e.verbose,
InitialPhysicalPlanWithStats,
explain_format,
),
);
}
Expand All @@ -1765,6 +1773,7 @@ impl DefaultPhysicalPlanner {
.to_stringified(
e.verbose,
InitialPhysicalPlanWithSchema,
explain_format,
),
);
}
Expand All @@ -1780,7 +1789,11 @@ impl DefaultPhysicalPlanner {
displayable(plan)
.set_show_statistics(config.show_statistics)
.set_show_schema(config.show_schema)
.to_stringified(e.verbose, plan_type),
.to_stringified(
e.verbose,
plan_type,
explain_format,
),
);
},
);
Expand All @@ -1791,7 +1804,11 @@ impl DefaultPhysicalPlanner {
displayable(input.as_ref())
.set_show_statistics(config.show_statistics)
.set_show_schema(config.show_schema)
.to_stringified(e.verbose, FinalPhysicalPlan),
.to_stringified(
e.verbose,
FinalPhysicalPlan,
explain_format,
),
);

// Show statistics + schema in verbose output even if not
Expand All @@ -1804,6 +1821,7 @@ impl DefaultPhysicalPlanner {
.to_stringified(
e.verbose,
FinalPhysicalPlanWithStats,
explain_format,
),
);
}
Expand All @@ -1814,6 +1832,7 @@ impl DefaultPhysicalPlanner {
.to_stringified(
e.verbose,
FinalPhysicalPlanWithSchema,
explain_format,
),
);
}
Expand Down Expand Up @@ -2717,7 +2736,9 @@ mod tests {
impl DisplayAs for NoOpExecutionPlan {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "NoOpExecutionPlan")
}
}
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/tests/custom_sources_cases/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,9 @@ impl DisplayAs for CustomExecutionPlan {
f: &mut std::fmt::Formatter,
) -> std::fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "CustomExecutionPlan: projection={:#?}", self.projection)
}
}
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/tests/custom_sources_cases/provider_filter_pushdown.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,9 @@ impl DisplayAs for CustomPlan {
f: &mut std::fmt::Formatter,
) -> std::fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "CustomPlan: batch_size={}", self.batches.len(),)
}
}
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/tests/custom_sources_cases/statistics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,9 @@ impl DisplayAs for StatisticsValidation {
f: &mut std::fmt::Formatter,
) -> std::fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(
f,
"StatisticsValidation: col_count={}, row_count={:?}",
Expand Down
8 changes: 6 additions & 2 deletions datafusion/core/tests/physical_optimizer/join_selection.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -917,7 +917,9 @@ impl DisplayAs for UnboundedExec {
f: &mut std::fmt::Formatter,
) -> std::fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(
f,
"UnboundedExec: unbounded={}",
Expand Down Expand Up @@ -1011,7 +1013,9 @@ impl DisplayAs for StatisticsExec {
f: &mut std::fmt::Formatter,
) -> std::fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(
f,
"StatisticsExec: col_count={}, row_count={:?}",
Expand Down
4 changes: 3 additions & 1 deletion datafusion/core/tests/user_defined/user_defined_plan.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -697,7 +697,9 @@ impl Debug for TopKExec {
impl DisplayAs for TopKExec {
fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter) -> fmt::Result {
match t {
DisplayFormatType::Default | DisplayFormatType::Verbose => {
DisplayFormatType::Default
| DisplayFormatType::Verbose
| DisplayFormatType::TreeRender => {
write!(f, "TopKExec: k={}", self.k)
}
}
Expand Down
Loading
Loading