Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datafusion 46 support #2417

Merged
merged 34 commits into from
Mar 9, 2025
Merged

Datafusion 46 support #2417

merged 34 commits into from
Mar 9, 2025

Conversation

AdamGS
Copy link
Contributor

@AdamGS AdamGS commented Feb 18, 2025 •
edited
Loading

I mostly just like to see changes in Github.
We can merge this once Datafusion 46 released, assuming it includes the following PRs:

Seems like everything we wanted to get in made it, this branch now compiles AND passes all checks, so we're only waiting for the actual release in apache/datafusion#14123

@AdamGS AdamGS marked this pull request as draft February 18, 2025 17:54
@AdamGS AdamGS added the do not merge Pull requests that are not intended to merge label Feb 19, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025 •
edited
Loading

Benchmarks: TPC-H on NVME

Table of Results
name PR b97b9f2 base 9f331e1 ratio (PR/base) unit
tpch_q01/arrow 46814764 4.08447e+07 1.14617 ns
tpch_q02/arrow 48973737 4.6717e+07 1.04831 ns
tpch_q03/arrow 30766041 2.91328e+07 1.05606 ns
tpch_q04/arrow 23304113 2.0694e+07 1.12613 ns
tpch_q05/arrow 48195454 4.60596e+07 1.04637 ns
tpch_q06/arrow 10343090 8.52471e+06 1.21331 ns
tpch_q07/arrow 79271919 6.97104e+07 1.13716 ns
tpch_q08/arrow 58796600 5.39776e+07 1.08928 ns
tpch_q09/arrow 73187245 6.64376e+07 1.10159 ns
tpch_q10/arrow 48567260 4.20389e+07 1.15529 ns
tpch_q11/arrow 25206092 2.22554e+07 1.13258 ns
tpch_q12/arrow 29349650 2.589e+07 1.13363 ns
tpch_q13/arrow 17323939 1.54277e+07 1.12291 ns
tpch_q14/arrow 15513910 1.23542e+07 1.25576 ns
tpch_q15/arrow 31051360 2.264e+07 1.37153 ns
tpch_q16/arrow 23337823 1.97894e+07 1.17931 ns
tpch_q17/arrow 61250816 5.85785e+07 1.04562 ns
tpch_q18/arrow 103425701 9.32697e+07 1.10889 ns
tpch_q19/arrow 30350457 2.53769e+07 1.19599 ns
tpch_q20/arrow 37540004 3.01326e+07 1.24583 ns
tpch_q21/arrow 117939389 1.08658e+08 1.08542 ns
tpch_q22/arrow 15957822 1.35142e+07 1.18082 ns
tpch_q01/parquet 116625506 1.12174e+08 1.03969 ns
tpch_q02/parquet 118449296 1.11287e+08 1.06436 ns
tpch_q03/parquet 107499112 9.90065e+07 1.08578 ns
tpch_q04/parquet 63227101 5.47861e+07 1.15407 ns
tpch_q05/parquet 125470068 1.138e+08 1.10255 ns
tpch_q06/parquet 26903619 2.43043e+07 1.10695 ns
tpch_q07/parquet 134657840 1.23816e+08 1.08756 ns
tpch_q08/parquet 165129586 1.50245e+08 1.09907 ns
tpch_q09/parquet 215404776 2.01786e+08 1.06749 ns
tpch_q10/parquet 132651231 1.24476e+08 1.06568 ns
tpch_q11/parquet 53823206 5.15499e+07 1.0441 ns
tpch_q12/parquet 97547382 8.82884e+07 1.10487 ns
tpch_q13/parquet 151779872 1.43811e+08 1.05541 ns
tpch_q14/parquet 49731367 4.32881e+07 1.14885 ns
tpch_q15/parquet 66476861 6.10372e+07 1.08912 ns
tpch_q16/parquet 55233680 4.73471e+07 1.16657 ns
tpch_q17/parquet 135906590 1.25096e+08 1.08642 ns
tpch_q18/parquet 198121554 1.8296e+08 1.08287 ns
tpch_q19/parquet 76526929 7.19709e+07 1.0633 ns
tpch_q20/parquet 101077350 9.16694e+07 1.10263 ns
tpch_q21/parquet 191549858 1.74484e+08 1.09781 ns
tpch_q22/parquet 52292197 4.72311e+07 1.10716 ns
tpch_q01/vortex-file-compressed 37410820 3.2454e+07 1.15273 ns
tpch_q02/vortex-file-compressed 65463877 5.2561e+07 1.24548 ns
tpch_q03/vortex-file-compressed 35470417 3.10933e+07 1.14077 ns
tpch_q04/vortex-file-compressed 22536876 1.89111e+07 1.19173 ns
tpch_q05/vortex-file-compressed 51843094 4.38517e+07 1.18224 ns
tpch_q06/vortex-file-compressed 9912669 8.84167e+06 1.12113 ns
tpch_q07/vortex-file-compressed 75159849 6.40491e+07 1.17347 ns
tpch_q08/vortex-file-compressed 60266602 5.23924e+07 1.15029 ns
tpch_q09/vortex-file-compressed 71360755 6.48479e+07 1.10043 ns
tpch_q10/vortex-file-compressed 58357352 5.12777e+07 1.13807 ns
tpch_q11/vortex-file-compressed 29945096 2.30017e+07 1.30187 ns
tpch_q12/vortex-file-compressed 23459399 2.38949e+07 0.981772 ns
tpch_q13/vortex-file-compressed 28364620 2.51642e+07 1.12718 ns
tpch_q14/vortex-file-compressed 15973074 1.5311e+07 1.04324 ns
tpch_q15/vortex-file-compressed 34100890 2.76424e+07 1.23364 ns
tpch_q16/vortex-file-compressed 32229223 2.58082e+07 1.2488 ns
tpch_q17/vortex-file-compressed 58383602 4.90882e+07 1.18936 ns
tpch_q18/vortex-file-compressed 90671415 8.15663e+07 1.11163 ns
tpch_q19/vortex-file-compressed 32544099 3.06786e+07 1.06081 ns
tpch_q20/vortex-file-compressed 42858326 3.48887e+07 1.22843 ns
tpch_q21/vortex-file-compressed 98374579 8.19945e+07 1.19977 ns
tpch_q22/vortex-file-compressed 30265138 2.6364e+07 1.14797 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025 •
edited
Loading

Benchmarks: TPC-H on S3

Table of Results
name PR b97b9f2 base 9f331e1 ratio (PR/base) unit
tpch_q01/parquet 260708968 2.66292e+08 0.979035 ns
tpch_q02/parquet 678727230 6.69487e+08 1.0138 ns
tpch_q03/parquet 411388517 4.29232e+08 0.95843 ns
tpch_q04/parquet 216431000 2.27385e+08 0.951824 ns
tpch_q05/parquet 552127397 5.80036e+08 0.951884 ns
tpch_q06/parquet 182282351 1.81014e+08 1.007 ns
tpch_q07/parquet 611844490 6.28693e+08 0.973201 ns
tpch_q08/parquet 774012355 8.14386e+08 0.950424 ns
tpch_q09/parquet 705467847 6.61229e+08 1.0669 ns
tpch_q10/parquet 554948955 5.23239e+08 1.0606 ns
tpch_q11/parquet 275690421 2.87372e+08 0.959351 ns
tpch_q12/parquet 278217071 2.79256e+08 0.996278 ns
tpch_q13/parquet 381604593 3.89036e+08 0.980897 ns
tpch_q14/parquet 245971146 2.52142e+08 0.975525 ns
tpch_q15/parquet 457070181 4.63399e+08 0.986342 ns
tpch_q16/parquet 257782409 2.63092e+08 0.97982 ns
tpch_q17/parquet 387588015 3.92452e+08 0.987606 ns
tpch_q18/parquet 519680170 5.31858e+08 0.977102 ns
tpch_q19/parquet 277595773 2.96374e+08 0.936639 ns
tpch_q20/parquet 520019788 5.12223e+08 1.01522 ns
tpch_q21/parquet 599233430 5.98214e+08 1.0017 ns
tpch_q22/parquet 257725737 2.70901e+08 0.951365 ns
tpch_q01/vortex-file-compressed 142731476 1.43859e+08 0.992159 ns
tpch_q02/vortex-file-compressed 422206783 4.18795e+08 1.00815 ns
tpch_q03/vortex-file-compressed 281118634 2.76606e+08 1.01631 ns
tpch_q04/vortex-file-compressed 197566856 1.9735e+08 1.0011 ns
tpch_q05/vortex-file-compressed 351263799 3.36328e+08 1.04441 ns
tpch_q06/vortex-file-compressed 116929262 1.19863e+08 0.975526 ns
tpch_q07/vortex-file-compressed 383040718 3.88968e+08 0.984762 ns
tpch_q08/vortex-file-compressed 446356878 4.40612e+08 1.01304 ns
tpch_q09/vortex-file-compressed 391595164 3.89547e+08 1.00526 ns
tpch_q10/vortex-file-compressed 384176687 3.98045e+08 0.965158 ns
tpch_q11/vortex-file-compressed 160052228 1.56294e+08 1.02404 ns
tpch_q12/vortex-file-compressed 205903109 2.12569e+08 0.968642 ns
tpch_q13/vortex-file-compressed 178549495 1.73468e+08 1.02929 ns
tpch_q14/vortex-file-compressed 136056322 1.37913e+08 0.986535 ns
tpch_q15/vortex-file-compressed 302303030 3.05556e+08 0.989355 ns
tpch_q16/vortex-file-compressed 188250700 1.84416e+08 1.0208 ns
tpch_q17/vortex-file-compressed 201852460 2.12865e+08 0.948266 ns
tpch_q18/vortex-file-compressed 281819838 2.59974e+08 1.08403 ns
tpch_q19/vortex-file-compressed 199187823 1.95573e+08 1.01848 ns
tpch_q20/vortex-file-compressed 335804673 3.32338e+08 1.01043 ns
tpch_q21/vortex-file-compressed 484938363 4.74994e+08 1.02094 ns
tpch_q22/vortex-file-compressed 149268656 1.6136e+08 0.925066 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025 •
edited
Loading

Benchmarks: Clickbench on NVME

Table of Results
name PR b97b9f2 base 9f331e1 ratio (PR/base) unit
clickbench_q00/parquet 2656236 2.65939e+06 0.998814 ns
clickbench_q01/parquet 35414010 3.41856e+07 1.03593 ns
clickbench_q02/parquet 70541109 7.17033e+07 0.983792 ns
clickbench_q03/parquet 59588335 5.50326e+07 1.08278 ns
clickbench_q04/parquet 339702596 3.55589e+08 0.955323 ns
clickbench_q05/parquet 345251703 3.23446e+08 1.06742 ns
clickbench_q06/parquet 2548505 1.94148e+06 1.31266 ns
clickbench_q07/parquet 38506566 3.21745e+07 1.1968 ns
clickbench_q08/parquet 422737899 3.93006e+08 1.07565 ns
clickbench_q09/parquet 622516931 5.89922e+08 1.05525 ns
clickbench_q10/parquet 129919377 1.24223e+08 1.04586 ns
clickbench_q11/parquet 153909179 1.49992e+08 1.02611 ns
clickbench_q12/parquet 335097882 3.33239e+08 1.00558 ns
clickbench_q13/parquet 518189026 5.07543e+08 1.02098 ns
clickbench_q14/parquet 348342579 3.31522e+08 1.05074 ns
clickbench_q15/parquet 367879479 3.72468e+08 0.987681 ns
clickbench_q16/parquet 816549219 7.81671e+08 1.04462 ns
clickbench_q17/parquet 711023750 6.93433e+08 1.02537 ns
clickbench_q18/parquet 1615806081 1.63466e+09 0.988467 ns
clickbench_q19/parquet 45276197 4.65668e+07 0.972284 ns
clickbench_q20/parquet 605644668 5.81967e+08 1.04069 ns
clickbench_q21/parquet 648416835 6.48385e+08 1.00005 ns
clickbench_q22/parquet 1049632621 1.01556e+09 1.03355 ns
clickbench_q23/parquet 4233286161 4.05897e+09 1.04294 ns
clickbench_q24/parquet 215288991 2.0253e+08 1.063 ns
clickbench_q25/parquet 189478037 1.82817e+08 1.03643 ns
clickbench_q26/parquet 242186191 2.33935e+08 1.03527 ns
clickbench_q27/parquet 827786956 7.72837e+08 1.0711 ns
clickbench_q28/parquet 4880736540 4.7768e+09 1.02176 ns
clickbench_q29/parquet 270618408 2.58673e+08 1.04618 ns
clickbench_q30/parquet 349663776 3.42947e+08 1.01958 ns
clickbench_q31/parquet 394186450 3.80248e+08 1.03666 ns
clickbench_q32/parquet 1865440161 1.78958e+09 1.04239 ns
clickbench_q33/parquet 1594913210 1.54834e+09 1.03008 ns
clickbench_q34/parquet 1570131226 1.51439e+09 1.03681 ns
clickbench_q35/parquet 540899451 5.25953e+08 1.02842 ns
clickbench_q36/parquet 157022581 1.53831e+08 1.02075 ns
clickbench_q37/parquet 74398807 7.12327e+07 1.04445 ns
clickbench_q38/parquet 102125442 9.91129e+07 1.03039 ns
clickbench_q39/parquet 301221396 2.81491e+08 1.07009 ns
clickbench_q40/parquet 48636384 4.73886e+07 1.02633 ns
clickbench_q41/parquet 45674189 4.49148e+07 1.01691 ns
clickbench_q42/parquet 56744418 5.56779e+07 1.01915 ns
clickbench_q00/vortex-file-compressed 4616433 4.22226e+06 1.09336 ns
clickbench_q01/vortex-file-compressed 19997101 1.64884e+07 1.2128 ns
clickbench_q02/vortex-file-compressed 36129668 3.05251e+07 1.18361 ns
clickbench_q03/vortex-file-compressed 42781839 4.06043e+07 1.05363 ns
clickbench_q04/vortex-file-compressed 330976575 3.5486e+08 0.932697 ns
clickbench_q05/vortex-file-compressed 347887609 3.37546e+08 1.03064 ns
clickbench_q06/vortex-file-compressed 4698680 4.18671e+06 1.12228 ns
clickbench_q07/vortex-file-compressed 23153006 1.81192e+07 1.27782 ns
clickbench_q08/vortex-file-compressed 395170738 4.24105e+08 0.931775 ns
clickbench_q09/vortex-file-compressed 520232874 5.02995e+08 1.03427 ns
clickbench_q10/vortex-file-compressed 68326020 6.60657e+07 1.03421 ns
clickbench_q11/vortex-file-compressed 78531033 7.41136e+07 1.0596 ns
clickbench_q12/vortex-file-compressed 267400693 2.60301e+08 1.02727 ns
clickbench_q13/vortex-file-compressed 384635762 3.66537e+08 1.04938 ns
clickbench_q14/vortex-file-compressed 254321058 2.60563e+08 0.976044 ns
clickbench_q15/vortex-file-compressed 402151383 4.17499e+08 0.963239 ns
clickbench_q16/vortex-file-compressed 819637866 8.07829e+08 1.01462 ns
clickbench_q17/vortex-file-compressed 798899931 8.00281e+08 0.998274 ns
clickbench_q18/vortex-file-compressed 1378590932 1.33808e+09 1.03028 ns
clickbench_q19/vortex-file-compressed 30698855 2.55615e+07 1.20098 ns
clickbench_q20/vortex-file-compressed 476962746 4.32449e+08 1.10293 ns
clickbench_q21/vortex-file-compressed 491150442 4.43262e+08 1.10804 ns
clickbench_q22/vortex-file-compressed 630146769 5.98793e+08 1.05236 ns
clickbench_q23/vortex-file-compressed 1317039298 1.28007e+09 1.02888 ns
clickbench_q24/vortex-file-compressed 91281806 8.64505e+07 1.05588 ns
clickbench_q25/vortex-file-compressed 105020790 1.02517e+08 1.02442 ns
clickbench_q26/vortex-file-compressed 126554954 1.1816e+08 1.07105 ns
clickbench_q27/vortex-file-compressed 757366534 6.99274e+08 1.08308 ns
clickbench_q28/vortex-file-compressed 5599883720 5.31106e+09 1.05438 ns
clickbench_q29/vortex-file-compressed 275041030 2.4964e+08 1.10175 ns
clickbench_q30/vortex-file-compressed 240931313 2.32739e+08 1.0352 ns
clickbench_q31/vortex-file-compressed 251416343 2.41182e+08 1.04244 ns
clickbench_q32/vortex-file-compressed 1394931950 1.3242e+09 1.05342 ns
clickbench_q33/vortex-file-compressed 1453567430 1.4039e+09 1.03538 ns
clickbench_q34/vortex-file-compressed 1418277400 1.34507e+09 1.05443 ns
clickbench_q35/vortex-file-compressed 618576381 6.13977e+08 1.00749 ns
clickbench_q36/vortex-file-compressed 64478992 5.94661e+07 1.0843 ns
clickbench_q37/vortex-file-compressed 40190051 3.60688e+07 1.11426 ns
clickbench_q38/vortex-file-compressed 28494623 2.76737e+07 1.02966 ns
clickbench_q39/vortex-file-compressed 120628746 1.11403e+08 1.08281 ns
clickbench_q40/vortex-file-compressed 26192403 2.09268e+07 1.25162 ns
clickbench_q41/vortex-file-compressed 25746705 2.03894e+07 1.26275 ns
clickbench_q42/vortex-file-compressed 31746333 3.15281e+07 1.00692 ns

@AdamGS AdamGS mentioned this pull request Feb 20, 2025
Closed
@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Feb 20, 2025 •
edited
Loading

CodSpeed Performance Report

Merging #2417 will not alter performance

Comparing adamg/datafusion-46-branch (aab23a3) with develop (9f331e1)

Summary

✅ 775 untouched benchmarks

@AdamGS AdamGS requested a review from onursatici February 21, 2025 11:20
@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Feb 25, 2025 •
edited
Loading

Deploying vortex-bench with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3e2c75a
Status: ✅  Deploy successful!
Preview URL: https://a949d820.vortex-bench.pages.dev
Branch Preview URL: https://adamg-datafusion-46-branch.vortex-bench.pages.dev

View logs

onursatici
onursatici approved these changes Mar 3, 2025
View reviewed changes
vortex-datafusion/src/persistent/format.rs Outdated
fn file_source(&self) -> Arc<dyn FileSource> {
Arc::new(VortexSource::new(
self.footer_cache.clone(),
VortexExecMetrics::default(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are creating a new metrics container per call but as far as I see this is only used by datafusion for downcasting, so it should be fine

@robert3005
Copy link
Member

fwiw datafusion 46 is out

@AdamGS AdamGS marked this pull request as ready for review March 9, 2025 16:15
@AdamGS AdamGS changed the title [WIP] Datafusion 46 support Datafusion 46 support Mar 9, 2025
@AdamGS AdamGS added the benchmark Run benchmarks on this branch label Mar 9, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Mar 9, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 9, 2025

Benchmarks: random_access

Table of Results
name PR 6ecb15e base 9f331e1 ratio (PR/base) unit
random-access/vortex-tokio-local-disk 4536930 3.6962e+06 1.22746 ns
random-access/parquet-tokio-local-disk 237671576 2.35674e+08 1.00848 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 9, 2025

Benchmarks: compress

Table of Results
name PR 6ecb15e base 9f331e1 ratio (PR/base) unit
compress time/taxi throughput 0.293745 0.318358 0.922688 bytes/ns
parquet_rs-zstd compress time/taxi throughput 0.336101 0.366336 0.917468 bytes/ns
decompress time/taxi throughput 2.05347 2.13232 0.963022 bytes/ns
parquet_rs-zstd decompress time/taxi throughput 1.83627 1.98433 0.925385 bytes/ns
compress time/AirlineSentiment throughput 0.00323045 0.00381827 0.84605 bytes/ns
parquet_rs-zstd compress time/AirlineSentiment throughput 0.0576746 0.0616052 0.936198 bytes/ns
decompress time/AirlineSentiment throughput 0.0128051 0.0139299 0.919253 bytes/ns
parquet_rs-zstd decompress time/AirlineSentiment throughput 0.0922078 0.0944367 0.976398 bytes/ns
compress time/Arade throughput 0.211985 0.239195 0.886243 bytes/ns
parquet_rs-zstd compress time/Arade throughput 0.479843 0.524092 0.91557 bytes/ns
decompress time/Arade throughput 1.96872 2.15756 0.912475 bytes/ns
parquet_rs-zstd decompress time/Arade throughput 1.9332 2.07207 0.93298 bytes/ns
compress time/Bimbo throughput 0.551375 0.596863 0.923788 bytes/ns
parquet_rs-zstd compress time/Bimbo throughput 0.425078 0.463411 0.91728 bytes/ns
decompress time/Bimbo throughput 2.43373 2.61018 0.932399 bytes/ns
parquet_rs-zstd decompress time/Bimbo throughput 3.03312 3.26121 0.930062 bytes/ns
compress time/CMSprovider throughput 0.0642161 0.0702219 0.914474 bytes/ns
parquet_rs-zstd compress time/CMSprovider throughput 0.421241 0.459591 0.916556 bytes/ns
decompress time/CMSprovider throughput 4.57624 4.82146 0.94914 bytes/ns
parquet_rs-zstd decompress time/CMSprovider throughput 2.10423 2.28773 0.919791 bytes/ns
compress time/Euro2016 throughput 0.177567 0.192061 0.924533 bytes/ns
parquet_rs-zstd compress time/Euro2016 throughput 0.345533 0.372171 0.928424 bytes/ns
decompress time/Euro2016 throughput 3.35727 3.7491 0.895486 bytes/ns
parquet_rs-zstd decompress time/Euro2016 throughput 1.17589 1.27558 0.921843 bytes/ns
compress time/Food throughput 0.248863 0.273966 0.908371 bytes/ns
parquet_rs-zstd compress time/Food throughput 0.39116 0.428981 0.911836 bytes/ns
decompress time/Food throughput 6.43487 7.20261 0.893408 bytes/ns
parquet_rs-zstd decompress time/Food throughput 1.94258 2.11684 0.917678 bytes/ns
compress time/HashTags throughput 0.225188 0.244853 0.919685 bytes/ns
parquet_rs-zstd compress time/HashTags throughput 0.947074 1.01183 0.936 bytes/ns
decompress time/HashTags throughput 7.53622 8.39421 0.897787 bytes/ns
parquet_rs-zstd decompress time/HashTags throughput 3.28018 3.57385 0.917828 bytes/ns
compress time/TPC-H l_comment chunked throughput 0.240286 0.258248 0.930444 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked throughput 0.313111 0.339303 0.922807 bytes/ns
decompress time/TPC-H l_comment chunked throughput 3.98011 4.09856 0.971101 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked throughput 1.37648 1.52439 0.902973 bytes/ns
compress time/TPC-H l_comment canonical throughput 0.0318291 0.0344444 0.92407 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment canonical throughput 0.309153 0.337628 0.915661 bytes/ns
decompress time/TPC-H l_comment canonical throughput 4.39257 4.69208 0.936166 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical throughput 1.37996 1.52104 0.907247 bytes/ns
compress time/wide table cols=10 chunks=1 rows=1000 throughput 0.173176 0.18121 0.955664 bytes/ns
parquet_rs-zstd compress time/wide table cols=10 chunks=1 rows=1000 throughput 0.240631 0.250275 0.961465 bytes/ns
decompress time/wide table cols=10 chunks=1 rows=1000 throughput 0.847576 0.476448 1.77895 bytes/ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=1 rows=1000 throughput 0.522696 0.557055 0.93832 bytes/ns
compress time/wide table cols=100 chunks=1 rows=1000 throughput 0.170791 0.182348 0.936625 bytes/ns
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 throughput 0.228476 0.24686 0.925526 bytes/ns
decompress time/wide table cols=100 chunks=1 rows=1000 throughput 1.23615 1.44198 0.857258 bytes/ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 throughput 0.516414 0.559773 0.922542 bytes/ns
compress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.162846 0.179628 0.906575 bytes/ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.21315 0.242205 0.880042 bytes/ns
decompress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.955398 1.01911 0.937479 bytes/ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.489688 0.551662 0.887659 bytes/ns
compress time/wide table cols=10 chunks=50 rows=1000 throughput 0.0894774 0.0943711 0.948144 bytes/ns
parquet_rs-zstd compress time/wide table cols=10 chunks=50 rows=1000 throughput 0.165779 0.172426 0.961451 bytes/ns
decompress time/wide table cols=10 chunks=50 rows=1000 throughput 0.838107 0.545271 1.53705 bytes/ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=50 rows=1000 throughput 0.513581 0.573166 0.896042 bytes/ns
compress time/wide table cols=100 chunks=50 rows=1000 throughput 0.0861333 0.0927617 0.928544 bytes/ns
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 throughput 0.156215 0.166722 0.936981 bytes/ns
decompress time/wide table cols=100 chunks=50 rows=1000 throughput 1.16826 1.44746 0.807116 bytes/ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 throughput 0.5291 0.571058 0.926527 bytes/ns
compress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.0699323 0.0852127 0.82068 bytes/ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.125104 0.139929 0.89405 bytes/ns
decompress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.95582 1.04846 0.911646 bytes/ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.501524 0.562071 0.892278 bytes/ns
vortex:raw size/taxi 0.119399 0.119399 1 ratio
vortex size/taxi 5.9079e+07 5.9079e+07 1 bytes
vortex:parquet-zstd size/taxi 1.05572 1.05572 1 ratio
vortex:raw size/AirlineSentiment 1.35456 1.35456 1 ratio
vortex size/AirlineSentiment 4424 4424 1 bytes
vortex:parquet-zstd size/AirlineSentiment 4.57497 4.57497 1 ratio
vortex:raw size/Arade 0.255857 0.255857 1 ratio
vortex size/Arade 3.03621e+08 3.03621e+08 1 bytes
vortex:parquet-zstd size/Arade 0.994197 0.994197 1 ratio
vortex:raw size/Bimbo 0.116911 0.116911 1 ratio
vortex size/Bimbo 8.35811e+08 8.35811e+08 1 bytes
vortex:parquet-zstd size/Bimbo 2.15331 2.15331 1 ratio
vortex:raw size/CMSprovider 0.18503 0.18503 1 ratio
vortex size/CMSprovider 1.16293e+09 1.16293e+09 1 bytes
vortex:parquet-zstd size/CMSprovider 1.51124 1.51124 1 ratio
vortex:raw size/Euro2016 0.471335 0.471335 1 ratio
vortex size/Euro2016 2.14468e+08 2.14468e+08 1 bytes
vortex:parquet-zstd size/Euro2016 1.80392 1.80392 1 ratio
vortex:raw size/Food 0.177323 0.177323 1 ratio
vortex size/Food 5.973e+07 5.973e+07 1 bytes
vortex:parquet-zstd size/Food 1.64863 1.64863 1 ratio
vortex:raw size/HashTags 0.142791 0.142791 1 ratio
vortex size/HashTags 2.73577e+08 2.73577e+08 1 bytes
vortex:parquet-zstd size/HashTags 2.04215 2.04215 1 ratio
vortex:raw size/TPC-H l_comment chunked 0.418167 0.418718 0.998682 ratio
vortex size/TPC-H l_comment chunked 1.04206e+08 1.04343e+08 0.998682 bytes
vortex:parquet-zstd size/TPC-H l_comment chunked 1.83021 1.83262 0.998686 ratio
vortex:raw size/TPC-H l_comment canonical 0.425488 0.425482 1.00001 ratio
vortex size/TPC-H l_comment canonical 1.06028e+08 1.06026e+08 1.00001 bytes
vortex:parquet-zstd size/TPC-H l_comment canonical 1.86219 1.8621 1.00005 ratio
vortex:raw size/wide table cols=10 chunks=1 rows=1000 0.626224 0.626224 1 ratio
vortex size/wide table cols=10 chunks=1 rows=1000 100256 100256 1 bytes
vortex:parquet-zstd size/wide table cols=10 chunks=1 rows=1000 1.07244 1.07244 1 ratio
vortex:raw size/wide table cols=100 chunks=1 rows=1000 0.623267 0.623267 1 ratio
vortex size/wide table cols=100 chunks=1 rows=1000 997736 997736 1 bytes
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000 1.06732 1.06732 1 ratio
vortex:raw size/wide table cols=1000 chunks=1 rows=1000 0.622971 0.622971 1 ratio
vortex size/wide table cols=1000 chunks=1 rows=1000 9.97254e+06 9.97254e+06 1 bytes
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000 1.06681 1.06681 1 ratio
vortex:raw size/wide table cols=10 chunks=50 rows=1000 0.6098 0.6098 1 ratio
vortex size/wide table cols=10 chunks=50 rows=1000 100256 100256 1 bytes
vortex:parquet-zstd size/wide table cols=10 chunks=50 rows=1000 1.07244 1.07244 1 ratio
vortex:raw size/wide table cols=100 chunks=50 rows=1000 0.608224 0.608224 1 ratio
vortex size/wide table cols=100 chunks=50 rows=1000 997736 997736 1 bytes
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000 1.06732 1.06732 1 ratio
vortex:raw size/wide table cols=1000 chunks=50 rows=1000 0.608066 0.608066 1 ratio
vortex size/wide table cols=1000 chunks=50 rows=1000 9.97254e+06 9.97254e+06 1 bytes
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000 1.06681 1.06681 1 ratio

@AdamGS AdamGS merged commit f514610 into develop Mar 9, 2025
33 of 34 checks passed
@AdamGS AdamGS deleted the adamg/datafusion-46-branch branch March 9, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@onursatici onursatici onursatici approved these changes

Assignees
No one assigned
Labels
do not merge Pull requests that are not intended to merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@AdamGS @robert3005 @onursatici @0ax1