[DISCUSS] publishing nightly / non-official releases to crates.io (even more frequent releases)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
DataFusion's mission statement is to be "easy to embed" from https://arrow.apache.org/datafusion/user-guide/introduction.html#why-datafusion

Within the rust ecosystem, publishing to crates.io is the key way to make something easy to embed (other projects can then use it with a single line of toml). Thanks to @andygrove and other's work, we now are publishing official versions about once a month ❤️ (e.g. #4776 )

However, releasing even every month is still slower than optimal, as sometimes there are changes / bugs to the interfaces which aren't caught until after the release (e.g. most recently necessitating a patch release #4948)

Many active users of DataFusion today still use a fork or work off the master branch directly but this is not always possible for all projects as it is somewhat challenging.

Having more frequently published versions on crates.io would help others integrate datafusion more quickly between major official releases

Describe the solution you'd like
I propose publishing "nightly" releases of DataFusion to crates.io, much like the rust compiler itself.

These releases would be clearly labeled as "NON OFFICIAL RELEASES" perhaps with versions like `17.0.0.dev-2023-01-22

Describe alternatives you've considered
We could also try to run the official release process more frequently (every 2 weeks perhaps) -- given the voting process takes at least three days it will still be limited in how quickly it can be done

Additional context
See #2327 for prior discussion

[andygrove]

+1 for this. There is precedent for nightly builds in Arrow and other Apache projects:

• https://nightlies.apache.org/

I am also open to increasing the frequency of official releases to every two weeks. The process is mainly automated (except for the vote), and I could automate it more. The only painful part is generating the changelog.

[andygrove]

There was a discussion about publishing nightly releases to crates.io ~5 years ago and the conclusion at the time was that this was not allowed, so we should also have a discussion about this on the mailing list.

https://lists.apache.org/thread/o4328kdrhc9yodkvw5flop4wjsbqfxv3

[andygrove]

There is another option here. The approach we have adopted in the Python bindings, and in Ballista, is to have the main/master branch depend on the other Arrow repos via GitHub rather than using a crates.io release. This allows us to PR as often as we need to to update the rev being used in order to pull in the latest changes. Once there is an official release of the dependency, then we switch back to using that and create a release, then switch back to using GitHub revs.

[alamb]

This allows us to PR as often as we need to to update the rev being used in order to pull in the latest changes.

This is effectively what we do in IOx as well:

https://github.com/influxdata/influxdb_iox/blob/840923abab56f5f5f7fedf2a622e5f976a3d9eac/Cargo.toml#L116-L122

[andygrove]

Pinging @avantgardnerio as this is a pain point for him.

[andygrove]

Somewhat related to this subject, I would like to move to having release branches rather than releasing from master. I am trying this approach out for the next release of the Python bindings. This allows to have a stable branch-0.8 where we can continue to release patch releases from and keep master pointing to latest DataFusion.

[andygrove]

Here is a draft PR explaining the branching policy and release process that I am proposing we adopt in DataFusion:

apache/datafusion-python#149

[avantgardnerio]

I don't think crates.io releases would impact us much - either for better or worse.

There is another option here. The approach we have adopted in the Python bindings

I think this approach would be very helpful, but I see we're just trying it out now with Ballista? I think we should "let it soak" over there for a few releases, and if everyone is happy try it with DataFusion as well.

[findepi]

Related: #14428