feat: Use opendal in rattler-index and add executable
#1076
Conversation
|
Exciting!! |
|
showcase: rattler-index.mp4 |
| access_key_id = os.environ.get("RATTLER_TEST_R2_ACCESS_KEY_ID") | ||
| if not access_key_id: | ||
| pytest.skip("RATTLER_TEST_R2_ACCESS_KEY_ID environment variable is not set") | ||
| secret_access_key = os.environ.get("RATTLER_TEST_R2_SECRET_ACCESS_KEY") | ||
| if not secret_access_key: | ||
| pytest.skip("RATTLER_TEST_R2_SECRET_ACCESS_KEY environment variable is not set") |
There was a problem hiding this comment.
These are already set but I think they are read only. Do you want these secrets to be read-write?
| [[bin]] | ||
| name = "rattler-index" | ||
| path = "src/main.rs" |
There was a problem hiding this comment.
i would suggest we don't publish binaries for rattler-index in this repo as this will probably complicate the release-plz workflow in this repo quite a bit. instead, we can only build binaries on conda-forge (and other package managers like brew), imo this should be enough
| ] | ||
| rustls-tls = [ | ||
| "rattler_redaction/rustls-tls", | ||
| "rattler_package_streaming/rustls-tls", |
There was a problem hiding this comment.
this is not used in rattler_conda_types, correct?
| default = ["rayon"] | ||
| default = ["rayon", "rustls-tls"] | ||
| native-tls = [ | ||
| "rattler_redaction/native-tls", |
| rattler_macros = { path = "../rattler_macros", version = "1.0.6", default-features = false } | ||
| rattler_redaction = { version = "0.1.6", path = "../rattler_redaction", default-features = false } |
There was a problem hiding this comment.
is default-features = false okay here? otherwise this will pull in reqwest stuff
|
not sure in what way the python failures are related or not, but conda-forge/rust-activation-feedstock#72 is the conda-forge update to 1.85.0 |
|
Things are fixed on main now but looks like I broke this PR. Sorry :( |
| use clap_verbosity_flag::Verbosity; | ||
| use rattler_conda_types::Platform; | ||
| use rattler_index::{index_fs, index_s3}; | ||
| use tracing_log::AsTrace; |
| let cli = Cli::parse(); | ||
|
|
||
| tracing_subscriber::FmtSubscriber::builder() | ||
| .with_max_level(cli.verbose.log_level_filter().as_trace()) |
There was a problem hiding this comment.
You can enable the tracing feature on clap_verbose_... and then use the right function.
There was a problem hiding this comment.
| console::style(filename.clone()).dim() | ||
| )); | ||
| let file_path = format!("{subdir}/{filename}"); | ||
| let buffer = op.read(&file_path).await?; |
There was a problem hiding this comment.
The returning Buffer implements bytes::Buf so we can use:
package_record_from_tar_bz2_reader(buffer.reader())In this way, we can avoid some extra copy of data.
There was a problem hiding this comment.
thanks! implemented in #1123
Description
Final piece of work for implementing #960.