Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use opendal in rattler-index and add executable #1076

Merged
merged 38 commits into from
Feb 27, 2025
Merged

feat: Use opendal in rattler-index and add executable #1076

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
d0085a3
Init opendal
delsner Jan 28, 2025
8c58912
Merge branch 'main' of github.com:pavelzw/rattler into rattler-index-…
delsner Feb 19, 2025
f326733
wip: Add index function
delsner Feb 19, 2025
76afe88
wip
pavelzw Feb 19, 2025
87b4d84
wip
delsner Feb 20, 2025
a347827
Initial version
delsner Feb 20, 2025
dd01d4f
don't fail on not found repodata
pavelzw Feb 20, 2025
ac8ea01
add progress bars
pavelzw Feb 20, 2025
554f00b
Finalize implementation
delsner Feb 20, 2025
8a99102
Fixes
delsner Feb 20, 2025
ab1b0a3
Fixes
delsner Feb 20, 2025
59a0bdc
Prune cargo toml
delsner Feb 20, 2025
563c66e
Fix comment
delsner Feb 20, 2025
6b8a13a
Remove tracing log
delsner Feb 20, 2025
52fae88
Merge branch 'main' into rattler-index-opendal
delsner Feb 20, 2025
1344176
Update tests and pyrattler
delsner Feb 20, 2025
131138a
fix lint
pavelzw Feb 20, 2025
7341218
Fix docstrings
delsner Feb 20, 2025
efe483c
Add s3 integration test to pyrattler
delsner Feb 20, 2025
39713a8
Remove assert_cmd
delsner Feb 20, 2025
b1a5bb2
Add finish print
delsner Feb 20, 2025
63a09e0
Apply suggestions from code review
delsner Feb 20, 2025
b1425d2
Review
delsner Feb 20, 2025
6e7add6
Update crates/rattler_index/src/lib.rs
delsner Feb 23, 2025
e38dc2d
Merge branch 'main' into rattler-index-opendal
delsner Feb 23, 2025
7b7261a
add rustls-tls native certs
pavelzw Feb 24, 2025
c7785b8
rustls and native-tls
pavelzw Feb 24, 2025
bf8445d
Merge remote-tracking branch 'upstream/main' into rattler-index-opendal
pavelzw Feb 26, 2025
d010241
fix
pavelzw Feb 26, 2025
a869c2d
fix
pavelzw Feb 26, 2025
c592135
Merge branch 'main' into rattler-index-opendal
wolfv Feb 26, 2025
57d52c3
clippy
delsner Feb 26, 2025
867a49d
Fix clippy
delsner Feb 26, 2025
e2829f0
bump rust in py-rattler as well?
pavelzw Feb 26, 2025
cdb5daa
Merge remote-tracking branch 'upstream/main' into rattler-index-opendal
pavelzw Feb 27, 2025
0133a55
add gitattributes
pavelzw Feb 27, 2025
ded2ee7
lockfile
pavelzw Feb 27, 2025
8fc824c
add keys
pavelzw Feb 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ chrono = { version = "0.4.39", default-features = false, features = [
"alloc",
] }
clap = { version = "4.5.29", features = ["derive"] }
clap-verbosity-flag = "3.0.1"
cmake = "0.1.54"
console = { version = "0.15.10", features = ["windows-console-colors"] }
criterion = "0.5"
Expand All @@ -67,6 +68,7 @@ futures = "0.3.31"
futures-util = "0.3.31"
fxhash = "0.2.1"
# lots of other crates are still stuck on older version which breaks `deserialize`

generic-array = "0.14.7"
getrandom = { version = "0.3.1", default-features = false }
glob = "0.3.2"
Expand Down Expand Up @@ -105,6 +107,7 @@ memmap2 = "0.9.5"
netrc-rs = "0.1.2"
nom = "7.1.3"
num_cpus = "1.16.0"
opendal = { version = "0.51.2" }
once_cell = "1.20.3"
ouroboros = "0.18.5"
parking_lot = "0.12.3"
Expand Down Expand Up @@ -161,6 +164,7 @@ tokio-util = "0.7.13"
tower = { version = "0.5.2", default-features = false }
tower-http = { version = "0.6.2", default-features = false }
tracing = "0.1.41"
tracing-log = "0.2.0"
tracing-subscriber = { version = "0.3.19", default-features = false }
tracing-test = { version = "0.2.5" }
trybuild = { version = "1.0.103" }
Expand Down
28 changes: 22 additions & 6 deletions crates/rattler_index/Cargo.toml
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we still need to do the whole native-tls / rustls-tls and native-roots dance

Copy link
Contributor

@pavelzw pavelzw Feb 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in c7785b8 (#1076)

❯ cargo tree --package rattler_index --features rustls-tls --no-default-features | grep native
│   │   │   │   ├── rustls-native-certs v0.8.1
│   │   │   ├── rustls-native-certs v0.8.1 (*)
❯ cargo tree --package rattler_index --features native-tls --no-default-features -i rustls
warning: nothing to print.

To find dependencies that require specific target platforms, try to use option `--target all` first, and then narrow your search scope accordingly.

Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,38 @@ name = "rattler_index"
version = "0.20.12"
edition.workspace = true
authors = []
description = "A crate that indexes directories containing conda packages to create local conda channels"
description = "A crate to index conda channels and create a repodata.json file."
categories.workspace = true
homepage.workspace = true
repository.workspace = true
license.workspace = true
readme.workspace = true
default-run = "rattler-index"

[[bin]]
name = "rattler-index"
path = "src/main.rs"
Comment on lines +31 to +33
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would suggest we don't publish binaries for rattler-index in this repo as this will probably complicate the release-plz workflow in this repo quite a bit. instead, we can only build binaries on conda-forge (and other package managers like brew), imo this should be enough


[dependencies]
anyhow = { workspace = true }
clap = { workspace = true, features = ["derive", "env"] }
clap-verbosity-flag = { workspace = true }
console = { workspace = true }
fs-err = { workspace = true }
rattler_conda_types = { path="../rattler_conda_types", version = "0.30.3", default-features = false }
rattler_digest = { path="../rattler_digest", version = "1.0.6", default-features = false }
rattler_package_streaming = { path="../rattler_package_streaming", version = "0.22.28", default-features = false }
fxhash = { workspace = true }
futures = { workspace = true }
indicatif = { workspace = true }
opendal = { workspace = true, features = ["services-s3", "services-fs"] }
rattler_networking = { path = "../rattler_networking", version = "0.22.4", default-features = false }
rattler_conda_types = { path = "../rattler_conda_types", version = "0.30.3", default-features = false }
rattler_digest = { path = "../rattler_digest", version = "1.0.6", default-features = false }
rattler_package_streaming = { path = "../rattler_package_streaming", version = "0.22.28", default-features = false }
serde_json = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["full"] }
tracing = { workspace = true }
walkdir = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter", "fmt"] }
tracing-log = { workspace = true }
url = { workspace = true }

[dev-dependencies]
tempfile = { workspace = true }
Expand Down
Loading
Loading