feat(js): compile rattler_solve and rattler_repodata_gateway
#1108
Conversation
| @@ -1,8 +1,12 @@ | |||
| #[cfg(not(target_arch = "wasm32"))] | |||
There was a problem hiding this comment.
rattler_cache is basically a library to cache stuff to disk. Most of the functionality is gated to not be available on wasm32.
| .url(url.clone()) | ||
| .body(body.to_string()) | ||
| .unwrap(), | ||
| ) | ||
| } | ||
|
|
||
| #[cfg(target_arch = "wasm32")] | ||
| pub(crate) fn create_404_response(_url: &Url, _body: &str) -> Response { | ||
| todo!("This is not implemented in reqwest, we need to contribute that.") |
There was a problem hiding this comment.
This is an issue, that might come up when we add support for middleware.
| // | ||
| // Let's start by fetching the records. If an error occurs we immediately return the error. | ||
| // This will drop the sender and all other waiting tasks will receive an error. | ||
| let records = match tokio::spawn({ |
There was a problem hiding this comment.
I removed the spawn here because I dont think its needed. All implementations fan out to spawn blocking when blocking operations are performed.
| // self-referential field which indexes the data in the memory map with a | ||
| // sparsely parsed json struct. See [`LazyRepoData`]. | ||
| #[cfg(any(unix, windows))] | ||
| self_cell::self_cell!( |
There was a problem hiding this comment.
I replaced ouroborous with self_cell because I couldnt get to conditionally compile ouroboros. self_cell should also compile faster anyway.
js-rattler/Cargo.toml
Outdated
| # We want to use version 0.4.19 but its not released yet. | ||
| # https://github.com/Nullus157/async-compression/pull/319 | ||
| async-compression = { git = "https://github.com/Nullus157/async-compression.git" } | ||
| bzip2 = { git = "https://github.com/baszalmstra/bzip2-rs.git", branch = "fix/wasm" } |
There was a problem hiding this comment.
I need to upstream this before merging.
There was a problem hiding this comment.
There was a problem hiding this comment.
Yeah, now all we need is an async compression release! But we can merge without it
There was a problem hiding this comment.
Want to undraft it and we merge?
|
I merged main in preparation of merging this! |
This PR adds the ability to use both
rattler_solveandrattler_repodata_gateway.WASM does not support filesystem access without extensions like emscripten or wasi, neither are supported by our build. But rattler aggressively caches data to disk so this PR adds additional code paths to not cache anything (except for maybe in memory). This allows compiling rattler_repodata_gateway to WASM.
Its also not possible to spawn futures or run blocking code on a different thread. Concurrency is supported, and its also used. But spawning is not supported because there is no tokio runtime active. I worked around this by conditionally running blocking code on the current thread.
I did not yet expose these crates to the javascript API yet, that should be done in a separate PR. I did add a proof of concept that fetches repodata and performs a solve from javascript. This also supports sharded repodata!
Running a solve for
python,matplotlib,numpyforemscripten-wasm32in the browser takes 0.2 seconds (without any cache) and results in this nice view in the chrome network debugger:(Uses
https://prefix.dev/emscripten-forge-devandhttps://prefix.dev/conda-forge)