-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Web Client #437
Add Web Client #437
Conversation
Awesome work! Thank you! I'll start reviewing this early next week, but to make one comment:
I actually think it is OK to keep |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good Job with this implementation! Left a few comments.
@@ -150,7 +150,7 @@ impl<N: NodeRpcClient, R: FeltRng, S: Store, A: TransactionAuthenticator> Client | |||
&mut self.rpc_api | |||
} | |||
|
|||
#[cfg(any(test, feature = "testing"))] | |||
#[cfg(any(test, feature = "testing", feature = "idxdb"))] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like exposing the store (even just for idxdb
) may let the user access store methods which are only intended to be used by the client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree; I don't like this either. It currently is an unfortunate consequence of the get_signature method of the StoreAuthenticator
being synchronous. This is a problem because in the context of indexedDB, the store method method this function calls should be asynchronously called. Because it's not (and very difficult to change), we had to expose this method on the store that is asynchronous, and basically serves the purpose of fetching the account auth info in an async manner, and caches that info in memory. Then, when get_signature
is called during a transaction request later downstream, the store call made to get_account_auth_by_pub_key simply retrieves the auth info from this cached object in a synchronous manner.
In any case, for now, the quick solution is to allow my web-client to access the indexedDB store directly so that I may be able to call the function that caches the auth object before a transaction call. This change is found in this PR in this file crates/web-client/src/account.rs
.
In the future, the hope is that we can collaborate to find a better solution in general to the get_signature
method relying on the store in general, especially in a synchronous manner. My current idea is that beyond keeping a WebClient IndexedDB store that the client needs to run its operations smoothly, the application (in this case the wallet), can keep a separate store and organize data as they see fit. This could include storing auth or private info in an encrypted manner for the application, and then passing the auth directly to the client. This would eliminate the need to access the store directly and then we could keep the store private to the client for all cases.
Sorry for the long explanation, but this one required some context! 😅
@@ -15,7 +15,7 @@ export async function insertBlockHeader( | |||
blockNum: blockNum, | |||
header: header, | |||
chainMmrPeaks: chainMmrPeaks, | |||
hasClientNotes: hasClientNotes | |||
hasClientNotes: hasClientNotes.toString() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small question, why were booleans changed to strings? Did they not work with idxdb
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great question and something I stumbled into by accident. Whenever you have a query for indexedDB like this
const allMatchingRecords = await blockHeaders
.where('hasClientNotes')
.equals("true")
.toArray();
where you need to query a store via a specific field, you must add that field as an index when creating the schema. As such, indexedDB does not recognize certain data types like booleans (or blobs, I believe), as valid keys. So in this case, I decided to store the booleans instead as strings and wherever I found myself needing to index something that the SQLite implementation stores as a blob, I used a base64 encoded string for those.
plugins: [ | ||
rust({ | ||
cargoArgs: [ | ||
"--features", "testing", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want this feature always on for the web client?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, we don't. For context, my colleague, John, and I are planning on publishing a "test" package and a "production" package for now for this reason. The reason we currently are using this one is because when creating a new account without the testing
feature, the proof of work takes an incredibly long time to finish on a web browser. I think @bobbinth and I have discussed this briefly, but haven't found a good solution to it yet. As such, we think it'll be most helpful for teams to use the test version of the npm package first, and we can publish a production version separately so teams get a feel for the current state of the "real" times needed to create accounts, execute proofs, etc. in the browser.
crates/web-client/src/sync.rs
Outdated
|
||
#[wasm_bindgen] | ||
impl WebClient { | ||
pub async fn sync_state(&mut self) -> Result<JsValue, JsValue> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the current interface we don't have the ability to update ignored notes in the WebClient
. Maybe we could add a boolean parameter stating whether we want to update them or not (like the cli does) or add a separate method (like the one in the client).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
crates/web-client/src/tags.rs
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we could add methods for getting the tracked tags/removing them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
crates/rust-client/src/lib.rs
Outdated
/// | ||
/// TODO: we should make the function in base public and once that gets released use that one and | ||
/// delete this implementation. | ||
pub fn build_swap_tag( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like this function and get_input_note_with_id_prefix
don't belong here.
Maybe a better place for the former is crates/rust-client/src/transactions/mod.rs
or crates/rust-client/src/sync/tags.rs
and the latter could go in crates/rust-client/src/notes/mod.rs
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense to me. Went ahead and moved them and updated the paths to anything that imports them 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome work. Left some comments. Some more general things to discuss:
- I don't have a strong opinion on leaving the web store and RPC clients within
rust-client
vs the new web crate. I think ideally if we were to leave them where they are, if both feature flags are turned on, it wouldn't fail to compile (which I think is not the case right now, due to conflicting names) - I didn't take a deep look yet at some sections of the code (specifically the client's API). Something that caught my eye are the
New*
struct names which we probably want to change, but as you mentioned these refinements can come later. - The build seems to be broken. After a quick look I wasn't able to figure out why precisely but I can investigate further if you need help.
crates/rust-client/src/lib.rs
Outdated
@@ -12,7 +12,7 @@ mod errors; | |||
pub mod notes; | |||
pub mod rpc; | |||
pub mod store; | |||
mod store_authenticator; | |||
pub mod store_authenticator; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this needed? I think it might not be (StoreAuthenticator
is re-exported)
@@ -150,7 +150,7 @@ impl<N: NodeRpcClient, R: FeltRng, S: Store, A: TransactionAuthenticator> Client | |||
&mut self.rpc_api | |||
} | |||
|
|||
#[cfg(any(test, feature = "testing"))] | |||
#[cfg(any(test, feature = "testing", feature = "idxdb"))] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this needed? It might not be ideal if users can access the store (especially if it's based on the idxdb
flag
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great question! This is something that @tomyrd also brought up. I replied here #437 (comment). If there is a different workaround solution I'm happy to change this, but this was my bandaid for now 🤔
@@ -15,7 +15,7 @@ export async function insertBlockHeader( | |||
blockNum: blockNum, | |||
header: header, | |||
chainMmrPeaks: chainMmrPeaks, | |||
hasClientNotes: hasClientNotes | |||
hasClientNotes: hasClientNotes.toString() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of curiosity, why was this changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tomyrd also had the same question. Gave a reply here as well! #437 (comment)
crates/rust-client/src/lib.rs
Outdated
/// | ||
/// TODO: we should make the function in base public and once that gets released use that one and | ||
/// delete this implementation. | ||
pub fn build_swap_tag( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe not for this PR, but if this was moved up to the library level, should the CLI version be deleted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Went ahead and deleted this and the get_input_note_with_id_prefix
from the bin
crate and updated imports to point to the implementations defined in the rust-client
👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice readme! I think there are some stylistic nits ("miden vm" -> "Miden VM"); tagging @Dominik1999 in case he wants to take a look
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. I just checked the README.md
and added some nit comments. Really cool work!
```typescript | ||
import { WebClient } from "@demox-labs/miden-sdk"; | ||
|
||
const webClient = new WebClient(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the WebClient automatically run with --features testing
? The WebClient should have the same defaults as the Rust client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I explained a bit here #437 (comment), but when it comes to building the web-client crate, you can build with any combination of the features specified in the Cargo.toml
file. However, when it comes to publishing an actual NPM package, I think it may have to be based on a specific set of features. This is because we have to build the crate, create a WASM, and publish this code as a package for others to consume directly via a package.json. I think for this reason, the easiest route for now may be to publish a "test" version of the miden-client which is build with the "testing" feature, and a "production" version of the miden-client which is built with the default features. Another idea would be to publish a single NPM package, but include multiple build artifacts that reflect the different ways in which the crate could be built. Let me sync up with John and we will create a plan for this ASAP!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot! I think a production
mode for the WebClient
wouldn't work right now. The PoW in the browser takes too long. So maybe we should default to testing
? What do you think @bobbinth ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah - testing
should be the default for now. This will change after we refactor how account ID generation works.
crates/web-client/README.md
Outdated
const webClient = new WebClient(); | ||
|
||
// Creates the internal client of a previously instantiated WebClient. | ||
// Can provide `node_url` as an optional parameter. Defaults to "http://localhost:57291". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you default the WebClient to our remote testnet Node? 18.203.155.106
It should have the same defaults as the Rust client
crates/web-client/README.md
Outdated
|
||
// Creates the internal client of a previously instantiated WebClient. | ||
// Can provide `node_url` as an optional parameter. Defaults to "http://localhost:57291". | ||
// See https://github.com/0xPolygonMiden/miden-node for setting up and running your own node locally |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be better not to ask ppl to run a Miden node locally. We can do that for testing, but let's try to simplify it as much as we can for users
/** | ||
* Creates a new wallet account. | ||
* | ||
* @param storage_mode String. Either "OffChain" or "OnChain". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we consistently rename those to Private
and Public
? @bobbinth ?
I can open an issue to make the changes everywhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we already made these renamings in miden-base
and should propagate them everywhere.
crates/web-client/README.md
Outdated
``` | ||
|
||
### Transactions | ||
You can use the webClient to facilitate transactions between accounts. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In line 51
you capitalize WebClient
, here you use lower case. Can you make it consistent?
crates/web-client/README.md
Outdated
* @param {string} amount | ||
* @returns {Promise<NewTransactionResult>} | ||
* | ||
* Example of NewTransactionResult object: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: of a
crates/web-client/README.md
Outdated
* @param {string | undefined} [recall_height] | ||
* @returns {Promise<NewTransactionResult>} | ||
* | ||
* Example of NewTransactionResult object: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: of a
crates/web-client/README.md
Outdated
* @param {(string)[]} list_of_notes | ||
* @returns {Promise<NewTransactionResult>} | ||
* | ||
* Example of NewTransactionResult object: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: of a
crates/web-client/README.md
Outdated
* @param {string} note_type | ||
* @returns {Promise<NewSwapTransactionResult>} | ||
* | ||
* Example of NewSwapTransactionResult object: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: of a
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is so cool, to see that working! Congrats
@igamigo, @tomyrd, @Dominik1999, @bobbinth first round of feedback has been addressed! Will work now on trying to get these builds passing. Can also address any more feedback that is generated! |
I've been checking the build errors for the workspace and the underlying problem is that the When Cargo tries to build the whole workspace it adds all the features together assuming they are additive and it breaks. An example of these problems can be seen here where activating both features causes a collision in the name. I don't know what the correct solution would be in this case, some things that come to mind are:
|
Hey @tomyrd thanks for investigating! I think messing with the generation of the two sets of files is tricky. I actually gave this a shot in a different PR where I tried basically answering the question "can I generate the files for everything except the actual client ( If we want to keep the current crate structure (which I think seems to be the preference checking the comments through the PR), then another easy solution would be to duplicate the I think putting everything web-related in one crate is a similar approach to previous approach and would involve duplicating the |
Yeah, I think we might have to do this. Most alternatives are hacky (macros, build scripts, etc.), and it does not seem like we could improve this with features. |
I'm fine with doing w/e is easier at this point, but in the longer run, we should probably not allow building with both features enabled (and update the Makefile commands accordingly). One way to make this explicit is something like this: #[cfg(all(feature = "tonic", feature = "web-tonic"))]
compile_error!("features `tonic` and `web-tonic` are mutually exclusive");
#[cfg(all(feature = "sqlite", feature = "idxdb"))]
compile_error!("features `sqlite` and `idxdb` are mutually exclusive"); |
Ok I just duplicated the |
fab4ec7
to
9837ded
Compare
Ok small update. After taking a look at the github actions that were failing, the main reason they were failing is because now that this repo has a workspace structure, commands like these in the
were causing all the crates to build. The result of this (I think) is that there is a conflict of stores and clients built with and without async. The reason I think this is due to errors like this
which signals to me that for some reason, the A downstream effect of modifying the |
Yeah, I think as @tomyrd mentioned it adds up all the features and so the CLI is built with web versions of the store and tonic client which means it enables Docs are failing to build due to a similar cause (happened with the web tonic client PR as well), it seems like. I think we should try to find a way to make the build less fragmented in the future. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! I left some comments but in general I believe we could merge this and address work in future PRs. We should track this work on an issue. Some of the things that I can think of off the top of my head that we might want to tackle next:
- Adding CI checks for the web client (if not done on this PR)
- Cleaning up (and removing unwraps, etc.) of the API to expose methods more similar to the client library
- Removing the need of exposing the client store (might be better to open an issue on
miden-base
for this, since it's related toTransactionAuthenticator
) - Finding a way of running existing integration tests with the web client
Ideally something else that would be great to have is to not fail to compile when std and wasm versions of the components are included at the same time, but this might not be easily achievable, or maybe not without changing the project structure.
Makefile
Outdated
.PHONY: clippy-wasm | ||
clippy-wasm: ## Runs Clippy with configs | ||
cargo +nightly clippy --workspace --exclude miden-client --exclude miden-cli --exclude miden-client-tests --all-targets --features $(FEATURES_CLI) -- -D warnings |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add these to the CI checks? We can do it in a follow up or in this PR, as you want
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, should it run clippy just for web-client
instead of excluding the other projects or is there any reason why this is preferrable?
Makefile
Outdated
|
||
build-wasm: ## Builds the client library for wasm32 | ||
cargo build --target wasm32-unknown-unknown --features idxdb,web-tonic --no-default-features --package miden-client | ||
cargo build --workspace --exclude miden-client --exclude miden-cli --features $(FEATURES_WEB_CLIENT) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could just build the web client here instead of building the workspace and excluding the other projects, no?
Makefile
Outdated
cargo +nightly clippy --workspace --exclude miden-client-web --all-targets --features $(FEATURES_CLI) -- -D warnings | ||
|
||
.PHONY: clippy-wasm | ||
clippy-wasm: ## Runs Clippy with configs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: The comment could be updated to reflect the fact that it's mainly for web-client
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Left a couple of comments/questions.
Makefile
Outdated
|
||
.PHONY: clippy-wasm | ||
clippy-wasm: ## Runs Clippy with configs | ||
cargo +nightly clippy --workspace --exclude miden-client --exclude miden-cli --exclude miden-client-tests --all-targets --features $(FEATURES_CLI) -- -D warnings |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should use FEATURES_WEB_CLIENT
here instead of FEATURES_CLI
Makefile
Outdated
|
||
build-wasm: ## Builds the client library for wasm32 | ||
cargo build --target wasm32-unknown-unknown --features idxdb,web-tonic --no-default-features --package miden-client | ||
cargo build --workspace --exclude miden-client --exclude miden-cli --features $(FEATURES_WEB_CLIENT) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The flag --target wasm32-unknown-unknown
was removed is this ok?
6c59bb5
to
0923256
Compare
Thanks so much for the feedback!
The rest of your points are top of mind for me for sure! I will make issues out of them and start working on them ASAP. Specifically, improving the testing infrastructure and error handling I think will be the first to come. In parallel, maybe I can revisit the conversation surrounding the |
0923256
to
ac9f86f
Compare
Looks like the target is not correctly being installed? Perhaps it should be |
Looks like the check passed this time! I think we are good to merge after the other checks are green |
I think we are ready to merge this unless @bobbinth wants to take a final look. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! Thank you! This is a relatively superficial review from me - but I did leave a some comments/questions inline. The main ones are about the need of concurrent
and std
features for the web-client
crate.
3228bb4
to
6926529
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Thank you! I left just one nit inline. Other than that, we should be good to merge.
@dagarcia7 - seems like some tests are failing? This is probably due to some recent changes merged into |
Yeah, this is because we merged #445 (didn't think about it breaking this PR). I contacted Dennis earlier as well; we should have an update soon (I can fix the build in any case) |
1ebed6e
to
10d71d2
Compare
10d71d2
to
cdec1e0
Compare
Summary
This PR implements the
WebClient
, which is a WASM-compatible version of themiden-client
. With #402 and #409 merged into the codebase, the interfaces for a WASM-compatible store and a WASM-compatible RPC, have already been implemented and this PR puts all the components together to enable the creation of a full WASM module. With thepackage.json
androllup.config.js
file at the root of this new crate, this module can easily be published as an NPM package allowing seamless integration of Miden node interactions in web environments.The
WebClient
is built using aWebStore
and aWebTonicRpcClient
, mirroring the approach taken by the CLI tool that uses aSQLiteStore
and aTonicRpcClient
. This cohesive integration ensures that all the necessary functionalities for interacting with the Miden node are available in a web-compatible format, paving the way for broader usage and easier access in web-based applications.A Few Notes to the Reviewers
The biggest glaring issue with this PR right now is that the interface still needs some work. We started out making this interface mirror that of the CLI, which after talking to @bobbinth, we know isn't the correct strategy long term. After this code is merged, we hope to polish this interface and the methods it exposes to WASM to more correctly align with the general miden-client interface.
The second thing to discuss that can be handled in this PR is how we want to structure this new crate. For this first round of review, I added the
web-client
as a separate crate and attempted to clean up the workspaceCargo.toml
file a bit as well as any redundancies in therust-client
'sCargo.toml
file as well. However, I left theWebStore
andWebTonicRpcClient
implementations in therust-client
crate. This feels off and was planning on changing it before putting this up for review, but figured I'd get the OK first given that I think we did discuss the potential of that code remaining with the other sqlite and tonic client code. It shouldn't be a big effort take theWebStore
andWebTonicRpcClient
code and drop it all in theweb-client
crate so everything WASM-related lives together. This should also clean up some more of the Cargo files and make them a bit more distinct. Again, I think this is my preferred route, but maybe I'm missing an argument for the way it is currently structured in this first round of review.Finally, I'll add the same addendum here that we are hoping to push this code through so that development of the
miden-client
also includes the necessary changes to theWebStore
andWebTonicRpcClient
so we don't get too behind. After this is done, we will work on refining the interfaces as mentioned above, but also add better testing infrastructure, and general polish.