Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch: Use github.com/expr-lang/expr #41

Merged
merged 2 commits into from
Jan 9, 2025
Merged

Conversation

breml
Copy link
Contributor

@breml breml commented Jan 7, 2025

replace include/exclude regex with include expression

@gibmat, @stgraber I looked into the task of improving the way we define a batch. In the respective task (Figure out better way of defining a batch) we listed 3 options:

After quickly checking on them, I found a 4th option, that was mentioned in an issue on govaluate and I found, that this actually the best option in my opinion.

This PR implements a PoC with the above mentioned expression package. Since this package provides complete boolean expressions, we are able to replace both, the include and the exclude regex we used so far. The expression package also allows to match against regex, so there is no reduction in functionality at all.

By passing the instance to the expression evaluation as argument, one does have access to all the methods (and fields, if the argument is a struct), which allows as for more fine grained logic if an instance should be included or excluded from a batch.
The functionality of the expr package can be extended by custom function. I also added an example for this by adding the filepath.Dir and filepath.Base functions additionally.

I think, the provided test cases do explain the usage the best.

Additional resources:

Side note: While integrating the package into our code base, I discovered an issue with the way interfaces are handled. There is already an open issue for this, so I added some context there as well: expr-lang/expr#744

@breml breml requested a review from gibmat January 7, 2025 09:52
@breml breml self-assigned this Jan 7, 2025
@stgraber
Copy link
Contributor

stgraber commented Jan 7, 2025

Looks pretty good from the reference link.

I actually expect regexp queries to be pretty rare overall, I suspect we're likely to want something closer to this as an example batch filter:

source in ["vcenter01", "vcenter02", "vcenter03"] && ("/DEV/foo" in path || "/DEV/bar" in path) && instance.resources.cpus <= 4 && instance.resources.memory <= 1024*1024*1024*8 && instance.resources.disks == 1 && !instance.resources.shared_disks && instance.os in ["Ubuntu 22.04", "Ubuntu 24.04"]

Which it seems like this project can handle just fine, including support for some extra string management functions.

@stgraber
Copy link
Contributor

stgraber commented Jan 7, 2025

How good is their error reporting? We're going to be exposing this to end users, so it'd be nice to have clear error messages so they can easily tell what's wrong with their filter and can fix it.

internal/batch/batch.go Outdated Show resolved Hide resolved
shared/api/batch.go Outdated Show resolved Hide resolved
@breml
Copy link
Contributor Author

breml commented Jan 9, 2025

How good is their error reporting? We're going to be exposing this to end users, so it'd be nice to have clear error messages so they can easily tell what's wrong with their filter and can fix it.

The error returned from the expression package look like this:

Example 1:

unknown name GetInventoryPath1 (1:1)
| GetInventoryPath1() == "a/b/c"
| ^

Example 2:

literal not terminated (1:29)
 | GetInventoryPath() == "a/b/c
 | ............................^

It will be our responsibility to prefix these errors with additional meaningful details about the step, that failed (e.g. compiling the expression or running it).

breml added 2 commits January 9, 2025 09:24
replace include/exclude regex with include expression

Signed-off-by: Lucas Bremgartner <lucas.bremgartner@futurfusion.io>
Signed-off-by: Lucas Bremgartner <lucas.bremgartner@futurfusion.io>
@breml breml force-pushed the batch-definition-poc-expr-lang branch from cffdc7d to 3a43e2e Compare January 9, 2025 10:46
@breml
Copy link
Contributor Author

breml commented Jan 9, 2025

@stgraber, @gibmat I addressed the review comments and I extended the poc such that the more complex / realistic example provided by @stgraber in #41 (comment) becomes possible to filter for (see testcase https://github.com/FuturFusion/migration-manager/pull/41/files#diff-e3d7bfdcdd97202d2d2778d99715109ec9cb180125c0c318a51059193f027856R118-R124).

@breml breml requested a review from gibmat January 9, 2025 14:36
@stgraber stgraber changed the title Batch: github.com/expr-lang/expr PoC batch: Use github.com/expr-lang/expr Jan 9, 2025
@gibmat gibmat merged commit fde8681 into main Jan 9, 2025
4 checks passed
@gibmat gibmat deleted the batch-definition-poc-expr-lang branch January 9, 2025 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants