(WIP) LIU-5: Provide improved FileDROP naming and support for input-output filepaths. #314
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…port. An existing limitation of the DALiuGE engine is one of it's strengths; we create our own path names for intermediary data products because hard-coding them is prone to error, and so we have complete knowledge of the data. The problem here when we have applications or functions that have an input parameter that is the name of the file (i.e. a runtime-generated output DROP). To fix this, I have: - Added support for an input-output flag for a given parameter that is an ApplicationArgument - Provided a way to reference which output the input-output parameter we want to associated the flagged parameter - Provided support for prefix and suffix behaviour
Reviewer's Guide by SourceryThis pull request introduces several new features and improvements to the DALiuGE engine, including support for input-output filepaths in PyFuncApp, INI configuration files and SLURM template files for job submission, and an updated build process. Updated class diagram for PyFuncAppclassDiagram
class PyFuncApp {
-func_name
-input_parser
-output_parser
-func_arg_mapping
-func_defaults
-func
-argsig
-argnames
-fn_nargs
-fn_defaults
-fn_nposkw
-varargs
-varkw
-fn_ndef
-func_def_keywords
-num_args
-posonly
-poskw
-keyw
-defaults
-_recompute_data
-_inputs
-_outputs
-_applicationArgs
+initialize()
+run()
+write_results()
+generate_recompute_data()
-_mixin_func_defaults()
-_init_fn_defaults()
-_clean_applicationArgs()
-_initialise_args()
-_get_arg_info()
-_init_appArgs()
-_map_output_to_input()
-_arg_to_output()
-_ports2args()
+_match_parser()
+initialize_with_func_code()
}
class Argument {
value: object
encoding: DropParser
type: Union [ArgType, None]
input_output: bool
precious: bool
positional: bool
}
PyFuncApp -- Argument : uses
note for Argument "Encapsulate data associated with all arguments."
Class diagram for ConfigManagerclassDiagram
class ConfigManager {
-facilities
-prompt_setup
+setup_user()
+prompt_user()
+create_dlg_config_directory()
+copy_defaults_to_user_config_directory()
+get_user_configs()
+print_available_config()
+load_user_config()
}
class ConfigType {
ENV
SLURM
}
ConfigManager -- ConfigType : uses
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
myxie
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA Ticket
LIU-5
Type
Problem/Issue
DRAFT
An existing limitation of the DALiuGE engine is one of it's strengths; we create our own path names for intermediary data products because hard-coding them is prone to error, and so we have complete knowledge of the data.
The problem here when we have applications or functions that have an input parameter that is the name of the file (i.e. a runtime-generated output DROP).
Solution
To fix this, I have:
Checklist