diff --git a/docs/config.md b/docs/config.md index 6d90a8b121..954bb43a28 100644 --- a/docs/config.md +++ b/docs/config.md @@ -789,7 +789,13 @@ The following settings are available: : Set the minimum CPU Platform, e.g. `'Intel Skylake'`. See [Specifying a minimum CPU Platform for VM instances](https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform#specifications) (default: none). `google.batch.network` -: Set network name to attach the VM's network interface to. The value will be prefixed with `global/networks/` unless it contains a `/`, in which case it is assumed to be a fully specified network resource URL. If unspecified, the global default network is used. +: The URL of an existing network resource to which the VM will be attached. + + You can specify the network as a full or partial URL. For example, the following are all valid URLs: + + - https://www.googleapis.com/compute/v1/projects/{project}/global/networks/{network} + - projects/{project}/global/networks/{network} + - global/networks/{network} `google.batch.serviceAccountEmail` : Define the Google service account email to use for the pipeline execution. If not specified, the default Compute Engine service account for the project will be used. @@ -798,7 +804,13 @@ The following settings are available: : When `true` enables the usage of *spot* virtual machines or `false` otherwise (default: `false`). `google.batch.subnetwork` -: Define the name of the subnetwork to attach the instance to must be specified here, when the specified network is configured for custom subnet creation. The value is prefixed with `regions/subnetworks/` unless it contains a `/`, in which case it is assumed to be a fully specified subnetwork resource URL. +: The URL of an existing subnetwork resource in the network to which the VM will be attached. + + You can specify the subnetwork as a full or partial URL. For example, the following are all valid URLs: + + - https://www.googleapis.com/compute/v1/projects/{project}/regions/{region}/subnetworks/{subnetwork} + - projects/{project}/regions/{region}/subnetworks/{subnetwork} + - regions/{region}/subnetworks/{subnetwork} `google.batch.usePrivateAddress` : When `true` the VM will NOT be provided with a public IP address, and only contain an internal IP. If this option is enabled, the associated job can only load docker images from Google Container Registry, and the job executable cannot use external services other than Google APIs (default: `false`). @@ -1335,6 +1347,12 @@ The following settings are available: `singularity.noHttps` : Pull the Singularity image with http protocol (default: `false`). +`singularity.oci` +: :::{versionadded} 23.11.0-edge + ::: +: Enable OCI-mode the allows the use of native OCI-compatible containers with Singularity. See [Singularity documentation](https://docs.sylabs.io/guides/4.0/user-guide/oci_runtime.html#oci-mode) for more details and requirements (default: `false`). + + `singularity.pullTimeout` : The amount of time the Singularity pull can last, exceeding which the process is terminated (default: `20 min`). @@ -1630,10 +1648,13 @@ The following environment variables control the configuration of the Nextflow ru : Allows the setting Java VM options. This is similar to `NXF_OPTS` however it's only applied the JVM running Nextflow and not to any java pre-launching commands. `NXF_OFFLINE` -: When `true` disables the project automatic download and update from remote repositories (default: `false`). +: When `true` prevents Nextflow from automatically downloading and updating remote project repositories (default: `false`). : :::{versionchanged} 23.09.0-edge This option also disables the automatic version check (see `NXF_DISABLE_CHECK_LATEST`). ::: +: :::{versionchanged} 23.11.0-edge + This option also prevents plugins from being downloaded. Plugin versions must be specified in offline mode, or else Nextflow will fail. + ::: `NXF_OPTS` : Provides extra options for the Java and Nextflow runtime. It must be a blank separated list of `-Dkey[=value]` properties. diff --git a/docs/executor.md b/docs/executor.md index a14d867146..0f50eff6ae 100644 --- a/docs/executor.md +++ b/docs/executor.md @@ -328,6 +328,10 @@ The `local` executor is used by default. It runs the pipeline processes on the c The `local` executor is useful for developing and testing a pipeline script on your computer, before switching to a cluster or cloud environment with production data. +:::{note} +While the `local` executor limits the number of concurrent tasks based on requested vs available resources, it does not enforce task resource requests. In other words, it is possible for a local task to use more CPUs and memory than it requested, in which case it may starve other tasks. An exception to this behavior is when using {ref}`container-docker` or {ref}`container-podman` containers, in which case the resource requests are enforced by the container runtime. +::: + (lsf-executor)= ## LSF diff --git a/docs/google.md b/docs/google.md index 36a8846598..1dcddfc2d8 100644 --- a/docs/google.md +++ b/docs/google.md @@ -175,11 +175,14 @@ google { } ``` +:::{versionadded} 23.11.0-edge +::: + Since this type of virtual machines can be retired by the provider before the job completion, it is advisable to add the following retry strategy to your config file to instruct Nextflow to automatically re-execute a job if the virtual machine was terminated preemptively: ```groovy process { - errorStrategy = { task.exitStatus==14 ? 'retry' : 'terminate' } + errorStrategy = { task.exitStatus==50001 ? 'retry' : 'terminate' } maxRetries = 5 } ``` diff --git a/docs/module.md b/docs/module.md index 11298699dc..59fc886887 100644 --- a/docs/module.md +++ b/docs/module.md @@ -270,3 +270,11 @@ Those scripts will be made accessible like any other command in the task environ :::{note} This feature requires the use of a local or shared file system for the pipeline work directory, or {ref}`wave-page` when using cloud-based executors. ::: + +## Sharing modules + +Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it: + +- Simply copy the module files into your pipeline repository +- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy +- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community diff --git a/docs/plugins.md b/docs/plugins.md index 061207a9b9..1ceeb18bb1 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -249,6 +249,33 @@ channel The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +### Process directives + +Plugins that implement a [custom executor](#executors) will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task's configuration. As a best practice, custom executors should try to support all process directives that have executor-specific behavior and are relevant to your executor. + +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, you can use the {ref}`process-ext` directive to provide custom process settings to your executor. Try to use specific names that are not likely to conflict with other plugins or existing pipelines. + +Here is an example of a custom executor that uses existing process directives as well as a custom setting through the `ext` directive: + +```groovy +class MyExecutor extends Executor { + + @Override + TaskHandler createTaskHandler(TaskRun task) { + final cpus = task.config.cpus + final memory = task.config.memory + final myOption = task.config.ext.myOption + + println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" + + // ... + } + + // ... + +} +``` + ### Trace observers A *trace observer* in Nextflow is an entity that can listen and react to workflow events, such as when a workflow starts, a task completes, a file is published, etc. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. diff --git a/docs/process.md b/docs/process.md index 7f99670ed5..73d6dec191 100644 --- a/docs/process.md +++ b/docs/process.md @@ -1177,6 +1177,52 @@ output: In this example, the process is normally expected to produce an `output.txt` file, but in the cases where the file is legitimately missing, the process does not fail. The output channel will only contain values for those processes that produce `output.txt`. +(process-multiple-outputs)= + +### Multiple outputs + +When a process declares multiple outputs, each output can be accessed by index. The following example prints the second process output (indexes start at zero): + +```groovy +process FOO { + output: + path 'bye_file.txt' + path 'hi_file.txt' + + """ + echo "bye" > bye_file.txt + echo "hi" > hi_file.txt + """ +} + +workflow { + FOO() + FOO.out[1].view() +} +``` + +You can also use the `emit` option to assign a name to each output and access them by name: + +```groovy +process FOO { + output: + path 'bye_file.txt', emit: bye_file + path 'hi_file.txt', emit: hi_file + + """ + echo "bye" > bye_file.txt + echo "hi" > hi_file.txt + """ +} + +workflow { + FOO() + FOO.out.hi_file.view() +} +``` + +See {ref}`workflow-process-invocation` for more details. + ## When The `when` block allows you to define a condition that must be satisfied in order to execute the process. The condition can be any expression that returns a boolean value. @@ -1653,15 +1699,26 @@ process mapping { tuple val(sampleId), path(reads) """ - STAR --genomeDir $genome --readFilesIn $reads + STAR --genomeDir $genome --readFilesIn $reads ${task.ext.args ?: ''} """ } ``` -In the above example, the process uses a container whose version is controlled by the `ext.version` property. This can be defined in the `nextflow.config` file as shown below: +In the above example, the process container version is controlled by `ext.version`, and the script supports additional command line arguments through `ext.args`. + +The `ext` directive can be set in the process definition: + +```groovy +process mapping { + ext version: '2.5.3', args: '--foo --bar' +} +``` + +Or in the Nextflow configuration: ```groovy process.ext.version = '2.5.3' +process.ext.args = '--foo --bar' ``` (process-fair)= @@ -2075,6 +2132,9 @@ The following options are available: `runAsUser: ''` : Specifies the user ID with which to run the container. Shortcut for the `securityContext` option. +`schedulerName: ''` +: Specifies which [scheduler](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/#specify-schedulers-for-pods) is used to schedule the container. + `secret: '/', mountPath: ''` : *Can be specified multiple times* : Mounts a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) with name and optional key to the given path. If the key is omitted, the path is interpreted as a directory and all entries in the `Secret` are exposed in that path. diff --git a/docs/workflow.md b/docs/workflow.md index 209b3e769b..a81dcc1a83 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -41,6 +41,8 @@ The `main:` label can be omitted if there are no `take:` or `emit:` blocks. Workflows were introduced in DSL2. If you are still using DSL1, see the {ref}`dsl1-page` page to learn how to migrate your Nextflow pipelines to DSL2. ::: +(workflow-process-invocation)= + ## Process invocation A process can be invoked like a function in a workflow definition, passing the expected input channels like function arguments. For example: @@ -142,6 +144,8 @@ workflow { } ``` +See {ref}`process-multiple-outputs` for more details. + ### Process named stdout The `emit` option can also be used to name a `stdout` output: diff --git a/modules/nextflow/src/main/groovy/nextflow/NF.groovy b/modules/nextflow/src/main/groovy/nextflow/NF.groovy index 3e02805646..3f13630088 100644 --- a/modules/nextflow/src/main/groovy/nextflow/NF.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/NF.groovy @@ -50,11 +50,6 @@ class NF { NextflowMeta.instance.isDsl2() } - @Deprecated - static boolean isDsl2Final() { - NextflowMeta.instance.isDsl2Final() - } - static Binding getBinding() { isDsl2() ? ExecutionStack.binding() : session().getBinding() } diff --git a/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy b/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy index dd1b03f8e8..5976f229e3 100644 --- a/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy @@ -407,6 +407,4 @@ class Nextflow { */ static Closure multiMapCriteria(Closure closure) { closure } - @Deprecated - static Closure forkCriteria(Closure closure) { closure } } diff --git a/modules/nextflow/src/main/groovy/nextflow/NextflowMeta.groovy b/modules/nextflow/src/main/groovy/nextflow/NextflowMeta.groovy index b642697709..8074dd968a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/NextflowMeta.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/NextflowMeta.groovy @@ -112,12 +112,9 @@ class NextflowMeta { result.version = version.toString() result.build = build result.timestamp = parseDateStr(timestamp) - if( isDsl2Final() ) { + if( isDsl2() ) { result.enable = featuresMap() } - else if( isDsl2() ) { - result.preview = featuresMap() - } return result } @@ -135,17 +132,6 @@ class NextflowMeta { enable.dsl == 2f } - /** - * As of the removal of DSL2 preview mode, the semantic of this method - * is identical to {@link #isDsl2()}. - * @return - * {@code true} when the workflow script uses DSL2 syntax, {@code false} otherwise. - */ - @Deprecated - boolean isDsl2Final() { - enable.dsl == 2f - } - void enableDsl2() { this.enable.dsl = 2f } diff --git a/modules/nextflow/src/main/groovy/nextflow/ast/NextflowDSLImpl.groovy b/modules/nextflow/src/main/groovy/nextflow/ast/NextflowDSLImpl.groovy index 73dbbc0aa2..0586019d7a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/ast/NextflowDSLImpl.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/ast/NextflowDSLImpl.groovy @@ -38,7 +38,6 @@ import nextflow.script.TokenVar import org.codehaus.groovy.ast.ASTNode import org.codehaus.groovy.ast.ClassCodeVisitorSupport import org.codehaus.groovy.ast.ClassNode -import org.codehaus.groovy.ast.ConstructorNode import org.codehaus.groovy.ast.MethodNode import org.codehaus.groovy.ast.Parameter import org.codehaus.groovy.ast.VariableScope @@ -79,8 +78,6 @@ import org.codehaus.groovy.transform.GroovyASTTransformation @GroovyASTTransformation(phase = CompilePhase.CONVERSION) class NextflowDSLImpl implements ASTTransformation { - @Deprecated final static private String WORKFLOW_GET = 'get' - @Deprecated final static private String WORKFLOW_PUBLISH = 'publish' final static private String WORKFLOW_TAKE = 'take' final static private String WORKFLOW_EMIT = 'emit' final static private String WORKFLOW_MAIN = 'main' @@ -151,63 +148,6 @@ class NextflowDSLImpl implements ASTTransformation { super.visitMethod(node) } - protected Statement makeSetProcessNamesStm() { - final names = new ListExpression() - for( String it: processNames ) { - names.addExpression(new ConstantExpression(it.toString())) - } - - // the method list argument - final args = new ArgumentListExpression() - args.addExpression(names) - - // some magic code - // this generates the invocation of the method: - // nextflow.script.ScriptMeta.get(this).setProcessNames() - final scriptMeta = new PropertyExpression( new PropertyExpression(new VariableExpression('nextflow'),'script'), 'ScriptMeta') - final thiz = new ArgumentListExpression(); thiz.addExpression( new VariableExpression('this') ) - final meta = new MethodCallExpression( scriptMeta, 'get', thiz ) - final call = new MethodCallExpression( meta, 'setDsl1ProcessNames', args) - final stm = new ExpressionStatement(call) - return stm - } - - /** - * Add to constructor a method call to inject parsed metadata. - * Only needed by DSL1. - * - * @param node The node representing the class to be invoked - */ - protected void injectMetadata(ClassNode node) { - for( ConstructorNode constructor : node.getDeclaredConstructors() ) { - def code = constructor.getCode() - if( code instanceof BlockStatement ) { - code.addStatement(makeSetProcessNamesStm()) - } - else if( code instanceof ExpressionStatement ) { - def expr = code - def block = new BlockStatement() - block.addStatement(expr) - block.addStatement(makeSetProcessNamesStm()) - constructor.setCode(block) - } - else - throw new IllegalStateException("Invalid constructor expression: $code") - } - } - - /** - * Only needed by DSL1 to inject process names declared in the script - * - * @param node The node representing the class to be invoked - */ - @Override - protected void visitObjectInitializerStatements(ClassNode node) { - if( node.getSuperClass().getName() == BaseScript.getName() ) - injectMetadata(node) - super.visitObjectInitializerStatements(node) - } - @Override void visitMethodCallExpression(MethodCallExpression methodCall) { // pre-condition to be verified to apply the transformation @@ -508,12 +448,6 @@ class NextflowDSLImpl implements ASTTransformation { visited[context] = true switch (context) { - case WORKFLOW_GET: - syntaxError(stm, "Workflow 'get' is not supported anymore use 'take' instead") - - case WORKFLOW_PUBLISH: - syntaxError(stm, "Workflow 'publish' is not supported anymore use process 'publishDir' instead") - case WORKFLOW_TAKE: case WORKFLOW_EMIT: if( !(stm instanceof ExpressionStatement) ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/ast/OpXformImpl.groovy b/modules/nextflow/src/main/groovy/nextflow/ast/OpXformImpl.groovy index 039ccfda86..c60a2a341e 100644 --- a/modules/nextflow/src/main/groovy/nextflow/ast/OpXformImpl.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/ast/OpXformImpl.groovy @@ -79,10 +79,6 @@ class OpXformImpl implements ASTTransformation { static final public String MULTIMAP_CRITERIA_FUN = 'multiMapCriteria' - @Deprecated static final public String FORK_METHOD_NAME = 'fork' - - @Deprecated static final public String FORK_CRITERIA_FUN = 'forkCriteria' - SourceUnit unit static class BranchCondition { @@ -385,14 +381,6 @@ class OpXformImpl implements ASTTransformation { return ret if( name==MULTIMAP_CRITERIA_FUN && args.size()==1 && m.objectExpression.text=='this') return ret - if( name==FORK_METHOD_NAME && args.size()==1 ) { - log.debug "Operator `fork` has been renamed to `multiMap`" - return ret - } - if( name==FORK_CRITERIA_FUN && args.size()==1 && m.objectExpression.text=='this') { - log.warn "Function `forkCriteria` has been renamed to `multiMapCriteria`" - return ret - } } return null diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdSecret.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdSecret.groovy index 3ef3163d86..d7e86278d3 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdSecret.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdSecret.groovy @@ -56,7 +56,6 @@ class CmdSecret extends CmdBase implements UsageAware { CmdSecret() { commands.add( new GetCmd() ) - commands.add( new PutCmd() ) commands.add( new SetCmd() ) commands.add( new ListCmd() ) commands.add( new DeleteCmd() ) @@ -146,19 +145,6 @@ class CmdSecret extends CmdBase implements UsageAware { } } - /** - * Implements the secret `put` sub-command - */ - @Deprecated - class PutCmd extends SetCmd { - String getName() { 'put' } - - void apply(List result) { - log.warn "Put command is deprecated - use 'set' instead'" - super.apply(result) - } - } - class SetCmd implements SubCmd { @Override diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy index be9458eaa2..8ed41196c1 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy @@ -101,7 +101,6 @@ class Launcher { new CmdView(), new CmdHelp(), new CmdSelfUpdate(), - new CmdPlugins(), new CmdPlugin(), new CmdInspect() ] diff --git a/modules/nextflow/src/main/groovy/nextflow/container/CharliecloudBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/CharliecloudBuilder.groovy index e219cddcf4..b72aa61569 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/CharliecloudBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/CharliecloudBuilder.groovy @@ -61,7 +61,7 @@ class CharliecloudBuilder extends ContainerBuilder { CharliecloudBuilder build(StringBuilder result) { assert image - result << 'ch-run --unset-env="*" -c "$PWD" --set-env ' + result << 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env ' if (!readOnlyInputs) result << '-w ' diff --git a/modules/nextflow/src/main/groovy/nextflow/container/ContainerBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/ContainerBuilder.groovy index 9f505f0a29..e552a8aaaa 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/ContainerBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/ContainerBuilder.groovy @@ -295,7 +295,7 @@ abstract class ContainerBuilder { // -- append by default the current path -- this is needed when `scratch` is set to true if( mountWorkDir ) { - result << composeVolumePath('$PWD') + result << composeVolumePath('$NXF_TASK_WORKDIR') result << ' ' } diff --git a/modules/nextflow/src/main/groovy/nextflow/container/ContainerConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/container/ContainerConfig.groovy index 8b93ac5cfe..78e8caef71 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/ContainerConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/ContainerConfig.groovy @@ -55,6 +55,10 @@ class ContainerConfig extends LinkedHashMap { get('engine') } + boolean singularityOciMode() { + getEngine()=='singularity' && get('oci')?.toString() == 'true' + } + List getEnvWhitelist() { def result = get('envWhitelist') if( !result ) diff --git a/modules/nextflow/src/main/groovy/nextflow/container/ContainerHandler.groovy b/modules/nextflow/src/main/groovy/nextflow/container/ContainerHandler.groovy index aafa544913..524ce6d9e3 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/ContainerHandler.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/ContainerHandler.groovy @@ -67,6 +67,8 @@ class ContainerHandler { final normalizedImageName = normalizeSingularityImageName(imageName) if( !config.isEnabled() || !normalizedImageName ) return normalizedImageName + if( normalizedImageName.startsWith('docker://') && config.singularityOciMode() ) + return normalizedImageName final requiresCaching = normalizedImageName =~ IMAGE_URL_PREFIX if( ContainerInspectMode.active() && requiresCaching ) return imageName @@ -192,7 +194,7 @@ class ContainerHandler { } - public static final Pattern IMAGE_URL_PREFIX = ~/^[^\/:\. ]+:\/\/(.*)/ + public static final Pattern IMAGE_URL_PREFIX = ~/^[^\/:. ]+:\/\/(.*)/ /** * Normalize Singularity image name resolving the absolute path or diff --git a/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy index 8e5e037822..edc5fc9cb0 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy @@ -155,7 +155,7 @@ class DockerBuilder extends ContainerBuilder { // mount the input folders result << makeVolumes(mounts) - result << '-w "$PWD" ' + result << '-w "$NXF_TASK_WORKDIR" ' if( entryPoint ) result << '--entrypoint ' << entryPoint << ' ' diff --git a/modules/nextflow/src/main/groovy/nextflow/container/PodmanBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/PodmanBuilder.groovy index 649a85c720..84ff98a72f 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/PodmanBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/PodmanBuilder.groovy @@ -112,7 +112,7 @@ class PodmanBuilder extends ContainerBuilder { // mount the input folders result << makeVolumes(mounts) - result << '-w "$PWD" ' + result << '-w "$NXF_TASK_WORKDIR" ' if( entryPoint ) result << '--entrypoint ' << entryPoint << ' ' diff --git a/modules/nextflow/src/main/groovy/nextflow/container/SarusBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/SarusBuilder.groovy index 72306660a8..f73cb2ae27 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/SarusBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/SarusBuilder.groovy @@ -52,7 +52,7 @@ class SarusBuilder extends ContainerBuilder { // mount the input folders result << makeVolumes(mounts) - result << '-w "$PWD" ' + result << '-w "$NXF_TASK_WORKDIR" ' if( runOptions ) result << runOptions.join(' ') << ' ' diff --git a/modules/nextflow/src/main/groovy/nextflow/container/SingularityBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/SingularityBuilder.groovy index 5c2d74abfc..a4829f0f54 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/SingularityBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/SingularityBuilder.groovy @@ -39,6 +39,8 @@ class SingularityBuilder extends ContainerBuilder { private String runCmd0 + private Boolean oci + SingularityBuilder(String name) { this.image = name this.homeMount = defaultHomeMount() @@ -92,6 +94,9 @@ class SingularityBuilder extends ContainerBuilder { if( params.containsKey('readOnlyInputs') ) this.readOnlyInputs = params.readOnlyInputs?.toString() == 'true' + if( params.oci!=null ) + oci = params.oci.toString() == 'true' + return this } @@ -117,9 +122,12 @@ class SingularityBuilder extends ContainerBuilder { if( !homeMount ) result << '--no-home ' - if( newPidNamespace ) + if( newPidNamespace && !oci ) result << '--pid ' + if( oci != null ) + result << (oci ? '--oci ' : '--no-oci ') + if( autoMounts ) { makeVolumes(mounts, result) } @@ -145,6 +153,11 @@ class SingularityBuilder extends ContainerBuilder { protected CharSequence appendEnv(StringBuilder result) { makeEnv('TMP',result) .append(' ') makeEnv('TMPDIR',result) .append(' ') + // add magic variables required by singularity to run in OCI-mode + if( oci ) { + result .append('${XDG_RUNTIME_DIR:+XDG_RUNTIME_DIR="$XDG_RUNTIME_DIR"} ') + result .append('${DBUS_SESSION_BUS_ADDRESS:+DBUS_SESSION_BUS_ADDRESS="$DBUS_SESSION_BUS_ADDRESS"} ') + } super.appendEnv(result) } @@ -212,7 +225,7 @@ class SingularityBuilder extends ContainerBuilder { if( launcher ) { def result = getRunCommand() - result += entryPoint ? " $entryPoint -c \"cd \$PWD; $launcher\"" : " $launcher" + result += entryPoint ? " $entryPoint -c \"cd \$NXF_TASK_WORKDIR; $launcher\"" : " $launcher" return result } return getRunCommand() + ' ' + launcher diff --git a/modules/nextflow/src/main/groovy/nextflow/container/UdockerBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/container/UdockerBuilder.groovy index afe35cf97d..31bd44332a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/container/UdockerBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/container/UdockerBuilder.groovy @@ -79,7 +79,7 @@ class UdockerBuilder extends ContainerBuilder { // mount the input folders result << makeVolumes(mounts) - result << '-w "$PWD" --bindhome ' + result << '-w "$NXF_TASK_WORKDIR" --bindhome ' if( runOptions ) diff --git a/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy index bc21186d1a..613c3dd706 100644 --- a/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy @@ -480,9 +480,13 @@ class BashWrapperBuilder { */ if( containerBuilder ) { String cmd = env ? 'eval $(nxf_container_env); ' + launcher : launcher - if( env && !containerConfig.entrypointOverride() ) { - if( containerBuilder instanceof SingularityBuilder ) - cmd = 'cd $PWD; ' + cmd + // wrap the command with an extra bash invocation either : + // - to propagate the container environment or + // - to change in the task work directory as required by singularity + final needChangeTaskWorkDir = containerBuilder instanceof SingularityBuilder + if( (env || needChangeTaskWorkDir) && !containerConfig.entrypointOverride() ) { + if( needChangeTaskWorkDir ) + cmd = 'cd $NXF_TASK_WORKDIR; ' + cmd cmd = "/bin/bash -c \"$cmd\"" } launcher = containerBuilder.getRunCommand(cmd) diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/CH.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/CH.groovy index 390064d34c..a4d9762cc6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/extension/CH.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/extension/CH.groovy @@ -66,8 +66,6 @@ class CH { } static private DataflowReadChannel getRead2(DataflowBroadcast channel) { - if( !NF.isDsl2() ) - throw new IllegalStateException("Broadcast channel are only allowed in a workflow definition scope") channel.createReadChannel() } diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/ChannelEx.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/ChannelEx.groovy index 499124ce44..2ebc20ae87 100644 --- a/modules/nextflow/src/main/groovy/nextflow/extension/ChannelEx.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/extension/ChannelEx.groovy @@ -83,18 +83,6 @@ class ChannelEx { return target } - /** - * Close a dataflow queue channel binding a {@link Channel#STOP} item - * - * @param source The source dataflow channel to be closed. - */ - @Deprecated - static DataflowWriteChannel close(DataflowWriteChannel source) { - if( NF.isDsl2() ) - throw new DeprecationException("Channel `close` method is not supported anymore") - return CH.close0(source) - } - /** * INTERNAL ONLY API *

@@ -113,9 +101,6 @@ class ChannelEx { } static private void checkContext(String method, Object operand) { - if( !NF.isDsl2() ) - throw new MissingMethodException(method, operand.getClass()) - if( operand instanceof ComponentDef && !ExecutionStack.withinWorkflow() ) throw new IllegalArgumentException("Process invocation are only allowed within a workflow context") } diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/ChannelExtensionPoint.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/ChannelExtensionPoint.groovy deleted file mode 100644 index 9ef47aea60..0000000000 --- a/modules/nextflow/src/main/groovy/nextflow/extension/ChannelExtensionPoint.groovy +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2013-2023, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package nextflow.extension - -import nextflow.plugin.extension.PluginExtensionPoint - -/** - * This class is deprecated, use {@link PluginExtensionPoint} instead - * - * @author Paolo Di Tommaso - */ -@Deprecated -abstract class ChannelExtensionPoint extends PluginExtensionPoint { -} diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/DeprecatedDsl2.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/DeprecatedDsl2.groovy deleted file mode 100644 index 19eebad424..0000000000 --- a/modules/nextflow/src/main/groovy/nextflow/extension/DeprecatedDsl2.groovy +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright 2013-2023, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package nextflow.extension - -import static java.lang.annotation.ElementType.CONSTRUCTOR -import static java.lang.annotation.ElementType.FIELD -import static java.lang.annotation.ElementType.LOCAL_VARIABLE -import static java.lang.annotation.ElementType.METHOD -import static java.lang.annotation.ElementType.PACKAGE -import static java.lang.annotation.ElementType.PARAMETER -import static java.lang.annotation.ElementType.TYPE - -import java.lang.annotation.Retention -import java.lang.annotation.RetentionPolicy -import java.lang.annotation.Target - -/** - * Deprecation introduced by Dsl-2 syntax - * - * @author Paolo Di Tommaso - */ -@Retention(RetentionPolicy.RUNTIME) -@Target(value=[CONSTRUCTOR, FIELD, LOCAL_VARIABLE, METHOD, PACKAGE, PARAMETER, TYPE]) -@interface DeprecatedDsl2 { - String message() default '' -} diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/OpCall.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/OpCall.groovy index 11fd5c6dd5..a18e02dd72 100644 --- a/modules/nextflow/src/main/groovy/nextflow/extension/OpCall.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/extension/OpCall.groovy @@ -320,13 +320,6 @@ class OpCall implements Callable { protected void checkDeprecation(Method method) { if( method.getAnnotation(Deprecated) ) { def messg = "Operator `$methodName` is deprecated -- it will be removed in a future release" - if( NF.isDsl2() ) - throw new DeprecationException(messg) - log.warn messg - } - else if( method.getAnnotation(DeprecatedDsl2) && NF.isDsl2() ) { - def annot = method.getAnnotation(DeprecatedDsl2) - def messg = annot.message() ?: "Operator `$methodName` has been deprecated -- it's not available in DSL2 syntax".toString() throw new DeprecationException(messg) } } diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/OperatorImpl.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/OperatorImpl.groovy index 16a9b87a45..ceeb26f858 100644 --- a/modules/nextflow/src/main/groovy/nextflow/extension/OperatorImpl.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/extension/OperatorImpl.groovy @@ -1016,12 +1016,6 @@ class OperatorImpl { return tap.result } - @DeprecatedDsl2 - DataflowWriteChannel tap( final DataflowReadChannel source, final DataflowWriteChannel target ) { - def tap = new TapOp(source, target).apply() - return tap.result - } - /** * Empty the specified value only if the source channel to which is applied is empty i.e. do not emit diff --git a/modules/nextflow/src/main/groovy/nextflow/fusion/FusionHelper.groovy b/modules/nextflow/src/main/groovy/nextflow/fusion/FusionHelper.groovy index 6298f9583a..4d852ee983 100644 --- a/modules/nextflow/src/main/groovy/nextflow/fusion/FusionHelper.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/fusion/FusionHelper.groovy @@ -72,7 +72,7 @@ class FusionHelper { final containerCmd = containerBuilder .build() .getRunCommand(patchCmd.join(' ')) - .replaceAll('-w "\\$PWD" ','') // <-- hack to remove the PWD work dir + .replaceAll('-w "\\$NXF_TASK_WORKDIR" ','') // <-- hack to remove the PWD work dir return containerCmd } diff --git a/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodOptions.groovy b/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodOptions.groovy index 19ddc6d8d2..4c1d0000e8 100644 --- a/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodOptions.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodOptions.groovy @@ -67,6 +67,8 @@ class PodOptions { private List tolerations private Boolean privileged + + private String schedulerName PodOptions( List options=null ) { int size = options ? options.size() : 0 @@ -156,6 +158,9 @@ class PodOptions { else if( entry.privileged instanceof Boolean ) { this.privileged = entry.privileged as Boolean } + else if( entry.schedulerName ) { + this.schedulerName = entry.schedulerName + } else throw new IllegalArgumentException("Unknown pod options: $entry") } @@ -218,6 +223,8 @@ class PodOptions { String getPriorityClassName() { priorityClassName } + String getSchedulerName() { schedulerName } + List getTolerations() { tolerations } Boolean getPrivileged() { privileged } @@ -297,6 +304,9 @@ class PodOptions { // privileged execution result.privileged = other.privileged!=null ? other.privileged : this.privileged + // scheduler name + result.schedulerName = other.schedulerName ?: this.schedulerName + return result } } diff --git a/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodSpecBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodSpecBuilder.groovy index e87c0abeb2..24c56bc308 100644 --- a/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodSpecBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/k8s/model/PodSpecBuilder.groovy @@ -118,6 +118,8 @@ class PodSpecBuilder { Map resourcesLimits + String schedulerName + /** * @return A sequential volume unique identifier */ @@ -374,6 +376,8 @@ class PodSpecBuilder { tolerations.addAll(opts.tolerations) // -- privileged privileged = opts.privileged + // -- scheduler name + schedulerName = opts.schedulerName return this } @@ -436,6 +440,9 @@ class PodSpecBuilder { if( nodeSelector ) spec.nodeSelector = nodeSelector.toSpec() + if( schedulerName ) + spec.schedulerName = schedulerName + if( affinity ) spec.affinity = affinity diff --git a/modules/nextflow/src/main/groovy/nextflow/plugin/extension/PluginExtensionProvider.groovy b/modules/nextflow/src/main/groovy/nextflow/plugin/extension/PluginExtensionProvider.groovy index 2f3f050fb1..f24de108a5 100644 --- a/modules/nextflow/src/main/groovy/nextflow/plugin/extension/PluginExtensionProvider.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/plugin/extension/PluginExtensionProvider.groovy @@ -290,13 +290,4 @@ class PluginExtensionProvider implements ExtensionProvider { } } - @Deprecated - static void reloadExtensionPoints() { - if( !instance ) - return - instance.channelExtensionPoints=null - instance.operatorExtensions.clear() - instance.install() - } - } diff --git a/modules/nextflow/src/main/groovy/nextflow/processor/TaskBean.groovy b/modules/nextflow/src/main/groovy/nextflow/processor/TaskBean.groovy index c20a8ac2eb..441c19795e 100644 --- a/modules/nextflow/src/main/groovy/nextflow/processor/TaskBean.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/processor/TaskBean.groovy @@ -77,9 +77,6 @@ class TaskBean implements Serializable, Cloneable { String afterScript - @Deprecated - boolean containerExecutable - boolean containerNative boolean containerEnabled diff --git a/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy b/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy index bfcb34d4da..69b95e689a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy @@ -418,15 +418,15 @@ class TaskProcessor { if ( !taskBody ) throw new IllegalStateException("Missing task body for process `$name`") - // -- check that input set defines at least two elements - def invalidInputSet = config.getInputs().find { it instanceof TupleInParam && it.inner.size()<2 } - if( invalidInputSet ) - checkWarn "Input `tuple` must define at least two component -- Check process `$name`" + // -- check that input tuple defines at least two elements + def invalidInputTuple = config.getInputs().find { it instanceof TupleInParam && it.inner.size()<2 } + if( invalidInputTuple ) + checkWarn "Input `tuple` must define at least two elements -- Check process `$name`" - // -- check that output set defines at least two elements - def invalidOutputSet = config.getOutputs().find { it instanceof TupleOutParam && it.inner.size()<2 } - if( invalidOutputSet ) - checkWarn "Output `tuple` must define at least two component -- Check process `$name`" + // -- check that output tuple defines at least two elements + def invalidOutputTuple = config.getOutputs().find { it instanceof TupleOutParam && it.inner.size()<2 } + if( invalidOutputTuple ) + checkWarn "Output `tuple` must define at least two elements -- Check process `$name`" /** * Verify if this process run only one time @@ -610,7 +610,7 @@ class TaskProcessor { currentTask.set(task) // -- validate input lengths - validateInputSets(values) + validateInputTuples(values) // -- map the inputs to a map and use to delegate closure values interpolation final secondPass = [:] @@ -640,13 +640,13 @@ class TaskProcessor { } @Memoized - private List getDeclaredInputSet() { + private List getDeclaredInputTuple() { getConfig().getInputs().ofType(TupleInParam) } - protected void validateInputSets( List values ) { + protected void validateInputTuples( List values ) { - def declaredSets = getDeclaredInputSet() + def declaredSets = getDeclaredInputTuple() for( int i=0; i body ) { - if( NF.isDsl2() ) { - def process = new ProcessDef(this,body,name) - meta.addDefinition(process) - } - else { - throw new UnsupportedOperationException("DSL1 is not supported anymore") - } + final process = new ProcessDef(this,body,name) + meta.addDefinition(process) } /** diff --git a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy index 9ea328347a..3dd0ea1aad 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy @@ -56,17 +56,6 @@ class IncludeDef { @PackageScope Map addedParams private Session session - @Deprecated - IncludeDef( String module ) { - final msg = "Anonymous module inclusion is deprecated -- Replace `include '${module}'` with `include { MODULE_NAME } from '${module}'`" - if( NF.isDsl2() ) - throw new DeprecationException(msg) - log.warn msg - this.path = module - this.modules = new ArrayList<>(1) - this.modules << new Module(null,null) - } - IncludeDef(TokenVar token, String alias=null) { def component = token.name; if(alias) component += " as $alias" def msg = "Unwrapped module inclusion is deprecated -- Replace `include $component from './MODULE/PATH'` with `include { $component } from './MODULE/PATH'`" diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ProcessConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ProcessConfig.groovy index d88a11cf1d..a9c2a86cf9 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ProcessConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ProcessConfig.groovy @@ -82,7 +82,6 @@ class ProcessConfig implements Map, Cloneable { 'time', // input-output qualifiers 'file', - 'set', 'val', 'each', 'env', @@ -536,12 +535,6 @@ class ProcessConfig implements Map, Cloneable { new EachInParam(this).bind(obj) } - InParam _in_set( Object... obj ) { - final msg = "Input of type `set` is deprecated -- Use `tuple` instead" - if( NF.isDsl2() ) throw new DeprecationException(msg) - new TupleInParam(this).bind(obj) - } - InParam _in_tuple( Object... obj ) { new TupleInParam(this).bind(obj) } @@ -606,12 +599,6 @@ class ProcessConfig implements Map, Cloneable { } } - OutParam _out_set( Object... obj ) { - final msg = "Output of type `set` is deprecated -- Use `tuple` instead" - if( NF.isDsl2() ) throw new DeprecationException(msg) - new TupleOutParam(this) .bind(obj) - } - OutParam _out_tuple( Object... obj ) { new TupleOutParam(this) .bind(obj) } diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ScriptBinding.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ScriptBinding.groovy index 7cde04e282..09293c144b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ScriptBinding.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ScriptBinding.groovy @@ -186,9 +186,7 @@ class ScriptBinding extends WorkflowBinding { @Override void setVariable( String name, Object value ) { if( name == 'channel' ) { - final msg = 'The use of the identifier `channel` as variable name is discouraged and will be deprecated in a future version' - if( NF.isDsl2() ) throw new DeprecationException(msg) - log.warn(msg) + throw new IllegalAccessException("The use of the identifier `$name` as variable name is not allowed") } if( name != 'args' && name != 'params' ) super.setVariable(name, value) diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy index 726253eafe..784b40d784 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy @@ -97,8 +97,6 @@ class ScriptMeta { /** The module components included in the script */ private Map imports = new HashMap<>(10) - private List dsl1ProcessNames - /** Whenever it's a module script or the main script */ private boolean module @@ -162,25 +160,6 @@ class ScriptMeta { } } - /* - * This method invocation is made by the NF AST transformer to pass - * the process names declared in the workflow script. This is only required - * for DSL1 script. - * - * When using DSL2 process names can be discovered during - * the script execution since, the process declaration is de-coupled by the - * process invocations. - */ - @PackageScope - void setDsl1ProcessNames(List names) { - this.dsl1ProcessNames = names - } - - @PackageScope - List getDsl1ProcessNames() { - dsl1ProcessNames ?: Collections.emptyList() - } - @PackageScope static ScriptMeta register(BaseScript script) { def meta = new ScriptMeta(script) @@ -277,9 +256,6 @@ class ScriptMeta { } Set getProcessNames() { - if( NF.dsl1 ) - return new HashSet(getDsl1ProcessNames()) - def result = new HashSet(definitions.size() + imports.size()) // local definitions for( def item : definitions.values() ) { @@ -295,9 +271,6 @@ class ScriptMeta { } Set getLocalProcessNames() { - if( NF.dsl1 ) - return new HashSet(getDsl1ProcessNames()) - def result = new HashSet(definitions.size() + imports.size()) // local definitions for( def item : definitions.values() ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ScriptTokens.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ScriptTokens.groovy index 8edabdacb3..c7d38a1b7c 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ScriptTokens.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ScriptTokens.groovy @@ -36,10 +36,10 @@ class TokenVar { } /** - * A token used by the DSL to identify a 'file' declaration in a 'set' parameter, for example: + * A token used by the DSL to identify a 'file' declaration in a 'tuple' parameter, for example: *

  *      input:
- *      set( file('name'), ... )
+ *      tuple( file('name'), ... )
  *      
* */ @@ -49,10 +49,10 @@ class TokenFileCall { } /** - * A token used by the DSL to identify a 'path' declaration in a 'set' parameter, for example: + * A token used by the DSL to identify a 'path' declaration in a 'tuple' parameter, for example: *
  *      input:
- *      set( path('name'), ... )
+ *      tuple( path('name'), ... )
  *      
* */ @@ -99,7 +99,7 @@ class TokenStdoutCall { } * Token used by the DSL to identify a environment variable declaration, like this *
  *     input:
- *     set( env(X), ... )
+ *     tuple( env(X), ... )
  *     
  */
 @ToString
@@ -114,10 +114,10 @@ class TokenEnvCall {
  * This class is used to identify a 'val' when used like in this example:
  * 
  *  input:
- *  set ( val(x), ...  )
+ *  tuple ( val(x), ...  )
  *
  *  output:
- *  set( val(y), ...  )
+ *  tuple( val(y), ...  )
  *
  * 
* diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/BaseInParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/BaseInParam.groovy index 12d396414f..efe613805b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/BaseInParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/BaseInParam.groovy @@ -80,23 +80,9 @@ abstract class BaseInParam extends BaseParam implements InParam { return CH.getReadChannel(value) } - if( NF.isDsl2() ) { - final result = CH.value() - result.bind(value) - return result - } - - // wrap any collections with a DataflowQueue - if( value instanceof Collection ) { - return CH.emitAndClose(CH.queue(), value) - } - - // wrap any array with a DataflowQueue - if ( value && value.class.isArray() ) { - return CH.emitAndClose(CH.queue(), value as Collection) - } - - value!=null ? CH.value(value) : CH.value() + final result = CH.value() + result.bind(value) + return result } @@ -169,13 +155,6 @@ abstract class BaseInParam extends BaseParam implements InParam { throw new IllegalArgumentException(message) } - BaseInParam from( def obj ) { - if( NF.isDsl2() ) - throw new ScriptRuntimeException("Process clause `from` should not be provided when using DSL 2") - setFrom(obj) - return this - } - void setFrom( obj ) { checkFromNotNull(obj) fromObject = obj @@ -189,22 +168,6 @@ abstract class BaseInParam extends BaseParam implements InParam { throw new IllegalStateException("Missing input channel") } - BaseInParam from( Object... obj ) { - - def normalize = obj.collect { - if( it instanceof DataflowReadChannel ) - throw new IllegalArgumentException("Multiple channels are not allowed on 'from' input declaration") - - if( it instanceof Closure ) - return it.call() - else - it - } - - fromObject = normalize as List - return this - } - def decodeInputs( List inputs ) { final UNDEF = -1 as short def value = inputs[index] diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/BaseOutParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/BaseOutParam.groovy index bbbd9cd785..8cf9408175 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/BaseOutParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/BaseOutParam.groovy @@ -20,12 +20,10 @@ import groovy.transform.PackageScope import groovy.util.logging.Slf4j import groovyx.gpars.dataflow.DataflowWriteChannel import nextflow.NF -import nextflow.exception.ScriptRuntimeException import nextflow.extension.CH import nextflow.script.ProcessConfig import nextflow.script.TokenVar import nextflow.util.ConfigHelper - /** * Model a process generic output parameter * @@ -64,10 +62,8 @@ abstract class BaseOutParam extends BaseParam implements OutParam { void lazyInit() { - if( intoObj instanceof TokenVar[] ) { - if( NF.dsl2 ) - throw new IllegalArgumentException("Not a valid output channel argument: $intoObj") - for( def it : intoObj ) { lazyInitImpl(it) } + if( intoObj instanceof TokenVar || intoObj instanceof TokenVar[] ) { + throw new IllegalArgumentException("Not a valid output channel argument: $intoObj") } else if( intoObj != null ) { lazyInitImpl(intoObj) @@ -85,14 +81,9 @@ abstract class BaseOutParam extends BaseParam implements OutParam { @PackageScope void lazyInitImpl( def target ) { - def channel = null - if( target instanceof TokenVar ) { - assert !NF.dsl2 - channel = outputValToChannel(target.name) - } - else if( target != null ) { - channel = outputValToChannel(target) - } + final channel = (target != null) + ? outputValToChannel(target) + : null if( channel ) { outChannels.add(channel) @@ -158,20 +149,6 @@ abstract class BaseOutParam extends BaseParam implements OutParam { return this } - BaseOutParam into( def value ) { - if( NF.dsl2 ) - throw new ScriptRuntimeException("Process clause `into` should not be provided when using DSL 2") - this.intoObj = value - return this - } - - BaseOutParam into( TokenVar... vars ) { - if( NF.dsl2 ) - throw new ScriptRuntimeException("Process clause `into` should not be provided when using DSL 2") - intoObj = vars - return this - } - void setInto( Object obj ) { intoObj = obj } @@ -181,12 +158,6 @@ abstract class BaseOutParam extends BaseParam implements OutParam { return outChannels ? outChannels.get(0) : null } - @Deprecated - List getOutChannels() { - init() - return outChannels - } - String getName() { if( nameObj != null ) return nameObj.toString() diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/FileInParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/FileInParam.groovy index 8bebf83d70..397a759b15 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/FileInParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/FileInParam.groovy @@ -55,13 +55,6 @@ class FileInParam extends BaseInParam implements ArityParam, PathQualifier { return this } - // the ability to pass a closure as file name has been replaced by - // lazy gstring -- this should be deprecated - if( obj instanceof Closure && !NF.dsl2 ) { - filePattern = obj - return this - } - throw new IllegalArgumentException() } diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/FileOutParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/FileOutParam.groovy index a55cd957c7..97721ea7b2 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/FileOutParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/FileOutParam.groovy @@ -92,63 +92,6 @@ class FileOutParam extends BaseOutParam implements OutParam, ArityParam, Optiona */ boolean isDynamic() { dynamicObj || gstring != null } - @Deprecated - FileOutParam separatorChar( String value ) { - if( NF.dsl2 ) throw new DeprecationException("Option `separatorChar is not supported anymore") - this.separatorChar = value - return this - } - - @Deprecated - FileOutParam includeInputs( boolean flag ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `includeInputs $flag` with `, includeInputs: $flag` ") - this.includeInputs = flag - return this - } - - @Deprecated - FileOutParam includeHidden( boolean flag ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `includeHidden $flag` with `, includeHidden: $flag`") - this.hidden = flag - return this - } - - @Deprecated - FileOutParam hidden( boolean flag ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `hidden $flag` with use `, hidden: $flag`") - this.hidden = flag - return this - } - - @Deprecated - FileOutParam type( String value ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `type $value` with `, type: $value`") - assert value in ['file','dir','any'] - type = value - return this - } - - @Deprecated - FileOutParam maxDepth( int value ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `maxDepth $value` with `, maxDepth: $value`") - maxDepth = value - return this - } - - @Deprecated - FileOutParam followLinks( boolean value ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `followLinks $value` with `, followLinks: $value`") - followLinks = value - return this - } - - @Deprecated - FileOutParam glob( boolean value ) { - if( NF.dsl2 ) throw new DeprecationException("Deprecated syntax error - replace `glob $value` with `, glob: $value` ") - glob = value - return this - } - @Override BaseOutParam bind( obj ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/InParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/InParam.groovy index 35c95ecac0..d41e3c5c28 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/InParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/InParam.groovy @@ -31,10 +31,6 @@ interface InParam extends Cloneable { Object getRawChannel() - InParam from( Object value ) - - InParam from( Object... values ) - short index short mapIndex diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/OutParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/OutParam.groovy index 2a1570a2f3..e145beec56 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/OutParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/OutParam.groovy @@ -31,17 +31,6 @@ interface OutParam extends Cloneable { */ String getName() - /** - * Defines the channel to which bind the output(s) in the script context - * - * @param value It can be a string representing a channel variable name in the script context. If - * the variable does not exist it creates a {@code DataflowVariable} in the script with that name. - * If the specified {@code value} is a {@code DataflowWriteChannel} object, use this object - * as the output channel - * @return - */ - OutParam into( def value ) - /** * @return The output channel instance */ diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/TupleInParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/TupleInParam.groovy index 4c2bc3ff69..6255954eec 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/TupleInParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/TupleInParam.groovy @@ -35,7 +35,7 @@ class TupleInParam extends BaseInParam { protected List inner = [] - @Override String getTypeName() { 'set' } + @Override String getTypeName() { 'tuple' } List getInner() { inner } @@ -56,11 +56,7 @@ class TupleInParam extends BaseInParam { for( def item : obj ) { if( item instanceof TokenVar ) { - if( NF.dsl2 ) { - final msg = "Unqualified input value declaration has been deprecated - replace `tuple ${item.name},..` with `tuple val(${item.name}),..`" - throw new DeprecationException(msg) - } - newItem(ValueInParam).bind(item) + throw new IllegalArgumentException("Unqualified input value declaration is not allowed - replace `tuple ${item.name},..` with `tuple val(${item.name}),..`") } else if( item instanceof TokenFileCall ) { newItem(FileInParam).bind( item.target ) @@ -72,11 +68,7 @@ class TupleInParam extends BaseInParam { .bind( item.target ) } else if( item instanceof Map ) { - if( NF.dsl2 ) { - final msg = "Unqualified input file declaration has been deprecated - replace `tuple $item,..` with `tuple path(${item.key}, stageAs:'${item.value}'),..`" - throw new DeprecationException(msg) - } - newItem(FileInParam).bind(item) + throw new IllegalArgumentException("Unqualified input file declaration is not allowed - replace `tuple $item,..` with `tuple path(${item.key}, stageAs:'${item.value}'),..`") } else if( item instanceof TokenValCall ) { newItem(ValueInParam).bind(item.val) @@ -88,17 +80,13 @@ class TupleInParam extends BaseInParam { newItem(StdInParam) } else if( item instanceof GString ) { - if( NF.dsl2 ) - throw new DeprecationException("Unqualified input file declaration has been deprecated - replace `tuple \"$item\".. with `tuple path(\"$item\")..`") - newItem(FileInParam).bind(item) + throw new IllegalArgumentException("Unqualified input file declaration is not allowed - replace `tuple \"$item\".. with `tuple path(\"$item\")..`") } else if( item == '-' ) { newItem(StdInParam) } else if( item instanceof String ) { - if( NF.dsl2 ) - throw new DeprecationException("Unqualified input file declaration has been deprecated - replace `tuple '$item',..` with `tuple path('$item'),..`") - newItem(FileInParam).bind(item) + throw new IllegalArgumentException("Unqualified input file declaration is not allowed - replace `tuple '$item',..` with `tuple path('$item'),..`") } else throw new IllegalArgumentException() diff --git a/modules/nextflow/src/main/groovy/nextflow/script/params/TupleOutParam.groovy b/modules/nextflow/src/main/groovy/nextflow/script/params/TupleOutParam.groovy index d038eb7331..e0de9e24ed 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/params/TupleOutParam.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/params/TupleOutParam.groovy @@ -51,9 +51,7 @@ class TupleOutParam extends BaseOutParam implements OptionalParam { for( def item : obj ) { if( item instanceof TokenVar ) { - if( NF.dsl2 ) - throw new DeprecationException("Unqualified output value declaration has been deprecated - replace `tuple ${item.name},..` with `tuple val(${item.name}),..`") - create(ValueOutParam).bind(item) + throw new IllegalArgumentException("Unqualified output value declaration is not allowed - replace `tuple ${item.name},..` with `tuple val(${item.name}),..`") } else if( item instanceof TokenValCall ) { create(ValueOutParam).bind(item.val) @@ -62,17 +60,13 @@ class TupleOutParam extends BaseOutParam implements OptionalParam { create(EnvOutParam).bind(item.val) } else if( item instanceof GString ) { - if( NF.dsl2 ) - throw new DeprecationException("Unqualified output path declaration has been deprecated - replace `tuple \"$item\",..` with `tuple path(\"$item\"),..`") - create(FileOutParam).bind(item) + throw new IllegalArgumentException("Unqualified output path declaration is not allowed - replace `tuple \"$item\",..` with `tuple path(\"$item\"),..`") } else if( item instanceof TokenStdoutCall || item == '-' ) { create(StdOutParam).bind('-') } else if( item instanceof String ) { - if( NF.dsl2 ) - throw new DeprecationException("Unqualified output path declaration has been deprecated - replace `tuple '$item',..` with `tuple path('$item'),..`") - create(FileOutParam).bind(item) + throw new IllegalArgumentException("Unqualified output path declaration is not allowed - replace `tuple '$item',..` with `tuple path('$item'),..`") } else if( item instanceof TokenFileCall ) { // note that 'filePattern' can be a string or a GString diff --git a/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy b/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy index 3305d23121..a8c89e4266 100644 --- a/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy @@ -1,24 +1,34 @@ package nextflow.ast +import nextflow.Session import nextflow.script.BaseScript import nextflow.script.ScriptMeta +import nextflow.script.ScriptParser import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.control.MultipleCompilationErrorsException import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer -import test.BaseSpec +import test.Dsl2Spec +import test.MockExecutorFactory /** * * @author Paolo Di Tommaso */ -class NextflowDSLImplTest extends BaseSpec { +class NextflowDSLImplTest extends Dsl2Spec { - def 'should fetch method names' () { - - given: + def createCompilerConfig() { def config = new CompilerConfiguration() config.setScriptBaseClass(BaseScript.class.name) config.addCompilationCustomizers( new ASTTransformationCustomizer(NextflowDSL)) + return config + } + + + def 'should fetch method names' () { + + given: + def config = createCompilerConfig() + def SCRIPT = ''' def foo() { return 0 @@ -35,8 +45,8 @@ class NextflowDSLImplTest extends BaseSpec { process alpha { /hello/ } - ''' + when: new GroovyShell(config).parse(SCRIPT) then: @@ -45,12 +55,9 @@ class NextflowDSLImplTest extends BaseSpec { def 'should not allow duplicate processes' () { given: - def config = new CompilerConfiguration() - config.setScriptBaseClass(BaseScript.class.name) - config.addCompilationCustomizers( new ASTTransformationCustomizer(NextflowDSL)) + def config = createCompilerConfig() def SCRIPT = ''' - process alpha { /hello/ } @@ -58,7 +65,6 @@ class NextflowDSLImplTest extends BaseSpec { process alpha { /world/ } - ''' when: @@ -71,12 +77,9 @@ class NextflowDSLImplTest extends BaseSpec { def 'should not allow duplicate workflow' () { given: - def config = new CompilerConfiguration() - config.setScriptBaseClass(BaseScript.class.name) - config.addCompilationCustomizers( new ASTTransformationCustomizer(NextflowDSL)) + def config = createCompilerConfig() def SCRIPT = ''' - workflow alpha { /hello/ } @@ -84,7 +87,6 @@ class NextflowDSLImplTest extends BaseSpec { workflow alpha { /world/ } - ''' when: @@ -97,29 +99,24 @@ class NextflowDSLImplTest extends BaseSpec { def 'should throw illegal name exception' () { given: - def config = new CompilerConfiguration() - config.setScriptBaseClass(BaseScript.class.name) - config.addCompilationCustomizers( new ASTTransformationCustomizer(NextflowDSL)) + def config = createCompilerConfig() when: def SCRIPT = ''' workflow 'foo:bar' { /hello/ } - ''' new GroovyShell(config).parse(SCRIPT) then: def e = thrown(MultipleCompilationErrorsException) e.message.contains 'Process and workflow names cannot contain colon character' - when: SCRIPT = ''' process 'foo:bar' { /hello/ } - ''' new GroovyShell(config).parse(SCRIPT) then: @@ -129,12 +126,12 @@ class NextflowDSLImplTest extends BaseSpec { def 'should set process name in the script meta' () { given: - def config = new CompilerConfiguration() - config.setScriptBaseClass(BaseScript.class.name) - config.addCompilationCustomizers( new ASTTransformationCustomizer(NextflowDSL)) + def session = new Session() + session.executorFactory = new MockExecutorFactory() + and: + def parser = new ScriptParser(session) def SCRIPT = ''' - process alpha { /hello/ } @@ -142,14 +139,16 @@ class NextflowDSLImplTest extends BaseSpec { process beta { /world/ } - + + workflow { + alpha(); beta() + } ''' when: - def script = new GroovyShell(config).parse(SCRIPT) + parser.runScript(SCRIPT) then: - ScriptMeta.get(script).getDsl1ProcessNames() == ['alpha', 'beta'] - ScriptMeta.get(script).getProcessNames() == ['alpha', 'beta'] as Set + ScriptMeta.get(parser.getScript()).getProcessNames() == ['alpha', 'beta'] as Set } } diff --git a/modules/nextflow/src/test/groovy/nextflow/container/ApptainerBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/ApptainerBuilderTest.groovy index 6552918ea0..0f4404cf95 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/ApptainerBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/ApptainerBuilderTest.groovy @@ -60,14 +60,14 @@ class ApptainerBuilderTest extends Specification { .addMount(path2) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .addMount(path1) .addMount(path1) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /foo/data/file1 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /foo/data/file1 -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .addMount(path1) @@ -75,13 +75,13 @@ class ApptainerBuilderTest extends Specification { .params(autoMounts: true) .params(readOnlyInputs: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .addMount(path3) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /bar/data\\ file -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer run --no-home --pid -B /bar/data\\ file -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .params(newPidNamespace: false) @@ -102,17 +102,17 @@ class ApptainerBuilderTest extends Specification { expect: new ApptainerBuilder('busybox') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' new ApptainerBuilder('busybox') .params(engineOptions: '-q -v') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer -q -v exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer -q -v exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' new ApptainerBuilder('busybox') .params(runOptions: '--contain --writable') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$PWD" --contain --writable busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" --contain --writable busybox' new ApptainerBuilder('ubuntu') .addMount(path1) @@ -125,14 +125,14 @@ class ApptainerBuilderTest extends Specification { .addMount(path2) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .addMount(path1) .addMount(path1) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /foo/data/file1 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /foo/data/file1 -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .addMount(path1) @@ -140,18 +140,18 @@ class ApptainerBuilderTest extends Specification { .params(autoMounts: true) .params(readOnlyInputs: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .addMount(path3) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /bar/data\\ file -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B /bar/data\\ file -B "$NXF_TASK_WORKDIR" ubuntu' new ApptainerBuilder('ubuntu') .params(newPidNamespace: false) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home -B "$NXF_TASK_WORKDIR" ubuntu' } @@ -201,12 +201,12 @@ class ApptainerBuilderTest extends Specification { .addEnv('X=1') .addEnv(ALPHA:'aaa', BETA: 'bbb') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} APPTAINERENV_X="1" APPTAINERENV_ALPHA="aaa" APPTAINERENV_BETA="bbb" apptainer exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} APPTAINERENV_X="1" APPTAINERENV_ALPHA="aaa" APPTAINERENV_BETA="bbb" apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' new ApptainerBuilder('busybox') .addEnv('CUDA_VISIBLE_DEVICES') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} ${CUDA_VISIBLE_DEVICES:+APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"} apptainer exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} ${CUDA_VISIBLE_DEVICES:+APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"} apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' } @@ -216,17 +216,17 @@ class ApptainerBuilderTest extends Specification { when: def cmd = new ApptainerBuilder('ubuntu.img').build().getRunCommand() then: - cmd == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$PWD" ubuntu.img' + cmd == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" ubuntu.img' when: cmd = new ApptainerBuilder('ubuntu.img').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$PWD" ubuntu.img bwa --this --that file.fastq' + cmd == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" ubuntu.img bwa --this --that file.fastq' when: cmd = new ApptainerBuilder('ubuntu.img').params(entry:'/bin/sh').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$PWD" ubuntu.img /bin/sh -c "cd $PWD; bwa --this --that file.fastq"' + cmd == 'set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} apptainer exec --no-home --pid -B "$NXF_TASK_WORKDIR" ubuntu.img /bin/sh -c "cd $NXF_TASK_WORKDIR; bwa --this --that file.fastq"' } @Unroll diff --git a/modules/nextflow/src/test/groovy/nextflow/container/CharliecloudBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/CharliecloudBuilderTest.groovy index d049b59ed9..10b7ff99ae 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/CharliecloudBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/CharliecloudBuilderTest.groovy @@ -38,34 +38,34 @@ class CharliecloudBuilderTest extends Specification { expect: new CharliecloudBuilder('busybox') .build() - .runCommand == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b "$PWD" busybox --' + .runCommand == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b "$NXF_TASK_WORKDIR" busybox --' new CharliecloudBuilder('busybox') .params(runOptions: '-j --no-home') .build() - .runCommand == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b "$PWD" -j --no-home busybox --' + .runCommand == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b "$NXF_TASK_WORKDIR" -j --no-home busybox --' new CharliecloudBuilder('busybox') .params(temp: '/foo') .build() - .runCommand == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b /foo:/tmp -b "$PWD" busybox --' + .runCommand == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b /foo:/tmp -b "$NXF_TASK_WORKDIR" busybox --' new CharliecloudBuilder('busybox') .addEnv('X=1') .addEnv(ALPHA:'aaa', BETA: 'bbb') .build() - .runCommand == 'ch-run --unset-env="*" -c "$PWD" --set-env -w --set-env=X=1 --set-env=ALPHA=aaa --set-env=BETA=bbb -b "$PWD" busybox --' + .runCommand == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w --set-env=X=1 --set-env=ALPHA=aaa --set-env=BETA=bbb -b "$NXF_TASK_WORKDIR" busybox --' new CharliecloudBuilder('ubuntu') .addMount(path1) .build() - .runCommand == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b /foo/data/file1 -b "$PWD" ubuntu --' + .runCommand == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b /foo/data/file1 -b "$NXF_TASK_WORKDIR" ubuntu --' new CharliecloudBuilder('ubuntu') .addMount(path1) .addMount(path2) .build() - .runCommand == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b /foo/data/file1 -b /bar/data/file2 -b "$PWD" ubuntu --' + .runCommand == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b /foo/data/file1 -b /bar/data/file2 -b "$NXF_TASK_WORKDIR" ubuntu --' } def db_file = Paths.get('/home/db') @@ -74,27 +74,27 @@ class CharliecloudBuilderTest extends Specification { when: def cmd = new CharliecloudBuilder('ubuntu').build().getRunCommand() then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b "$PWD" ubuntu --' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b "$NXF_TASK_WORKDIR" ubuntu --' when: cmd = new CharliecloudBuilder('ubuntu').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b "$PWD" ubuntu -- bwa --this --that file.fastq' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b "$NXF_TASK_WORKDIR" ubuntu -- bwa --this --that file.fastq' when: cmd = new CharliecloudBuilder('ubuntu').params(entry:'/bin/sh').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b "$PWD" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b "$NXF_TASK_WORKDIR" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' when: cmd = new CharliecloudBuilder('ubuntu').params(entry:'/bin/sh').params(readOnlyInputs: 'true').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -b "$PWD" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -b "$NXF_TASK_WORKDIR" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' when: cmd = new CharliecloudBuilder('ubuntu').params(entry:'/bin/sh').params(readOnlyInputs: 'false').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b "$PWD" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b "$NXF_TASK_WORKDIR" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' when: cmd = new CharliecloudBuilder('ubuntu') @@ -104,7 +104,7 @@ class CharliecloudBuilderTest extends Specification { .params(readOnlyInputs: 'true') .build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -b /home -b "$PWD" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -b /home -b "$NXF_TASK_WORKDIR" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' when: cmd = new CharliecloudBuilder('ubuntu') @@ -115,7 +115,7 @@ class CharliecloudBuilderTest extends Specification { .build() .getRunCommand('bwa --this --that file.fastq') then: - cmd == 'ch-run --unset-env="*" -c "$PWD" --set-env -w -b /home/db -b "$PWD" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' + cmd == 'ch-run --unset-env="*" -c "$NXF_TASK_WORKDIR" --set-env -w -b /home/db -b "$NXF_TASK_WORKDIR" ubuntu -- /bin/sh -c "bwa --this --that file.fastq"' } @Unroll diff --git a/modules/nextflow/src/test/groovy/nextflow/container/ContainerConfigTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/ContainerConfigTest.groovy index 6c87437eb7..99ed6439fe 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/ContainerConfigTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/ContainerConfigTest.groovy @@ -61,6 +61,27 @@ class ContainerConfigTest extends Specification { } + + def 'should validate oci mode' () { + + when: + def cfg = new ContainerConfig(OPTS) + then: + cfg.singularityOciMode() == EXPECTED + + where: + OPTS | EXPECTED + [:] | false + [oci:false] | false + [oci:true] | false + [engine:'apptainer', oci:true] | false + [engine:'docker', oci:true] | false + [engine:'singularity'] | false + [engine:'singularity', oci:false] | false + [engine:'singularity', oci:true] | true + + } + def 'should get fusion options' () { when: def cfg = new ContainerConfig(OPTS) diff --git a/modules/nextflow/src/test/groovy/nextflow/container/ContainerHandlerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/ContainerHandlerTest.groovy index 43129de399..7459c9de21 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/ContainerHandlerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/ContainerHandlerTest.groovy @@ -200,11 +200,12 @@ class ContainerHandlerTest extends Specification { result == 'shifter://image' } + @Unroll def 'test normalize method for singularity' () { given: def BASE = Paths.get('/abs/path/') - def handler = Spy(new ContainerHandler(engine: 'singularity', enabled: true, baseDir: BASE)) + def handler = Spy(new ContainerHandler(engine: 'singularity', enabled: true, oci:OCI, baseDir: BASE)) when: def result = handler.normalizeImageName(IMAGE) @@ -215,17 +216,21 @@ class ContainerHandlerTest extends Specification { result == EXPECTED where: - IMAGE | NORMALIZED | X | EXPECTED - null | null | 0 | null - '' | null | 0 | null - '/abs/path/bar.img' | '/abs/path/bar.img' | 0 | '/abs/path/bar.img' - '/abs/path bar.img' | '/abs/path bar.img' | 0 | '/abs/path\\ bar.img' - 'file:///abs/path/bar.img' | '/abs/path/bar.img' | 0 | '/abs/path/bar.img' - 'foo.img' | Paths.get('foo.img').toAbsolutePath().toString() | 0 | Paths.get('foo.img').toAbsolutePath().toString() - 'shub://busybox' | 'shub://busybox' | 1 | '/path/to/busybox' - 'docker://library/busybox' | 'docker://library/busybox' | 1 | '/path/to/busybox' - 'foo' | 'docker://foo' | 1 | '/path/to/foo' - 'library://pditommaso/foo/bar.sif:latest' | 'library://pditommaso/foo/bar.sif:latest' | 1 | '/path/to/foo-bar-latest.img' + IMAGE | NORMALIZED | OCI | X | EXPECTED + null | null | false | 0 | null + '' | null | false | 0 | null + '/abs/path/bar.img' | '/abs/path/bar.img' | false | 0 | '/abs/path/bar.img' + '/abs/path bar.img' | '/abs/path bar.img' | false | 0 | '/abs/path\\ bar.img' + 'file:///abs/path/bar.img' | '/abs/path/bar.img' | false | 0 | '/abs/path/bar.img' + 'foo.img' | Paths.get('foo.img').toAbsolutePath().toString() | false | 0 | Paths.get('foo.img').toAbsolutePath().toString() + 'shub://busybox' | 'shub://busybox' | false | 1 | '/path/to/busybox' + 'docker://library/busybox' | 'docker://library/busybox' | false | 1 | '/path/to/busybox' + 'foo' | 'docker://foo' | false | 1 | '/path/to/foo' + 'library://pditommaso/foo/bar.sif:latest' | 'library://pditommaso/foo/bar.sif:latest' | false | 1 | '/path/to/foo-bar-latest.img' + and: + 'docker://library/busybox' | 'docker://library/busybox' | true | 0 | 'docker://library/busybox' + 'shub://busybox' | 'shub://busybox' | true | 1 | '/path/to/busybox' + } def 'should not invoke caching when engine is disabled' () { diff --git a/modules/nextflow/src/test/groovy/nextflow/container/DockerBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/DockerBuilderTest.groovy index fd94d44262..3a18743f3a 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/DockerBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/DockerBuilderTest.groovy @@ -38,10 +38,10 @@ class DockerBuilderTest extends Specification { def quotes = [ Paths.get('/folder with blanks/A'), Paths.get('/folder with blanks/B') ] expect: - builder.makeVolumes([]).toString() == '-v "$PWD":"$PWD" ' - builder.makeVolumes(files).toString() == '-v /folder:/folder -v "$PWD":"$PWD" ' - builder.makeVolumes(real).toString() == '-v /user/yo/nextflow:/user/yo/nextflow -v /db/pdb/local/data:/db/pdb/local/data -v "$PWD":"$PWD" ' - builder.makeVolumes(quotes).toString() == '-v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v "$PWD":"$PWD" ' + builder.makeVolumes([]).toString() == '-v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(files).toString() == '-v /folder:/folder -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(real).toString() == '-v /user/yo/nextflow:/user/yo/nextflow -v /db/pdb/local/data:/db/pdb/local/data -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(quotes).toString() == '-v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' } @@ -70,67 +70,67 @@ class DockerBuilderTest extends Specification { expect: new DockerBuilder('fedora') .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .addEnv(env) .build() - .runCommand == 'docker run -i -e "FOO=1" -e "BAR=hello world" -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i -e "FOO=1" -e "BAR=hello world" -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('ubuntu') .params(temp:'/hola') .build() - .runCommand == 'docker run -i -v /hola:/tmp -v "$PWD":"$PWD" -w "$PWD" ubuntu' + .runCommand == 'docker run -i -v /hola:/tmp -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu' new DockerBuilder('busybox') .params(sudo: true) .build() - .runCommand == 'sudo docker run -i -v "$PWD":"$PWD" -w "$PWD" busybox' + .runCommand == 'sudo docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" busybox' new DockerBuilder('busybox') .params(entry: '/bin/bash') .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash busybox' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --entrypoint /bin/bash busybox' new DockerBuilder('busybox') .params(runOptions: '-x --zeta') .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" -x --zeta busybox' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" -x --zeta busybox' new DockerBuilder('busybox') .params(userEmulation:true) .build() - .runCommand == 'docker run -i -u $(id -u) -e "HOME=${HOME}" -v /etc/passwd:/etc/passwd:ro -v /etc/shadow:/etc/shadow:ro -v /etc/group:/etc/group:ro -v $HOME:$HOME -v "$PWD":"$PWD" -w "$PWD" busybox' + .runCommand == 'docker run -i -u $(id -u) -e "HOME=${HOME}" -v /etc/passwd:/etc/passwd:ro -v /etc/shadow:/etc/shadow:ro -v /etc/group:/etc/group:ro -v $HOME:$HOME -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" busybox' new DockerBuilder('busybox') .setName('hola') .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --name hola busybox' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name hola busybox' new DockerBuilder('busybox') .params(engineOptions: '--tlsverify --tlscert="/path/to/my/cert"') .build() - .runCommand == 'docker --tlsverify --tlscert="/path/to/my/cert" run -i -v "$PWD":"$PWD" -w "$PWD" busybox' + .runCommand == 'docker --tlsverify --tlscert="/path/to/my/cert" run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" busybox' new DockerBuilder('fedora') .addEnv(env) .addMount(db_file) .addMount(db_file) // <-- add twice the same to prove that the final string won't contain duplicates .build() - .runCommand == 'docker run -i -e "FOO=1" -e "BAR=hello world" -v /home/db:/home/db -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i -e "FOO=1" -e "BAR=hello world" -v /home/db:/home/db -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .params(readOnlyInputs: true) .addMount(db_file) .build() - .runCommand == 'docker run -i -v /home/db:/home/db:ro -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i -v /home/db:/home/db:ro -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .params(mountFlags: 'Z') .addMount(db_file) .build() - .runCommand == 'docker run -i -v /home/db:/home/db:Z -v "$PWD":"$PWD":Z -w "$PWD" fedora' + .runCommand == 'docker run -i -v /home/db:/home/db:Z -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR":Z -w "$NXF_TASK_WORKDIR" fedora' } def 'test memory and cpuset' () { @@ -139,58 +139,58 @@ class DockerBuilderTest extends Specification { new DockerBuilder('fedora') .setCpus(2) .build() - .runCommand == 'docker run -i --cpu-shares 2048 -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --cpu-shares 2048 -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .setCpus(1) .build() - .runCommand == 'docker run -i --cpu-shares 1024 -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --cpu-shares 1024 -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .setCpus(8) .setCpuset('1,2') .build() - .runCommand == 'docker run -i --cpu-shares 8192 --cpuset-cpus 1,2 -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --cpu-shares 8192 --cpuset-cpus 1,2 -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .params(legacy: true) .setCpuset('1,2') .build() - .runCommand == 'docker run -i --cpuset 1,2 -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --cpuset 1,2 -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .params(legacy: true) .setCpus(1) .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .setMemory('10g') .build() - .runCommand == 'docker run -i --memory 10g -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --memory 10g -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .setMemory(new MemoryUnit('100M')) .build() - .runCommand == 'docker run -i --memory 100m -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --memory 100m -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .setCpuset('1-3') .setMemory(new MemoryUnit('100M')) .build() - .runCommand == 'docker run -i --cpuset-cpus 1-3 --memory 100m -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'docker run -i --cpuset-cpus 1-3 --memory 100m -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new DockerBuilder('fedora') .params(privileged: true) .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --privileged fedora' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --privileged fedora' new DockerBuilder('fedora') .params(device: '/dev/fuse') .params(capAdd: 'SYS_ADMIN') .build() - .runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --device /dev/fuse --cap-add SYS_ADMIN fedora' + .runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --device /dev/fuse --cap-add SYS_ADMIN fedora' } def 'test add mount'() { @@ -214,14 +214,14 @@ class DockerBuilderTest extends Specification { when: def docker = new DockerBuilder('busybox').setName('c1').build() then: - docker.runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --name c1 busybox' + docker.runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name c1 busybox' docker.removeCommand == 'docker rm c1' docker.killCommand == 'docker stop c1' when: docker = new DockerBuilder('busybox').setName('c2').params(sudo: true, remove: true).build() then: - docker.runCommand == 'sudo docker run -i -v "$PWD":"$PWD" -w "$PWD" --name c2 busybox' + docker.runCommand == 'sudo docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name c2 busybox' docker.removeCommand == 'sudo docker rm c2' docker.killCommand == 'sudo docker stop c2' @@ -229,7 +229,7 @@ class DockerBuilderTest extends Specification { when: docker = new DockerBuilder('busybox').setName('c3').params(remove: true).build() then: - docker.runCommand == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --name c3 busybox' + docker.runCommand == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name c3 busybox' docker.removeCommand == 'docker rm c3' docker.killCommand == 'docker stop c3' @@ -252,17 +252,17 @@ class DockerBuilderTest extends Specification { when: def cli = new DockerBuilder('ubuntu:14').build().getRunCommand() then: - cli == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" ubuntu:14' + cli == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14' when: cli = new DockerBuilder('ubuntu:14').build().getRunCommand('bwa --this --that file.fasta') then: - cli == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" ubuntu:14 bwa --this --that file.fasta' + cli == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14 bwa --this --that file.fasta' when: cli = new DockerBuilder('ubuntu:14').params(entry:'/bin/bash').build().getRunCommand('bwa --this --that file.fasta') then: - cli == 'docker run -i -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash ubuntu:14 -c "bwa --this --that file.fasta"' + cli == 'docker run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --entrypoint /bin/bash ubuntu:14 -c "bwa --this --that file.fasta"' } diff --git a/modules/nextflow/src/test/groovy/nextflow/container/PodmanBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/PodmanBuilderTest.groovy index 2d57837f84..3cfc0f2c36 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/PodmanBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/PodmanBuilderTest.groovy @@ -40,10 +40,10 @@ class PodmanBuilderTest extends Specification { def quotes = [ Paths.get('/folder with blanks/A'), Paths.get('/folder with blanks/B') ] expect: - builder.makeVolumes([]).toString() == '-v "$PWD":"$PWD" ' - builder.makeVolumes(files).toString() == '-v /folder:/folder -v "$PWD":"$PWD" ' - builder.makeVolumes(real).toString() == '-v /user/yo/nextflow:/user/yo/nextflow -v /db/pdb/local/data:/db/pdb/local/data -v "$PWD":"$PWD" ' - builder.makeVolumes(quotes).toString() == '-v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v "$PWD":"$PWD" ' + builder.makeVolumes([]).toString() == '-v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(files).toString() == '-v /folder:/folder -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(real).toString() == '-v /user/yo/nextflow:/user/yo/nextflow -v /db/pdb/local/data:/db/pdb/local/data -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(quotes).toString() == '-v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' and: builder.addMountWorkDir(false).makeVolumes([]).toString() == '' builder.addMountWorkDir(false).makeVolumes(files).toString() == '-v /folder:/folder ' @@ -74,67 +74,67 @@ class PodmanBuilderTest extends Specification { expect: new PodmanBuilder('fedora') .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new PodmanBuilder('fedora') .addEnv(env) .build() - .runCommand == 'podman run -i -e "FOO=1" -e "BAR=hello world" -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'podman run -i -e "FOO=1" -e "BAR=hello world" -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new PodmanBuilder('ubuntu') .params(temp:'/hola') .build() - .runCommand == 'podman run -i -v /hola:/tmp -v "$PWD":"$PWD" -w "$PWD" ubuntu' + .runCommand == 'podman run -i -v /hola:/tmp -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu' new PodmanBuilder('busybox') .params(entry: '/bin/bash') .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash busybox' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --entrypoint /bin/bash busybox' new PodmanBuilder('busybox') .params(runOptions: '-x --zeta') .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" -x --zeta busybox' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" -x --zeta busybox' new PodmanBuilder('busybox') .setName('hola') .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --name hola busybox' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name hola busybox' new PodmanBuilder('busybox') .params(engineOptions: '--tls-verify=false --cert-dir "/path/to/my/cert-dir"') .build() - .runCommand == 'podman --tls-verify=false --cert-dir "/path/to/my/cert-dir" run -i -v "$PWD":"$PWD" -w "$PWD" busybox' + .runCommand == 'podman --tls-verify=false --cert-dir "/path/to/my/cert-dir" run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" busybox' new PodmanBuilder('fedora') .addEnv(env) .addMount(db_file) .addMount(db_file) // <-- add twice the same to prove that the final string won't contain duplicates .build() - .runCommand == 'podman run -i -e "FOO=1" -e "BAR=hello world" -v /home/db:/home/db -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'podman run -i -e "FOO=1" -e "BAR=hello world" -v /home/db:/home/db -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new PodmanBuilder('fedora') .params(readOnlyInputs: true) .addMount(db_file) .build() - .runCommand == 'podman run -i -v /home/db:/home/db:ro -v "$PWD":"$PWD" -w "$PWD" fedora' + .runCommand == 'podman run -i -v /home/db:/home/db:ro -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new PodmanBuilder('fedora') .params(mountFlags: 'Z') .addMount(db_file) .build() - .runCommand == 'podman run -i -v /home/db:/home/db:Z -v "$PWD":"$PWD":Z -w "$PWD" fedora' + .runCommand == 'podman run -i -v /home/db:/home/db:Z -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR":Z -w "$NXF_TASK_WORKDIR" fedora' new PodmanBuilder('fedora') .params(privileged: true) .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --privileged fedora' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --privileged fedora' new PodmanBuilder('fedora') .params(device: '/dev/fuse') .params(capAdd: 'SYS_ADMIN') .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --device /dev/fuse --cap-add SYS_ADMIN fedora' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --device /dev/fuse --cap-add SYS_ADMIN fedora' } def 'test add mount'() { @@ -158,14 +158,14 @@ class PodmanBuilderTest extends Specification { when: def podman = new PodmanBuilder('busybox').setName('c1').build() then: - podman.runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --name c1 busybox' + podman.runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name c1 busybox' podman.removeCommand == 'podman rm c1' podman.killCommand == 'podman stop c1' when: podman = new PodmanBuilder('busybox').setName('c3').params(remove: true).build() then: - podman.runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --name c3 busybox' + podman.runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name c3 busybox' podman.removeCommand == 'podman rm c3' podman.killCommand == 'podman stop c3' @@ -188,17 +188,17 @@ class PodmanBuilderTest extends Specification { when: def cli = new PodmanBuilder('ubuntu:14').build().getRunCommand() then: - cli == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" ubuntu:14' + cli == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14' when: cli = new PodmanBuilder('ubuntu:14').build().getRunCommand('bwa --this --that file.fasta') then: - cli == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" ubuntu:14 bwa --this --that file.fasta' + cli == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14 bwa --this --that file.fasta' when: cli = new PodmanBuilder('ubuntu:14').params(entry:'/bin/bash').build().getRunCommand('bwa --this --that file.fasta') then: - cli == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash ubuntu:14 -c "bwa --this --that file.fasta"' + cli == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --entrypoint /bin/bash ubuntu:14 -c "bwa --this --that file.fasta"' } @@ -226,18 +226,18 @@ class PodmanBuilderTest extends Specification { new PodmanBuilder('fedora') .setCpus(3) .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --cpu-shares 3072 fedora' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --cpu-shares 3072 fedora' new PodmanBuilder('fedora') .setMemory(new MemoryUnit('100m')) .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --memory 100m fedora' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --memory 100m fedora' new PodmanBuilder('fedora') .setCpus(1) .setMemory(new MemoryUnit('400m')) .build() - .runCommand == 'podman run -i -v "$PWD":"$PWD" -w "$PWD" --cpu-shares 1024 --memory 400m fedora' + .runCommand == 'podman run -i -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --cpu-shares 1024 --memory 400m fedora' } } diff --git a/modules/nextflow/src/test/groovy/nextflow/container/SarusBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/SarusBuilderTest.groovy index 986f11d1ae..1f9037b582 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/SarusBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/SarusBuilderTest.groovy @@ -38,30 +38,30 @@ class SarusrBuilderTest extends Specification { expect: new SarusBuilder('busybox') .build() - .@runCommand == 'sarus run --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" busybox' + .@runCommand == 'sarus run --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" busybox' new SarusBuilder('busybox') .params(verbose: true) .build() - .@runCommand == 'sarus --verbose run --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" busybox' + .@runCommand == 'sarus --verbose run --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" busybox' new SarusBuilder('fedora') .addEnv([VAR_X:1, VAR_Y:2]) .addEnv("VAR_Z=3") .build() - .@runCommand == 'sarus run -e "VAR_X=1" -e "VAR_Y=2" -e "VAR_Z=3" --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" fedora' + .@runCommand == 'sarus run -e "VAR_X=1" -e "VAR_Y=2" -e "VAR_Z=3" --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' new SarusBuilder('busybox') .params(runOptions: '-x --zeta') .build() - .@runCommand == 'sarus run --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" -x --zeta busybox' + .@runCommand == 'sarus run --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" -x --zeta busybox' new SarusBuilder('fedora') .addEnv([VAR_X:1, VAR_Y:2]) .addMount(Paths.get('/home/db')) .addMount(Paths.get('/home/db')) // <-- add twice the same to prove that the final string won't contain duplicates .build() - .@runCommand == 'sarus run -e "VAR_X=1" -e "VAR_Y=2" --mount=type=bind,source=/home/db,destination=/home/db --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" fedora' + .@runCommand == 'sarus run -e "VAR_X=1" -e "VAR_Y=2" --mount=type=bind,source=/home/db,destination=/home/db --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" fedora' } @@ -72,7 +72,7 @@ class SarusrBuilderTest extends Specification { then: cli == '''\ sarus pull ubuntu:14 1>&2 - sarus run --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" ubuntu:14 + sarus run --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14 ''' .stripIndent().trim() @@ -81,7 +81,7 @@ class SarusrBuilderTest extends Specification { then: cli == '''\ sarus pull ubuntu:14 1>&2 - sarus run --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" ubuntu:14 bwa --this --that file.fasta + sarus run --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14 bwa --this --that file.fasta ''' .stripIndent().trim() @@ -90,7 +90,7 @@ class SarusrBuilderTest extends Specification { then: cli == '''\ sarus pull ubuntu:14 1>&2 - sarus run --mount=type=bind,source="$PWD",destination="$PWD" -w "$PWD" ubuntu:14 /bin/bash -c "bwa --this --that file.fasta" + sarus run --mount=type=bind,source="$NXF_TASK_WORKDIR",destination="$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" ubuntu:14 /bin/bash -c "bwa --this --that file.fasta" ''' .stripIndent().trim() diff --git a/modules/nextflow/src/test/groovy/nextflow/container/SingularityBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/SingularityBuilderTest.groovy index 661f459a29..dd475dc80d 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/SingularityBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/SingularityBuilderTest.groovy @@ -61,14 +61,14 @@ class SingularityBuilderTest extends Specification { .addMount(path2) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .addMount(path1) .addMount(path1) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /foo/data/file1 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /foo/data/file1 -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .addMount(path1) @@ -76,13 +76,13 @@ class SingularityBuilderTest extends Specification { .params(autoMounts: true) .params(readOnlyInputs: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .addMount(path3) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /bar/data\\ file -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity run --no-home --pid -B /bar/data\\ file -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .params(newPidNamespace: false) @@ -103,17 +103,17 @@ class SingularityBuilderTest extends Specification { expect: new SingularityBuilder('busybox') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' new SingularityBuilder('busybox') .params(engineOptions: '-q -v') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity -q -v exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity -q -v exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' new SingularityBuilder('busybox') .params(runOptions: '--contain --writable') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$PWD" --contain --writable busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" --contain --writable busybox' new SingularityBuilder('ubuntu') .params(autoMounts: false) @@ -125,14 +125,14 @@ class SingularityBuilderTest extends Specification { .addMount(path2) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /foo/data/file1 -B /bar/data/file2 -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .addMount(path1) .addMount(path1) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /foo/data/file1 -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /foo/data/file1 -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .addMount(path1) @@ -140,13 +140,13 @@ class SingularityBuilderTest extends Specification { .params(autoMounts: true) .params(readOnlyInputs: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /foo/data/file1:/foo/data/file1:ro -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .addMount(path3) .params(autoMounts: true) .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /bar/data\\ file -B "$PWD" ubuntu' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B /bar/data\\ file -B "$NXF_TASK_WORKDIR" ubuntu' new SingularityBuilder('ubuntu') .params(newPidNamespace: false) @@ -154,6 +154,11 @@ class SingularityBuilderTest extends Specification { .build() .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home ubuntu' + new SingularityBuilder('ubuntu') + .params(oci: true) + .build() + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${XDG_RUNTIME_DIR:+XDG_RUNTIME_DIR="$XDG_RUNTIME_DIR"} ${DBUS_SESSION_BUS_ADDRESS:+DBUS_SESSION_BUS_ADDRESS="$DBUS_SESSION_BUS_ADDRESS"} singularity exec --no-home --oci -B "$NXF_TASK_WORKDIR" ubuntu' + } def 'should mount home directory if specified' () { @@ -202,12 +207,12 @@ class SingularityBuilderTest extends Specification { .addEnv('X=1') .addEnv(ALPHA:'aaa', BETA: 'bbb') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_X="1" SINGULARITYENV_ALPHA="aaa" SINGULARITYENV_BETA="bbb" singularity exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_X="1" SINGULARITYENV_ALPHA="aaa" SINGULARITYENV_BETA="bbb" singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' new SingularityBuilder('busybox') .addEnv('CUDA_VISIBLE_DEVICES') .build() - .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${CUDA_VISIBLE_DEVICES:+SINGULARITYENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"} singularity exec --no-home --pid -B "$PWD" busybox' + .runCommand == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${CUDA_VISIBLE_DEVICES:+SINGULARITYENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"} singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" busybox' } @@ -217,17 +222,17 @@ class SingularityBuilderTest extends Specification { when: def cmd = new SingularityBuilder('ubuntu.img').build().getRunCommand() then: - cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$PWD" ubuntu.img' + cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" ubuntu.img' when: cmd = new SingularityBuilder('ubuntu.img').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$PWD" ubuntu.img bwa --this --that file.fastq' + cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" ubuntu.img bwa --this --that file.fastq' when: cmd = new SingularityBuilder('ubuntu.img').params(entry:'/bin/sh').build().getRunCommand('bwa --this --that file.fastq') then: - cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$PWD" ubuntu.img /bin/sh -c "cd $PWD; bwa --this --that file.fastq"' + cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} singularity exec --no-home --pid -B "$NXF_TASK_WORKDIR" ubuntu.img /bin/sh -c "cd $NXF_TASK_WORKDIR; bwa --this --that file.fastq"' } @Unroll diff --git a/modules/nextflow/src/test/groovy/nextflow/container/UdockerBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/container/UdockerBuilderTest.groovy index 07e374dc9a..3dac89771f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/container/UdockerBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/container/UdockerBuilderTest.groovy @@ -47,10 +47,10 @@ class UdockerBuilderTest extends Specification { def quotes = [ Paths.get('/folder with blanks/A'), Paths.get('/folder with blanks/B') ] expect: - builder.makeVolumes([]).toString() == '-v "$PWD":"$PWD" ' - builder.makeVolumes(files).toString() == '-v /folder:/folder -v "$PWD":"$PWD" ' - builder.makeVolumes(real).toString() == '-v /user/yo/nextflow:/user/yo/nextflow -v /db/pdb/local/data:/db/pdb/local/data -v "$PWD":"$PWD" ' - builder.makeVolumes(quotes).toString() == '-v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v "$PWD":"$PWD" ' + builder.makeVolumes([]).toString() == '-v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(files).toString() == '-v /folder:/folder -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(real).toString() == '-v /user/yo/nextflow:/user/yo/nextflow -v /db/pdb/local/data:/db/pdb/local/data -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' + builder.makeVolumes(quotes).toString() == '-v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" ' } @@ -63,38 +63,38 @@ class UdockerBuilderTest extends Specification { expect: new UdockerBuilder('fedora') .build() - .runCommandRaw == 'udocker.py run --rm -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "fedora:latest")' + .runCommandRaw == 'udocker.py run --rm -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "fedora:latest")' new UdockerBuilder('fedora') .addEnv(env) .build() - .runCommandRaw == 'udocker.py run --rm -e "FOO=1" -e "BAR=hello world" -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "fedora:latest")' + .runCommandRaw == 'udocker.py run --rm -e "FOO=1" -e "BAR=hello world" -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "fedora:latest")' new UdockerBuilder('fedora') .setCpuset('1,2') .build() - .runCommandRaw == 'udocker.py run --rm --cpuset-cpus=1,2 -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "fedora:latest")' + .runCommandRaw == 'udocker.py run --rm --cpuset-cpus=1,2 -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "fedora:latest")' new UdockerBuilder('fedora') .addMount(db_file) .addEnv(env) .build() - .runCommandRaw == 'udocker.py run --rm -e "FOO=1" -e "BAR=hello world" -v /home/db:/home/db -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "fedora:latest")' + .runCommandRaw == 'udocker.py run --rm -e "FOO=1" -e "BAR=hello world" -v /home/db:/home/db -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "fedora:latest")' new UdockerBuilder('busybox') .params(remove: false) .build() - .runCommandRaw == 'udocker.py run -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "busybox:latest")' + .runCommandRaw == 'udocker.py run -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "busybox:latest")' new UdockerBuilder('busybox') .params(runOptions: '-x --zeta') .build() - .runCommandRaw == 'udocker.py run --rm -v "$PWD":"$PWD" -w "$PWD" --bindhome -x --zeta $(udocker.py create "busybox:latest")' + .runCommandRaw == 'udocker.py run --rm -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome -x --zeta $(udocker.py create "busybox:latest")' new UdockerBuilder('busybox') .params(entry: '/bin/blah') .build() - .runCommandRaw == 'udocker.py run --rm -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "busybox:latest")' + .runCommandRaw == 'udocker.py run --rm -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "busybox:latest")' } @@ -109,7 +109,7 @@ class UdockerBuilderTest extends Specification { result == ''' ((udocker.py images | grep -E -o "^ubuntu:latest\\s") || udocker.py pull "ubuntu:latest")>/dev/null [[ $? != 0 ]] && echo "Udocker failed while pulling container \\`ubuntu:latest\\`" >&2 && exit 1 - udocker.py run --rm -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "ubuntu:latest") + udocker.py run --rm -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "ubuntu:latest") ''' .stripIndent().trim() @@ -126,7 +126,7 @@ class UdockerBuilderTest extends Specification { result == ''' ((udocker.py images | grep -E -o "^ubuntu:latest\\s") || udocker.py pull "ubuntu:latest")>/dev/null [[ $? != 0 ]] && echo "Udocker failed while pulling container \\`ubuntu:latest\\`" >&2 && exit 1 - udocker.py run --rm -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "ubuntu:latest") bwa --this --that + udocker.py run --rm -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "ubuntu:latest") bwa --this --that ''' .stripIndent().trim() @@ -141,7 +141,7 @@ class UdockerBuilderTest extends Specification { result == ''' ((udocker.py images | grep -E -o "^ubuntu:latest\\s") || udocker.py pull "ubuntu:latest")>/dev/null [[ $? != 0 ]] && echo "Udocker failed while pulling container \\`ubuntu:latest\\`" >&2 && exit 1 - udocker.py run --rm -v "$PWD":"$PWD" -w "$PWD" --bindhome $(udocker.py create "ubuntu:latest") /bin/bash -c "bwa --this --that" + udocker.py run --rm -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --bindhome $(udocker.py create "ubuntu:latest") /bin/bash -c "bwa --this --that" ''' .stripIndent().trim() diff --git a/modules/nextflow/src/test/groovy/nextflow/executor/BashWrapperBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/executor/BashWrapperBuilderTest.groovy index 4a8c012502..34617cdc78 100644 --- a/modules/nextflow/src/test/groovy/nextflow/executor/BashWrapperBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/executor/BashWrapperBuilderTest.groovy @@ -734,7 +734,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', temp: 'auto', enabled: true] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == 'docker rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'docker stop $NXF_BOXID' } @@ -748,7 +748,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', temp: 'auto', enabled: true] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID busybox /bin/bash -c "eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -c "eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' binding.cleanup_cmd == 'docker rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'docker stop $NXF_BOXID' and: @@ -769,7 +769,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', temp: 'auto', enabled: true, entrypointOverride: false] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == 'docker rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'docker stop $NXF_BOXID' } @@ -783,7 +783,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', temp: 'auto', enabled: true, entrypointOverride: false] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID busybox /bin/bash -c "eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -c "eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' binding.cleanup_cmd == 'docker rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'docker stop $NXF_BOXID' and: @@ -804,7 +804,7 @@ class BashWrapperBuilderTest extends Specification { containerEnabled: true ).makeBinding() then: - binding.launch_cmd == 'sudo docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'sudo docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == 'sudo docker rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'sudo docker stop $NXF_BOXID' } @@ -817,7 +817,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', temp: 'auto', enabled: true, remove:false, kill: false] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID ubuntu /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v $(nxf_mktemp):/tmp -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID ubuntu /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == "" binding.kill_cmd == null binding.containsKey('kill_cmd') @@ -831,7 +831,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', enabled: true, remove:false, kill: 'SIGXXX'] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID ubuntu /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID ubuntu /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == "" binding.kill_cmd == 'docker kill -s SIGXXX $NXF_BOXID' binding.containsKey('kill_cmd') @@ -846,7 +846,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', enabled: true] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v /work/dir:/work/dir -w "\$PWD" --name \$NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v /folder\\ with\\ blanks:/folder\\ with\\ blanks -v /work/dir:/work/dir -w "\$NXF_TASK_WORKDIR" --name \$NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == 'docker rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'docker stop $NXF_BOXID' } @@ -861,7 +861,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', sudo: true, enabled: true] ).makeBinding() then: - binding.launch_cmd == 'sudo docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -v "$PWD":"$PWD" -w "$PWD" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'sudo docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.kill_cmd == 'sudo docker stop $NXF_BOXID' binding.cleanup_cmd == '''\ (sudo -n true && sudo rm -rf "$NXF_SCRATCH" || rm -rf "$NXF_SCRATCH")&>/dev/null || true @@ -878,7 +878,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'docker', enabled: true] ).makeBinding() then: - binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$PWD" -v /foo:/bar --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'docker run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" -v /foo:/bar --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.kill_cmd == 'docker stop $NXF_BOXID' binding.cleanup_cmd == 'docker rm $NXF_BOXID &>/dev/null || true\n' } @@ -893,7 +893,7 @@ class BashWrapperBuilderTest extends Specification { then: binding.launch_cmd == '''\ sarus pull busybox 1>&2 - sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/work/dir,destination=/work/dir -w "$PWD" busybox /bin/bash -ue /work/dir/.command.sh + sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/work/dir,destination=/work/dir -w "$NXF_TASK_WORKDIR" busybox /bin/bash -ue /work/dir/.command.sh '''.stripIndent().rightTrim() binding.cleanup_cmd == "" binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' @@ -910,7 +910,7 @@ class BashWrapperBuilderTest extends Specification { then: binding.launch_cmd == '''\ sarus pull busybox 1>&2 - sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/work/dir,destination=/work/dir -w "$PWD" busybox /bin/bash -c "eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh" + sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/work/dir,destination=/work/dir -w "$NXF_TASK_WORKDIR" busybox /bin/bash -c "eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh" '''.stripIndent().rightTrim() binding.cleanup_cmd == "" binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' @@ -935,7 +935,7 @@ class BashWrapperBuilderTest extends Specification { then: binding.launch_cmd == '''\ sarus pull busybox 1>&2 - sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/folder\\ with\\ blanks,destination=/folder\\ with\\ blanks --mount=type=bind,source=/work/dir,destination=/work/dir -w "$PWD" busybox /bin/bash -ue /work/dir/.command.sh + sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/folder\\ with\\ blanks,destination=/folder\\ with\\ blanks --mount=type=bind,source=/work/dir,destination=/work/dir -w "$NXF_TASK_WORKDIR" busybox /bin/bash -ue /work/dir/.command.sh '''.stripIndent().rightTrim() binding.cleanup_cmd == "" binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' @@ -952,7 +952,7 @@ class BashWrapperBuilderTest extends Specification { then: binding.launch_cmd == '''\ sarus pull busybox 1>&2 - sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/work/dir,destination=/work/dir -w "$PWD" --mount=type=bind,source=/foo,destination=/bar busybox /bin/bash -ue /work/dir/.command.sh + sarus run -e "NXF_TASK_WORKDIR" --mount=type=bind,source=/work/dir,destination=/work/dir -w "$NXF_TASK_WORKDIR" --mount=type=bind,source=/foo,destination=/bar busybox /bin/bash -ue /work/dir/.command.sh '''.stripIndent().rightTrim() binding.cleanup_cmd == "" binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' @@ -991,7 +991,21 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [enabled: true, engine: 'singularity'] as ContainerConfig ).makeBinding() then: - binding.launch_cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${NXF_TASK_WORKDIR:+SINGULARITYENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} singularity exec --no-home --pid -B /work/dir docker://ubuntu:latest /bin/bash -c "cd $PWD; eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' + binding.launch_cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${NXF_TASK_WORKDIR:+SINGULARITYENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} singularity exec --no-home --pid -B /work/dir docker://ubuntu:latest /bin/bash -c "cd $NXF_TASK_WORKDIR; eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' + binding.cleanup_cmd == "" + binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' + } + + def 'should create wrapper with singularity and no env'() { + when: + def binding = newBashWrapperBuilder( + containerEnabled: true, + containerImage: 'docker://ubuntu:latest', + environment: [:], + containerConfig: [enabled: true, engine: 'singularity'] as ContainerConfig ).makeBinding() + + then: + binding.launch_cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${NXF_TASK_WORKDIR:+SINGULARITYENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} singularity exec --no-home --pid -B /work/dir docker://ubuntu:latest /bin/bash -c "cd $NXF_TASK_WORKDIR; /bin/bash -ue /work/dir/.command.sh"' binding.cleanup_cmd == "" binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' } @@ -1005,7 +1019,21 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [enabled: true, engine: 'singularity', entrypointOverride: true] as ContainerConfig ).makeBinding() then: - binding.launch_cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${NXF_TASK_WORKDIR:+SINGULARITYENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} singularity exec --no-home --pid -B /work/dir docker://ubuntu:latest /bin/bash -c "cd $PWD; eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' + binding.launch_cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${NXF_TASK_WORKDIR:+SINGULARITYENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} singularity exec --no-home --pid -B /work/dir docker://ubuntu:latest /bin/bash -c "cd $NXF_TASK_WORKDIR; eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' + binding.cleanup_cmd == "" + binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' + } + + def 'should create wrapper with singularity oci mode'() { + when: + def binding = newBashWrapperBuilder( + containerEnabled: true, + containerImage: 'docker://ubuntu:latest', + environment: [PATH: '/path/to/bin:$PATH', FOO: 'xxx'], + containerConfig: [enabled: true, engine: 'singularity', oci: true] as ContainerConfig ).makeBinding() + + then: + binding.launch_cmd == 'set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} ${XDG_RUNTIME_DIR:+XDG_RUNTIME_DIR="$XDG_RUNTIME_DIR"} ${DBUS_SESSION_BUS_ADDRESS:+DBUS_SESSION_BUS_ADDRESS="$DBUS_SESSION_BUS_ADDRESS"} ${NXF_TASK_WORKDIR:+SINGULARITYENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} singularity exec --no-home --oci -B /work/dir docker://ubuntu:latest /bin/bash -c "cd $NXF_TASK_WORKDIR; eval $(nxf_container_env); /bin/bash -ue /work/dir/.command.sh"' binding.cleanup_cmd == "" binding.kill_cmd == '[[ "$pid" ]] && nxf_kill $pid' } @@ -1145,7 +1173,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'podman', enabled: true] ).makeBinding() then: - binding.launch_cmd == 'podman run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$PWD" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'podman run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == 'podman rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'podman stop $NXF_BOXID' } @@ -1158,7 +1186,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'podman', enabled: true, entrypointOverride: true] ).makeBinding() then: - binding.launch_cmd == 'podman run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$PWD" --entrypoint /bin/bash --name $NXF_BOXID busybox -c "/bin/bash -ue /work/dir/.command.sh"' + binding.launch_cmd == 'podman run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -w "$NXF_TASK_WORKDIR" --entrypoint /bin/bash --name $NXF_BOXID busybox -c "/bin/bash -ue /work/dir/.command.sh"' binding.cleanup_cmd == 'podman rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'podman stop $NXF_BOXID' } @@ -1172,7 +1200,7 @@ class BashWrapperBuilderTest extends Specification { containerConfig: [engine: 'podman', enabled: true] ).makeBinding() then: - binding.launch_cmd == 'podman run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -v "$PWD":"$PWD" -w "$PWD" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' + binding.launch_cmd == 'podman run -i -e "NXF_TASK_WORKDIR" -v /work/dir:/work/dir -v "$NXF_TASK_WORKDIR":"$NXF_TASK_WORKDIR" -w "$NXF_TASK_WORKDIR" --name $NXF_BOXID busybox /bin/bash -ue /work/dir/.command.sh' binding.cleanup_cmd == 'rm -rf $NXF_SCRATCH || true\npodman rm $NXF_BOXID &>/dev/null || true\n' binding.kill_cmd == 'podman stop $NXF_BOXID' } diff --git a/modules/nextflow/src/test/groovy/nextflow/extension/ChannelExTest.groovy b/modules/nextflow/src/test/groovy/nextflow/extension/ChannelExTest.groovy deleted file mode 100644 index ca61e44ceb..0000000000 --- a/modules/nextflow/src/test/groovy/nextflow/extension/ChannelExTest.groovy +++ /dev/null @@ -1,41 +0,0 @@ -package nextflow.extension - -import spock.lang.Specification - -import nextflow.Channel -/** - * - * @author Paolo Di Tommaso - */ -class ChannelExTest extends Specification { - - - def 'should close the dataflow channel'() { - - when: - def source = Channel.create() - source << 10 - source << 20 - source << 30 - def result = source.close() - then: - result.is source - result.val == 10 - result.val == 20 - result.val == 30 - result.val == Channel.STOP - - when: - source = Channel.value().close() - then: - source.val == Channel.STOP - - when: - source = Channel.value(1).close() - then: - source.val == 1 - - } - - -} diff --git a/modules/nextflow/src/test/groovy/nextflow/extension/CombineOpTest.groovy b/modules/nextflow/src/test/groovy/nextflow/extension/CombineOpTest.groovy index 8d0e1b8daa..ae8bce0788 100644 --- a/modules/nextflow/src/test/groovy/nextflow/extension/CombineOpTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/extension/CombineOpTest.groovy @@ -149,7 +149,7 @@ class CombineOpTest extends Specification { def 'should combine with empty value' () { given: - def left = Channel.value().close() + def left = Channel.empty() def right = Channel.value('z') def op = new CombineOp(left, right) when: diff --git a/modules/nextflow/src/test/groovy/nextflow/extension/DataflowTapExtensionTest.groovy b/modules/nextflow/src/test/groovy/nextflow/extension/DataflowTapExtensionTest.groovy index ea3c65d241..de3c3d67e3 100644 --- a/modules/nextflow/src/test/groovy/nextflow/extension/DataflowTapExtensionTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/extension/DataflowTapExtensionTest.groovy @@ -79,55 +79,4 @@ class DataflowTapExtensionTest extends Specification { } - def 'should `tap` target channel' () { - - when: - def target = Channel.create() - def result = Channel.of( 8,2,5 ) .tap(target).map { it+1 } - then: - result instanceof DataflowQueue - target instanceof DataflowQueue - - target.val == 8 - target.val == 2 - target.val == 5 - target.val == Channel.STOP - - result.val == 9 - result.val == 3 - result.val == 6 - result.val == Channel.STOP - - !session.dag.isEmpty() - - } - - def 'should `tap` dataflow value' () { - - when: - def target = Channel.value() - def result = Channel.value(7) .tap(target).map { it+1 } - then: - result instanceof DataflowVariable - target instanceof DataflowVariable - - target.val == 7 - target.val == 7 - result.val == 8 - result.val == 8 - - !session.dag.isEmpty() - - } - - def 'should `tap` dataflow value and target as queue' () { - - when: - new DataflowVariable() .tap( new DataflowQueue() ) - then: - thrown(IllegalArgumentException) - session.dag.isEmpty() - - } - } diff --git a/modules/nextflow/src/test/groovy/nextflow/extension/OperatorImplTest.groovy b/modules/nextflow/src/test/groovy/nextflow/extension/OperatorImplTest.groovy index ebb92b1a2e..888b7aa765 100644 --- a/modules/nextflow/src/test/groovy/nextflow/extension/OperatorImplTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/extension/OperatorImplTest.groovy @@ -18,6 +18,7 @@ package nextflow.extension import java.nio.file.Paths +import groovyx.gpars.dataflow.DataflowQueue import groovyx.gpars.dataflow.DataflowVariable import nextflow.Channel import nextflow.Session @@ -225,7 +226,7 @@ class OperatorImplTest extends Specification { result.val == Channel.STOP when: - result = Channel.value().close().flatMap() + result = Channel.empty().flatMap() then: result.val == Channel.STOP @@ -443,7 +444,7 @@ class OperatorImplTest extends Specification { channel.toList().val == [1] when: - channel = Channel.value().close() + channel = Channel.empty() then: channel.toList().val == [] } @@ -472,7 +473,7 @@ class OperatorImplTest extends Specification { channel.toSortedList().val == [1] when: - channel = Channel.value().close() + channel = Channel.empty() then: channel.toSortedList().val == [] @@ -542,7 +543,7 @@ class OperatorImplTest extends Specification { result.val == Channel.STOP when: - result = Channel.value().close().flatten() + result = Channel.empty().flatten() then: result.val == Channel.STOP } @@ -1099,9 +1100,9 @@ class OperatorImplTest extends Specification { result.val == 1 when: - result = Channel.value().close().ifEmpty(100) + result = Channel.empty().ifEmpty(100) then: - result instanceof DataflowVariable + result instanceof DataflowQueue result.val == 100 } diff --git a/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodOptionsTest.groovy b/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodOptionsTest.groovy index 9070458eef..18f2592f78 100644 --- a/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodOptionsTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodOptionsTest.groovy @@ -553,4 +553,16 @@ class PodOptionsTest extends Specification { then: opts.getPrivileged() } + + def 'should set pod schedulerName' () { + when: + def opts = new PodOptions() + then: + opts.getSchedulerName() == null + + when: + opts = new PodOptions([ [schedulerName:'my-scheduler'] ]) + then: + opts.getSchedulerName() == 'my-scheduler' + } } diff --git a/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodSpecBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodSpecBuilderTest.groovy index 188e72dc81..dfb406c2be 100644 --- a/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodSpecBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/k8s/model/PodSpecBuilderTest.groovy @@ -581,6 +581,21 @@ class PodSpecBuilderTest extends Specification { } + def 'should create pod spec with schedulerName' () { + + when: + def pod = new PodSpecBuilder() + .withPodName('foo') + .withImageName('busybox') + .withCommand(['echo', 'hello']) + .withPodOptions(new PodOptions(schedulerName: 'my-scheduler')) + .build() + + then: + pod.spec.schedulerName == 'my-scheduler' + + } + def 'should create image pull request map' () { given: def builder = new PodSpecBuilder(imagePullSecret: 'MySecret') diff --git a/modules/nextflow/src/test/groovy/nextflow/processor/TaskProcessorTest.groovy b/modules/nextflow/src/test/groovy/nextflow/processor/TaskProcessorTest.groovy index 4307c7edb4..b616669c2e 100644 --- a/modules/nextflow/src/test/groovy/nextflow/processor/TaskProcessorTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/processor/TaskProcessorTest.groovy @@ -608,14 +608,14 @@ class TaskProcessorTest extends Specification { when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.type('file') + param.setType('file') result = fetchResultFiles(processor, param, '*.fa', folder) then: result == ['file2.fa'] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.type('dir') + param.setType('dir') result = fetchResultFiles(processor, param, '*.fa', folder) then: result == [] @@ -628,14 +628,14 @@ class TaskProcessorTest extends Specification { when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.followLinks(false) + param.setFollowLinks(false) result = fetchResultFiles(processor, param, '**.fa', folder) then: result == ['dir1/dir2/file4.fa', 'file2.fa'] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.maxDepth(1) + param.setMaxDepth(1) result = fetchResultFiles(processor, param, '**.fa', folder) then: result == ['file2.fa'] @@ -648,22 +648,22 @@ class TaskProcessorTest extends Specification { when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.type('dir') + param.setType('dir') result = fetchResultFiles(processor, param, '*', folder) then: result == ['dir1', 'dir_link'] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.type('file') + param.setType('file') result = fetchResultFiles(processor, param, '*', folder) then: result == ['file1.txt', 'file2.fa'] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.type('file') - param.hidden(true) + param.setType('file') + param.setHidden(true) result = fetchResultFiles(processor, param, '*', folder) then: result == ['.hidden.fa', 'file1.txt', 'file2.fa'] @@ -700,25 +700,25 @@ class TaskProcessorTest extends Specification { when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.type('dir') + param.setType('dir') then: processor.visitOptions(param,'dir-name') == [type:'dir', followLinks: true, maxDepth: null, hidden: false, relative: false] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.hidden(true) + param.setHidden(true) then: processor.visitOptions(param,'dir-name') == [type:'any', followLinks: true, maxDepth: null, hidden: true, relative: false] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.followLinks(false) + param.setFollowLinks(false) then: processor.visitOptions(param,'dir-name') == [type:'any', followLinks: false, maxDepth: null, hidden: false, relative: false] when: param = new FileOutParam(Mock(Binding), Mock(List)) - param.maxDepth(5) + param.setMaxDepth(5) then: processor.visitOptions(param,'dir-name') == [type:'any', followLinks: true, maxDepth: 5, hidden: false, relative: false] } diff --git a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy index 510ec22ce7..01b1b83164 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy @@ -168,6 +168,7 @@ class IncludeDefTest extends Specification { } } + @Unroll def 'should add includes' () { given: def binding = new ScriptBinding([params: [foo:1, bar:2]]) @@ -186,14 +187,12 @@ class IncludeDefTest extends Specification { where: - INCLUDE | PATH | NAME | ALIAS | PARAMS - "include 'some/path'" | 'some/path' | null | null | null - "include ALPHA from 'modules/path'" | 'modules/path'| 'ALPHA' | null | null - "include ALPHA as BRAVO from 'modules/x'" | 'modules/x' | 'ALPHA' | 'BRAVO' | null - "include 'modules/1' params(a:1, b:2)" | 'modules/1' | null | null | [a:1, b:2] - "include DELTA from 'abc' params(x:1)" | 'abc' | 'DELTA' | null | [x:1] - "include GAMMA as FOO from 'm1' params(p:2)" | 'm1' | 'GAMMA' | 'FOO' | [p:2] - "include GAMMA as FOO from 'm1' params([:])" | 'm1' | 'GAMMA' | 'FOO' | [:] + INCLUDE | PATH | NAME | ALIAS | PARAMS + "include { ALPHA } from 'modules/path'" | 'modules/path'| 'ALPHA' | null | null + "include { ALPHA as BRAVO } from 'modules/x'" | 'modules/x' | 'ALPHA' | 'BRAVO' | null + "include { DELTA } from 'abc' params(x:1)" | 'abc' | 'DELTA' | null | [x:1] + "include { GAMMA as FOO } from 'm1' params(p:2)" | 'm1' | 'GAMMA' | 'FOO' | [p:2] + "include { GAMMA as FOO } from 'm1' params([:])" | 'm1' | 'GAMMA' | 'FOO' | [:] } diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ProcessConfigTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ProcessConfigTest.groovy index 5fc7f31367..a9d8a108d8 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ProcessConfigTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ProcessConfigTest.groovy @@ -179,7 +179,7 @@ class ProcessConfigTest extends Specification { when: config._in_file([infile:'filename.fa']) - config._in_val('x') .from(1) + config._in_val('x').setFrom(1) config._in_stdin() then: @@ -208,9 +208,9 @@ class ProcessConfigTest extends Specification { when: config._out_stdout() - config._out_file(new TokenVar('file1')).into('ch1') - config._out_file(new TokenVar('file2')).into('ch2') - config._out_file(new TokenVar('file3')).into('ch3') + config._out_file(new TokenVar('file1')).setInto('ch1') + config._out_file(new TokenVar('file2')).setInto('ch2') + config._out_file(new TokenVar('file3')).setInto('ch3') then: config.outputs.size() == 4 diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy index d008e29491..6b8fc4805a 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy @@ -1,54 +1,13 @@ package nextflow.script -import spock.lang.Specification - import nextflow.Session -import nextflow.ast.NextflowDSL -import org.codehaus.groovy.control.CompilerConfiguration -import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer -import test.MockSession +import spock.lang.Specification /** * * @author Paolo Di Tommaso */ class ProcessDefTest extends Specification { - def 'should define processes' () { - - given: - def session = new MockSession() - def binding = new ScriptBinding(session) - def config = new CompilerConfiguration() - config.setScriptBaseClass(BaseScript.class.name) - config.addCompilationCustomizers( new ASTTransformationCustomizer(NextflowDSL)) - - def SCRIPT = ''' - - process foo { - input: val data - output: val result - exec: - result = "$data mundo" - } - - process bar { - input: val data - output: val result - exec: - result = data.toUpperCase() - } - - ''' - - when: - def script = (BaseScript)new GroovyShell(binding,config).parse(SCRIPT) - - then: - ScriptMeta.get(script).getProcessNames() == ['foo','bar'] as Set - - } - - def 'should clone a process with a new name '() { given: diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ScriptDslTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ScriptDslTest.groovy index 7f8a69ee5a..9e48c8e55e 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ScriptDslTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ScriptDslTest.groovy @@ -475,26 +475,6 @@ class ScriptDslTest extends Dsl2Spec { err.message == "Access to 'flow1.out' is undefined since the workflow 'flow1' has not been invoked before accessing the output attribute" } - def 'should report unsupported error' () { - when: - dsl_eval(''' - process foo { - /echo foo/ - } - - workflow { - get: - x - main: - flow() - } - ''') - - then: - def err = thrown(ScriptCompilationException) - err.message.contains "Workflow 'get' is not supported anymore use 'take' instead" - } - def 'should fail with wrong scope'() { when: dsl_eval('''\ diff --git a/modules/nextflow/src/test/groovy/nextflow/script/params/ParamsDsl2Test.groovy b/modules/nextflow/src/test/groovy/nextflow/script/params/ParamsDsl2Test.groovy index 2739e73c27..b703b22187 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/params/ParamsDsl2Test.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/params/ParamsDsl2Test.groovy @@ -35,8 +35,8 @@ class ParamsDsl2Test extends Dsl2Spec { when: new MockScriptRunner() .setScript(SCRIPT).execute() then: - def e = thrown(DeprecationException) - e.message == "Unqualified input file declaration has been deprecated - replace `tuple 'x',..` with `tuple path('x'),..`" + def e = thrown(IllegalArgumentException) + e.message == "Unqualified input file declaration is not allowed - replace `tuple 'x',..` with `tuple path('x'),..`" } def 'should not allow unqualified input val' () { @@ -57,8 +57,8 @@ class ParamsDsl2Test extends Dsl2Spec { when: new MockScriptRunner() .setScript(SCRIPT).execute() then: - def e = thrown(DeprecationException) - e.message == "Unqualified input value declaration has been deprecated - replace `tuple X,..` with `tuple val(X),..`" + def e = thrown(IllegalArgumentException) + e.message == "Unqualified input value declaration is not allowed - replace `tuple X,..` with `tuple val(X),..`" } @@ -80,8 +80,8 @@ class ParamsDsl2Test extends Dsl2Spec { when: new MockScriptRunner() .setScript(SCRIPT).execute() then: - def e = thrown(DeprecationException) - e.message == "Unqualified output path declaration has been deprecated - replace `tuple 'x',..` with `tuple path('x'),..`" + def e = thrown(IllegalArgumentException) + e.message == "Unqualified output path declaration is not allowed - replace `tuple 'x',..` with `tuple path('x'),..`" } def 'should not allow unqualified output value' () { @@ -102,8 +102,8 @@ class ParamsDsl2Test extends Dsl2Spec { when: new MockScriptRunner() .setScript(SCRIPT).execute() then: - def e = thrown(DeprecationException) - e.message == "Unqualified output value declaration has been deprecated - replace `tuple X,..` with `tuple val(X),..`" + def e = thrown(IllegalArgumentException) + e.message == "Unqualified output value declaration is not allowed - replace `tuple X,..` with `tuple val(X),..`" } diff --git a/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy b/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy index 4e773877af..98a5739273 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PluginUpdater.groovy @@ -224,7 +224,7 @@ class PluginUpdater extends UpdateManager { safeMove(dir, pluginPath) } catch (IOException e) { - throw new PluginRuntimeException(e, "Failed to write file '$pluginPath' to plugins folder"); + throw new PluginRuntimeException(e, "Failed to write file '$pluginPath' to plugins folder") } return pluginPath @@ -318,8 +318,14 @@ class PluginUpdater extends UpdateManager { if( version == null ) version = getLastPluginRelease(id)?.version - if( !version ) - throw new IllegalStateException("Cannot find latest version of $id plugin") + + final offline = SysEnv.get('NXF_OFFLINE')=='true' + if( !version ) { + final msg = offline + ? "Cannot find version for $id plugin -- plugin versions MUST be specified in offline mode" + : "Cannot find latest version of $id plugin" + throw new IllegalStateException(msg) + } def pluginPath = pluginsStore.resolve("$id-$version") if( !FilesEx.exists(pluginPath) ) { @@ -363,7 +369,7 @@ class PluginUpdater extends UpdateManager { // resolve the plugins pluginManager.resolvePlugins() // finally start it - PluginState state = pluginManager.startPlugin(id); + PluginState state = pluginManager.startPlugin(id) return PluginState.STARTED == state } @@ -381,13 +387,13 @@ class PluginUpdater extends UpdateManager { throw new PluginRuntimeException("Plugin $id cannot be updated since it is not installed") } - PluginInfo pluginInfo = getPluginsMap().get(id); + PluginInfo pluginInfo = getPluginsMap().get(id) if (pluginInfo == null) { throw new PluginRuntimeException("Plugin $id does not exist in any repository") } if (!pluginManager.deletePlugin(id)) { - return false; + return false } load0(id, version) diff --git a/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy b/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy index 5af5b2d2a6..b4320d435e 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy @@ -55,7 +55,7 @@ class PluginsFacade implements PluginStateListener { PluginsFacade() { mode = getPluginsMode() root = getPluginsDir() - if( mode=='dev' && root.toString()=='plugins' && !isRunningFromDistArchive() ) + if( mode==DEV_MODE && root.toString()=='plugins' && !isRunningFromDistArchive() ) root = detectPluginsDevRoot() System.setProperty('pf4j.mode', mode) } diff --git a/nextflow b/nextflow index 2a73246ddf..fdbdedccd3 100755 --- a/nextflow +++ b/nextflow @@ -320,7 +320,7 @@ else if [[ ! $JAVA_VER =~ ^(11|12|13|14|15|16|17|18|19|20|21) ]]; then echo_yellow "NOTE: Nextflow is not tested with Java $JAVA_VER -- It's recommended the use of version 11 up to 21\n" fi - mkdir -p $(dirname "$JAVA_KEY") + mkdir -p "$(dirname "$JAVA_KEY")" [[ -f $JAVA_VER ]] && echo $JAVA_VER > "$JAVA_KEY" fi @@ -404,8 +404,8 @@ else # we extract first part into `cmd_base`` and remainder into `cmd_tail`` and convert them to array as previous version cmd_pattern='"([^"]*)"(.*)' [[ "${cli[@]}" =~ $cmd_pattern ]] - cmd_base=(${BASH_REMATCH[1]}) - cmd_tail=(${BASH_REMATCH[2]}) + declare -a cmd_base="(${BASH_REMATCH[1]})" + declare -a cmd_tail="(${BASH_REMATCH[2]})" launcher="${cmd_base[@]}" [[ "$NXF_JVM_ARGS" ]] && launcher+=($NXF_JVM_ARGS) @@ -443,7 +443,7 @@ else if mkdir -p "${NXF_LAUNCHER}" 2>/dev/null; then STR='' for x in "${launcher[@]}"; do - [[ "$x" != "\"-Duser.dir=$PWD\"" ]] && [[ ! "$x" == *"-agentlib:jdwp"* ]] && STR+="$x " + [[ "$x" != "\"-Duser.dir=$PWD\"" ]] && [[ ! "$x" == *"-agentlib:jdwp"* ]] && STR+=$(printf '%q ' "$x") done printf "$STR">"$LAUNCH_FILE" else diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy b/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy index 1c7da0dd66..d1ae070bda 100644 --- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy +++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy @@ -20,6 +20,7 @@ package nextflow.cloud.aws.config import java.nio.file.Path import java.nio.file.Paths +import com.amazonaws.regions.Regions import groovy.transform.CompileStatic import groovy.util.logging.Slf4j import nextflow.Global @@ -80,6 +81,12 @@ class AwsConfig { return s3Legacy.getAwsClientConfig() } + String getS3GlobalRegion() { + return !region || !s3Config.endpoint || s3Config.endpoint.contains(".amazonaws.com") + ? Regions.US_EAST_1.getName() // always use US_EAST_1 as global region for AWS endpoints + : region // for custom endpoint use the config provided region + } + static protected String getAwsProfile0(Map env, Map config) { final profile = config?.profile as String diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsS3Config.groovy b/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsS3Config.groovy index 16e5a85ee3..df6629e889 100644 --- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsS3Config.groovy +++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsS3Config.groovy @@ -114,4 +114,8 @@ class AwsS3Config { Boolean getAnonymous() { return anonymous } + + boolean isCustomEndpoint() { + endpoint && !endpoint.contains(".amazonaws.com") + } } diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/fusion/AwsFusionEnv.groovy b/plugins/nf-amazon/src/main/nextflow/cloud/aws/fusion/AwsFusionEnv.groovy index c999b29e94..4c2bcb82d4 100644 --- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/fusion/AwsFusionEnv.groovy +++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/fusion/AwsFusionEnv.groovy @@ -46,6 +46,8 @@ class AwsFusionEnv implements FusionEnv { } if( endpoint ) result.AWS_S3_ENDPOINT = endpoint + if( awsConfig.region && awsConfig.s3Config.isCustomEndpoint() ) + result.FUSION_AWS_REGION = awsConfig.region if( awsConfig.s3Config.storageEncryption ) result.FUSION_AWS_SERVER_SIDE_ENCRYPTION = awsConfig.s3Config.storageEncryption if( awsConfig.s3Config.storageKmsKeyId ) diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java b/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java index 76bd7d3c87..b376ba7532 100644 --- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java +++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java @@ -836,7 +836,7 @@ protected S3FileSystem createFileSystem(URI uri, AwsConfig awsConfig) { final String bucketName = S3Path.bucketName(uri); final boolean global = bucketName!=null; - final AwsClientFactory factory = new AwsClientFactory(awsConfig, Regions.US_EAST_1.getName()); + final AwsClientFactory factory = new AwsClientFactory(awsConfig, globalRegion(awsConfig)); client = new S3Client(factory.getS3Client(clientConfig, global)); // set the client acl @@ -852,6 +852,12 @@ protected S3FileSystem createFileSystem(URI uri, AwsConfig awsConfig) { return new S3FileSystem(this, client, uri, props); } + protected String globalRegion(AwsConfig awsConfig) { + return awsConfig.getRegion() != null && awsConfig.getS3Config().isCustomEndpoint() + ? awsConfig.getRegion() + : Regions.US_EAST_1.getName(); + } + protected String getProp(Properties props, String... keys) { for( String k : keys ) { if( props.containsKey(k) ) { diff --git a/plugins/nf-amazon/src/test/nextflow/cloud/aws/config/AwsS3ConfigTest.groovy b/plugins/nf-amazon/src/test/nextflow/cloud/aws/config/AwsS3ConfigTest.groovy index 907e211c14..bd786bd849 100644 --- a/plugins/nf-amazon/src/test/nextflow/cloud/aws/config/AwsS3ConfigTest.groovy +++ b/plugins/nf-amazon/src/test/nextflow/cloud/aws/config/AwsS3ConfigTest.groovy @@ -39,6 +39,7 @@ class AwsS3ConfigTest extends Specification { !client.s3Acl !client.pathStyleAccess !client.anonymous + !client.isCustomEndpoint() } def 'should set config' () { @@ -139,4 +140,19 @@ class AwsS3ConfigTest extends Specification { SysEnv.pop() } + + @Unroll + def 'should check is custom endpoint' () { + given: + def config = new AwsS3Config(CONFIG) + + expect: + config.isCustomEndpoint() == EXPECTED + + where: + EXPECTED | CONFIG + false | [:] + false | [endpoint: 'https://s3.us-east-2.amazonaws.com'] + true | [endpoint: 'https://foo.com'] + } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPlugins.groovy b/plugins/nf-amazon/src/test/nextflow/cloud/aws/nio/S3FileSystemProviderTest.groovy similarity index 50% rename from modules/nextflow/src/main/groovy/nextflow/cli/CmdPlugins.groovy rename to plugins/nf-amazon/src/test/nextflow/cloud/aws/nio/S3FileSystemProviderTest.groovy index 360d29e2d3..032350fe05 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPlugins.groovy +++ b/plugins/nf-amazon/src/test/nextflow/cloud/aws/nio/S3FileSystemProviderTest.groovy @@ -15,29 +15,33 @@ * */ -package nextflow.cli +package nextflow.cloud.aws.nio -import groovy.transform.CompileStatic -import groovy.util.logging.Slf4j +import nextflow.cloud.aws.config.AwsConfig +import spock.lang.Specification +import spock.lang.Unroll /** - * Deprecate - see {@link CmdPlugin} instead * * @author Paolo Di Tommaso */ -@Slf4j -@Deprecated -@CompileStatic -class CmdPlugins extends CmdPlugin { - - @Override - String getName() { - return 'plugins' - } +class S3FileSystemProviderTest extends Specification { + + @Unroll + def 'should get global region' () { + given: + def provider = Spy(S3FileSystemProvider) + + expect: + provider.globalRegion(new AwsConfig(CONFIG)) == EXPECTED + + where: + EXPECTED | CONFIG + 'us-east-1' | [:] + 'us-east-1' | [region:'foo'] + 'us-east-1' | [region:'foo', client:[endpoint: 'http://s3.us-east-2.amazonaws.com']] + 'foo' | [region:'foo', client:[endpoint: 'http://bar.com']] - @Override - void run() { - log.info "Command 'plugins' has been deprecated - Use 'plugin' instead" - super.run() } + } diff --git a/plugins/nf-google/build.gradle b/plugins/nf-google/build.gradle index 644aa9502e..b1c9178d6f 100644 --- a/plugins/nf-google/build.gradle +++ b/plugins/nf-google/build.gradle @@ -39,7 +39,7 @@ dependencies { api 'com.google.apis:google-api-services-lifesciences:v2beta-rev20210527-1.31.5' api 'com.google.auth:google-auth-library-oauth2-http:0.18.0' - api 'com.google.cloud:google-cloud-batch:0.10.0' + api 'com.google.cloud:google-cloud-batch:0.29.0' api 'com.google.cloud:google-cloud-logging:3.8.0' api 'com.google.cloud:google-cloud-nio:0.124.8' api 'com.google.cloud:google-cloud-storage:2.9.3' diff --git a/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy b/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy index 5fe5c45548..7bd4e1367f 100644 --- a/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy +++ b/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy @@ -378,7 +378,9 @@ class GoogleBatchTaskHandler extends TaskHandler implements FusionAwareTask { if( state in TERMINATED ) { log.debug "[GOOGLE BATCH] Process `${task.lazyName()}` - terminated job=$jobId; state=$state" // finalize the task - task.exitStatus = readExitFile() + task.exitStatus = getJobExitCode() + if( task.exitStatus == null ) + task.exitStatus = readExitFile() if( state == 'FAILED' ) { task.stdout = executor.logging.stdout(uid) ?: outputFile task.stderr = executor.logging.stderr(uid) ?: errorFile @@ -394,6 +396,24 @@ class GoogleBatchTaskHandler extends TaskHandler implements FusionAwareTask { return false } + protected Integer getJobExitCode() { + try { + final status = client.getJobStatus(jobId) + final eventsCount = status.getStatusEventsCount() + final lastEvent = eventsCount > 0 ? status.getStatusEvents(eventsCount - 1) : null + log.debug "[GOOGLE BATCH] Process `${task.lazyName()}` - last event: ${lastEvent}" + + if( lastEvent?.getDescription()?.contains('due to Spot VM preemption with exit code 50001') ) { + return 50001 + } + } + catch (Throwable t) { + log.debug "[GOOGLE BATCH] Unable to fetch task `${task.lazyName()}` exit code - cause: ${t.message}" + } + + return null + } + @PackageScope Integer readExitFile() { try { exitFile.text as Integer diff --git a/plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy b/plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy index 846969f971..4d0d36c6f1 100644 --- a/plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy +++ b/plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy @@ -18,6 +18,9 @@ package nextflow.cloud.google.batch import com.google.cloud.batch.v1.GCS +import com.google.cloud.batch.v1.JobStatus +import com.google.cloud.batch.v1.StatusEvent +import com.google.cloud.batch.v1.TaskExecution import com.google.cloud.batch.v1.Volume import com.google.cloud.storage.contrib.nio.CloudStorageFileSystem import nextflow.cloud.google.batch.client.BatchClient @@ -330,4 +333,32 @@ class GoogleBatchTaskHandlerTest extends Specification { req.getAllocationPolicy().getInstances(0).policy.getMachineType() == "" } + + JobStatus makeJobStatus(String desc) { + JobStatus.newBuilder() + .addStatusEvents( + StatusEvent.newBuilder() + .setDescription(desc) + ) + .build() + } + + def 'should detect spot failures from status event'() { + given: + def jobId = 'job-id' + def client = Mock(BatchClient) + def task = Mock(TaskRun) { + lazyName() >> 'foo (1)' + } + def handler = Spy(new GoogleBatchTaskHandler(jobId: jobId, client: client, task: task)) + + when: + client.getJobStatus(jobId) >>> [ + makeJobStatus('Task failed due to Spot VM preemption with exit code 50001.'), + makeJobStatus('Task succeeded') + ] + then: + handler.getJobExitCode() == 50001 + handler.getJobExitCode() == null + } } diff --git a/plugins/nf-wave/build.gradle b/plugins/nf-wave/build.gradle index f4deb73c9c..93d749fba3 100644 --- a/plugins/nf-wave/build.gradle +++ b/plugins/nf-wave/build.gradle @@ -36,7 +36,7 @@ dependencies { api 'org.apache.commons:commons-lang3:3.12.0' api 'com.google.code.gson:gson:2.10.1' api 'org.yaml:snakeyaml:2.0' - api 'io.seqera:wave-utils:0.7.9' + api 'io.seqera:wave-utils:0.8.0' testImplementation(testFixtures(project(":nextflow"))) testImplementation "org.codehaus.groovy:groovy:3.0.19" diff --git a/plugins/nf-wave/src/main/io/seqera/wave/plugin/cli/WaveRunCmd.groovy b/plugins/nf-wave/src/main/io/seqera/wave/plugin/cli/WaveRunCmd.groovy index c9aaafc287..5f076694b3 100644 --- a/plugins/nf-wave/src/main/io/seqera/wave/plugin/cli/WaveRunCmd.groovy +++ b/plugins/nf-wave/src/main/io/seqera/wave/plugin/cli/WaveRunCmd.groovy @@ -87,7 +87,7 @@ class WaveRunCmd { final containerCmd = containerBuilder .build() .getRunCommand(args.join(' ')) - .replaceAll('-w "\\$PWD" ','') // <-- hack to remove the PWD work dir + .replaceAll('-w "\\$NXF_TASK_WORKDIR" ','') // <-- hack to remove the PWD work dir log.debug "Running: $containerCmd" final process = new ProcessBuilder() diff --git a/plugins/nf-wave/src/main/io/seqera/wave/plugin/resolver/WaveContainerResolver.groovy b/plugins/nf-wave/src/main/io/seqera/wave/plugin/resolver/WaveContainerResolver.groovy index a79d169b2f..28c64973f3 100644 --- a/plugins/nf-wave/src/main/io/seqera/wave/plugin/resolver/WaveContainerResolver.groovy +++ b/plugins/nf-wave/src/main/io/seqera/wave/plugin/resolver/WaveContainerResolver.groovy @@ -25,6 +25,7 @@ import groovy.util.logging.Slf4j import io.seqera.wave.plugin.WaveClient import nextflow.Global import nextflow.Session +import nextflow.container.ContainerConfig import nextflow.container.resolver.ContainerInfo import nextflow.container.resolver.ContainerResolver import nextflow.container.resolver.DefaultContainerResolver @@ -52,8 +53,7 @@ class WaveContainerResolver implements ContainerResolver { return client0 = new WaveClient( Global.session as Session ) } - private String getContainerEngine0(TaskRun task) { - final config = task.getContainerConfig() + private String getContainerEngine0(ContainerConfig config) { final result = config.getEngine() if( result ) return result @@ -68,13 +68,15 @@ class WaveContainerResolver implements ContainerResolver { return defaultResolver.resolveImage(task, imageName) final freeze = client().config().freezeMode() - final engine = getContainerEngine0(task) - final nativeSingularityBuild = freeze && engine in SINGULARITY_LIKE + final config = task.getContainerConfig() + final engine = getContainerEngine0(config) + final singularityOciMode = config.singularityOciMode() + final singularitySpec = freeze && engine in SINGULARITY_LIKE && !singularityOciMode if( !imageName ) { // when no image name is provided the module bundle should include a // Dockerfile or a Conda recipe or a Spack recipe to build // an image on-fly with an automatically assigned name - return waveContainer(task, null, nativeSingularityBuild) + return waveContainer(task, null, singularitySpec) } if( engine in DOCKER_LIKE ) { @@ -90,7 +92,7 @@ class WaveContainerResolver implements ContainerResolver { return defaultResolver.resolveImage(task, imageName) } // fetch the wave container name - final image = waveContainer(task, imageName, nativeSingularityBuild) + final image = waveContainer(task, imageName, singularitySpec) // oras prefixed container are served directly if( image && image.target.startsWith("oras://") ) return image diff --git a/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy b/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy index c0259af468..6842819bb2 100644 --- a/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy +++ b/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy @@ -534,6 +534,7 @@ class WaveClientTest extends Specification { && micromamba install -y -n base conda-forge::procps-ng \\ && micromamba clean -a -y USER root + ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH" '''.stripIndent() and: !assets.moduleResources @@ -572,6 +573,7 @@ class WaveClientTest extends Specification { && micromamba install -y -n base conda-forge::procps-ng \\ && micromamba clean -a -y USER root + ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH" '''.stripIndent() and: !assets.moduleResources @@ -647,6 +649,7 @@ class WaveClientTest extends Specification { && micromamba install -y -n base conda-forge::procps-ng \\ && micromamba clean -a -y USER root + ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH" '''.stripIndent() and: assets.condaFile == condaFile