From 976e002078e0ce9ec62cb79370b864bca284da38 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 19 Feb 2025 12:40:31 -0500 Subject: [PATCH 01/11] Improve params documentation (#5800) Signed-off-by: Ben Sherman Co-authored-by: Chris Hakkaart --- docs/cli.md | 45 ++++++++++++++++++++++++++++++-------- docs/config.md | 25 +++++++++------------ docs/reference/env-vars.md | 2 ++ 3 files changed, 48 insertions(+), 24 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index a6d0b24544..ef459673d8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -223,16 +223,38 @@ These commands will execute two different project revisions based on the given G ### Pipeline parameters -Pipeline scripts can use an arbitrary number of parameters that can be overridden using the command line or Nextflow configuration files. Any script parameter can be specified on the command line by prefixing the parameter name with double-dash characters. For example: +Pipeline scripts can define *parameters* that can be overridden on the command line. + +Parameters can be declared in the main script: + +```nextflow +params.alpha = 'default script value' + +workflow { + println "alpha = ${params.alpha}" +} +``` + +Or in a config file: + +```groovy +params { + alpha = 'default config value' +} +``` + +The above parameter can be specified on the command line as `--alpha`: ```console -$ nextflow run --foo Hello +$ nextflow run main.nf --alpha Hello ``` -Then, the parameter can be accessed in the pipeline script using the `params.foo` identifier. +:::{note} +Parameters that are specified on the command line without a value are set to `true`. +::: :::{note} -When the parameter name is formatted using `camelCase`, a second parameter is created with the same value using `kebab-case`, and vice versa. +Parameters that are specified on the command line in kebab case (e.g., `--foo-bar`) are automatically converted to camel case (e.g., `--fooBar`). Because of this, a parameter defined as `fooBar` in the pipeline script can be specified on the command line as `--fooBar` or `--foo-bar`. ::: :::{warning} @@ -245,14 +267,14 @@ $ nextflow run --files "*.fasta" Parameters specified on the command line can be also specified in a params file using the `-params-file` option. -```bash -nextflow run main.nf -params-file pipeline_params.yml +```console +$ nextflow run main.nf -params-file pipeline_params.yml ``` The `-params-file` option loads parameters for your Nextflow pipeline from a JSON or YAML file. Parameters defined in the file are equivalent to specifying them directly on the command line. For example, instead of specifying parameters on the command line: -```bash -nextflow run main.nf --alpha 1 --beta foo +```console +$ nextflow run main.nf --alpha 1 --beta foo ``` Parameters can be represented in YAML format: @@ -271,7 +293,12 @@ Or in JSON format: } ``` -The parameters specified in a params file are merged with the resolved configuration. The values provided via a params file overwrite those of the same name in the Nextflow configuration file, but not those specified on the command line. +Parameters are applied in the following order (from lowest to highest priority): + +1. Parameters defined in pipeline scripts (e.g. `main.nf`) +2. Parameters defined in {ref}`config files ` +6. Parameters specified in a params file (`-params-file`) +7. Parameters specified on the command line (`--something value`) ## Managing projects diff --git a/docs/config.md b/docs/config.md index 4076e96f80..0818f2e3c1 100644 --- a/docs/config.md +++ b/docs/config.md @@ -6,18 +6,13 @@ When a pipeline script is launched, Nextflow looks for configuration files in multiple locations. Since each configuration file may contain conflicting settings, they are applied in the following order (from lowest to highest priority): -1. Parameters defined in pipeline scripts (e.g. `main.nf`) -2. The config file `$HOME/.nextflow/config`, or `$NXF_HOME/.nextflow/config` when `NXF_HOME` is set (see [`NXF` prefixed variables](reference/env-vars.html#nextflow-settings)). -3. The config file `nextflow.config` in the project directory -4. The config file `nextflow.config` in the launch directory -5. Config file specified using the `-c ` option -6. Parameters specified in a params file (`-params-file` option) -7. Parameters specified on the command line (`--something value`) - -When more than one of these options for specifying configurations are used, they are merged, so that the settings in the first override the same settings appearing in the second, and so on. +1. The config file `$HOME/.nextflow/config` (or `$NXF_HOME/.nextflow/config` when {ref}`NXF_HOME ` is set). +2. The config file `nextflow.config` in the project directory +3. The config file `nextflow.config` in the launch directory +4. Config files specified using the `-c ` option :::{tip} -You can use the `-C ` option to use a single configuration file and ignore all other files. +You can alternatively use the `-C ` option to specify a fixed set of configuration files and ignore all other files. ::: (config-syntax)= @@ -129,16 +124,16 @@ The following functions are globally available in a Nextflow configuration file: Pipeline parameters can be defined in the config file using the `params` scope: ```groovy -params.custom_param = 123 -params.another_param = 'string value .. ' +params.alpha = 123 +params.beta = 'string value .. ' params { - alpha_1 = true - beta_2 = 'another string ..' + gamma = true + delta = "params.alpha is ${params.alpha}" } ``` -See {ref}`cli-params` for information about how to modify these on the command line. +See {ref}`cli-params` for information about how to specify pipeline parameters. (config-process)= diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index e9a6f65f6f..8bb17797bd 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -12,6 +12,8 @@ The following environment variables control the configuration of the Nextflow ru `JAVA_HOME` : Defines the path location of the Java VM installation used to run Nextflow. +(nxf-env-vars)= + ## Nextflow settings `NXF_ANSI_LOG` From f9c0cbfd7b1854d50271ccd4213b605f3f31ff1f Mon Sep 17 00:00:00 2001 From: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Date: Wed, 19 Feb 2025 21:15:17 +0000 Subject: [PATCH 02/11] Add cpu-shares and memory limits to Azure Batch tasks (#5799) The Azure Batch task did not include the --cpu-shares or --memory options of the docker run command, meaning resource allocation was only controlled by slots on the node. This PR introduces the additional options to the container run command, including --cpu-shares and --memory. The behaviour will now be more similar to AWS Batch which includes a soft limit on CPUs and hard limit on memory. It also refactors the container options to use a StringBuilder class instead of string concatenation. Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso Co-authored-by: Paolo Di Tommaso --- .../cloud/azure/batch/AzBatchService.groovy | 32 ++++++++----- .../azure/batch/AzBatchServiceTest.groovy | 46 +++++++++++++++++++ 2 files changed, 67 insertions(+), 11 deletions(-) diff --git a/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy b/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy index c09e43c800..7ab4ba539d 100644 --- a/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy +++ b/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy @@ -459,17 +459,28 @@ class AzBatchService implements Closeable { final pool = getPoolSpec(poolId) if( !pool ) throw new IllegalStateException("Missing Azure Batch pool spec with id: $poolId") + // container settings - // mount host certificates otherwise `azcopy` fails - def opts = "-v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/pki:/etc/pki:ro " - // shared volume mounts + String opts = "" + // Add CPU and memory constraints if specified + if( task.config.getCpus() ) + opts += "--cpu-shares ${task.config.getCpus() * 1024} " + if( task.config.getMemory() ) + opts += "--memory ${task.config.getMemory().toMega()}m " + + // Mount host certificates for azcopy + opts += "-v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/pki:/etc/pki:ro " + + // Add any shared volume mounts final shares = getShareVolumeMounts(pool) if( shares ) - opts += "${shares.join(' ')} " - // custom container settings + opts += shares.join(' ') + ' ' + + // Add custom container options if( task.config.getContainerOptions() ) - opts += "${task.config.getContainerOptions()} " - // fusion environment settings + opts += task.config.getContainerOptions() + ' ' + + // Handle Fusion settings final fusionEnabled = FusionHelper.isFusionEnabled((Session)Global.session) final launcher = fusionEnabled ? FusionScriptLauncher.create(task.toTaskBean(), 'az') : null if( fusionEnabled ) { @@ -478,9 +489,11 @@ class AzBatchService implements Closeable { opts += "-e $it.key=$it.value " } } - // config overall container settings + + // Create container settings final containerOpts = new BatchTaskContainerSettings(container) .setContainerRunOptions(opts) + // submit command line final String cmd = fusionEnabled ? launcher.fusionSubmitCli(task).join(' ') @@ -493,7 +506,6 @@ class AzBatchService implements Closeable { constraints.setMaxWallClockTime( Duration.of(task.config.getTime().toMillis(), ChronoUnit.MILLIS) ) log.trace "[AZURE BATCH] Submitting task: $taskId, cpus=${task.config.getCpus()}, mem=${task.config.getMemory()?:'-'}, slots: $slots" - return new BatchTaskCreateContent(taskId, cmd) .setUserIdentity(userIdentity(pool.opts.privileged, pool.opts.runAs, AutoUserScope.TASK)) .setContainerSettings(containerOpts) @@ -501,8 +513,6 @@ class AzBatchService implements Closeable { .setOutputFiles(outputFileUrls(task, sas)) .setRequiredSlots(slots) .setConstraints(constraints) - - } AzTaskKey runTask(String poolId, String jobId, TaskRun task) { diff --git a/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy b/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy index de139b81b3..a691ec46f1 100644 --- a/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy +++ b/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy @@ -633,6 +633,52 @@ class AzBatchServiceTest extends Specification { result.containerSettings.containerRunOptions == '-v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/pki:/etc/pki:ro ' } + def 'should create task for submit with cpu and memory' () { + given: + def POOL_ID = 'my-pool' + def SAS = '123' + + def CONFIG = [storage: [sasToken: SAS]] + def exec = Mock(AzBatchExecutor) {getConfig() >> new AzConfig(CONFIG) } + AzBatchService azure = Spy(new AzBatchService(exec)) + def session = Mock(Session) { + getConfig() >>[fusion:[enabled:false]] + statsEnabled >> true + } + Global.session = session + and: + def TASK = Mock(TaskRun) { + getHash() >> HashCode.fromInt(2) + getContainer() >> 'ubuntu:latest' + getConfig() >> Mock(TaskConfig) { + getTime() >> Duration.of('24 h') + getCpus() >> 4 + getMemory() >> MemoryUnit.of('8 GB') + } + + } + and: + def SPEC = new AzVmPoolSpec(poolId: POOL_ID, vmType: Mock(AzVmType), opts: new AzPoolOpts([:])) + + when: + def result = azure.createTask(POOL_ID, 'salmon', TASK) + then: + 1 * azure.getPoolSpec(POOL_ID) >> SPEC + 1 * azure.computeSlots(TASK, SPEC) >> 4 + 1 * azure.resourceFileUrls(TASK, SAS) >> [] + 1 * azure.outputFileUrls(TASK, SAS) >> [] + and: + result.id == 'nf-02000000' + result.requiredSlots == 4 + and: + result.commandLine == "sh -c 'bash .command.run 2>&1 | tee .command.log'" + and: + result.containerSettings.imageName == 'ubuntu:latest' + result.containerSettings.containerRunOptions == '--cpu-shares 4096 --memory 8192m -v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/pki:/etc/pki:ro ' + and: + Duration.of(result.constraints.maxWallClockTime.toMillis()) == TASK.config.time + } + def 'should create task for submit with extra options' () { given: def POOL_ID = 'my-pool' From c3f67db2619f302589c5379a7851174bc7f76a54 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 20 Feb 2025 03:07:16 -0500 Subject: [PATCH 03/11] Fix false error in variable visitor (#5765) Signed-off-by: Ben Sherman --- .../nextflow/ast/VariableVisitor.groovy | 6 +++-- .../nextflow/ast/NextflowDSLImplTest.groovy | 27 +++++++++++++++++++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/ast/VariableVisitor.groovy b/modules/nextflow/src/main/groovy/nextflow/ast/VariableVisitor.groovy index 71c453054b..848a04583d 100644 --- a/modules/nextflow/src/main/groovy/nextflow/ast/VariableVisitor.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/ast/VariableVisitor.groovy @@ -72,11 +72,13 @@ class VariableVisitor extends ClassCodeVisitorSupport { void visitDeclarationExpression(DeclarationExpression expr) { declaration = true try { - super.visitDeclarationExpression(expr) + visit(expr.getLeftExpression()) } finally { declaration = false } + + visit(expr.getRightExpression()) } @Override @@ -107,7 +109,7 @@ class VariableVisitor extends ClassCodeVisitorSupport { if( declaration ) { if( fAllVariables.containsKey(name) ) - sourceUnit.addError( new SyntaxException("Variable `$name` already defined in the process scope", line, coln)) + sourceUnit.addError( new SyntaxException("Variable `$name` already declared in the process scope", line, coln)) else localDef.add(name) } diff --git a/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy b/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy index a8c89e4266..c439d4728b 100644 --- a/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/ast/NextflowDSLImplTest.groovy @@ -151,4 +151,31 @@ class NextflowDSLImplTest extends Dsl2Spec { ScriptMeta.get(parser.getScript()).getProcessNames() == ['alpha', 'beta'] as Set } + def 'should fetch variable names' () { + + given: + def config = createCompilerConfig() + + def SCRIPT = ''' + process alpha { + input: + val foo + + exec: + if( foo == 'a' ) + log.info "foo is 'a'" + def bar = foo + } + + workflow { + alpha('a') + } + ''' + + when: + new GroovyShell(config).parse(SCRIPT) + then: + noExceptionThrown() + } + } From e0ba536df27ec8a443590fc70b3d567bcb7f6e53 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 21 Feb 2025 13:40:49 +0100 Subject: [PATCH 04/11] Bump Ubuntu 22.04 as default SKU for Azure Batch (#5804) Signed-off-by: Paolo Di Tommaso Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> --- docs/azure.md | 10 +++++----- .../main/nextflow/cloud/azure/config/AzPoolOpts.groovy | 8 ++++---- .../cloud/azure/batch/AzBatchServiceTest.groovy | 2 +- .../nextflow/cloud/azure/config/AzureConfigTest.groovy | 2 +- validation/azure.config | 1 - 5 files changed, 11 insertions(+), 12 deletions(-) diff --git a/docs/azure.md b/docs/azure.md index fd7906f9b3..9cb93d957a 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -342,14 +342,14 @@ When Nextflow creates a pool of compute nodes, it selects: Together, these settings determine the Operating System and version installed on each node. -By default, Nextflow creates pool nodes based on Ubuntu 20.04, but this behavior can be customised in the pool configuration. Below are configurations for image reference/SKU combinations to select two popular systems. +By default, Nextflow creates pool nodes based on Ubuntu 22.04, but this behavior can be customised in the pool configuration. Below are configurations for image reference/SKU combinations to select two popular systems. -- Ubuntu 20.04 (default): +- Ubuntu 22.04 (default): ```groovy - azure.batch.pools..sku = "batch.node.ubuntu 20.04" - azure.batch.pools..offer = "ubuntu-server-container" - azure.batch.pools..publisher = "microsoft-azure-batch" + azure.batch.pools..sku = "batch.node.ubuntu 22.04" + azure.batch.pools..offer = "ubuntu-hpc" + azure.batch.pools..publisher = "microsoft-dsvm" ``` - CentOS 8: diff --git a/plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOpts.groovy b/plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOpts.groovy index 01a3570c4b..2f424e5b5d 100644 --- a/plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOpts.groovy +++ b/plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOpts.groovy @@ -35,10 +35,10 @@ import nextflow.util.Duration @CompileStatic class AzPoolOpts implements CacheFunnel { - static public final String DEFAULT_PUBLISHER = "microsoft-azure-batch" - static public final String DEFAULT_OFFER = "ubuntu-server-container" - static public final String DEFAULT_SKU = "batch.node.ubuntu 20.04" - static public final String DEFAULT_VM_TYPE = "Standard_D4_v3" + static public final String DEFAULT_PUBLISHER = "microsoft-dsvm" + static public final String DEFAULT_OFFER = "ubuntu-hpc" + static public final String DEFAULT_SKU = "batch.node.ubuntu 22.04" + static public final String DEFAULT_VM_TYPE = "Standard_D4a_v4" static public final OSType DEFAULT_OS_TYPE = OSType.LINUX static public final String DEFAULT_SHARE_ROOT_PATH = "/mnt/batch/tasks/fsmounts" static public final Duration DEFAULT_SCALE_INTERVAL = Duration.of('5 min') diff --git a/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy b/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy index a691ec46f1..7923da46fc 100644 --- a/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy +++ b/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchServiceTest.groovy @@ -452,7 +452,7 @@ class AzBatchServiceTest extends Specification { then: 1 * svc.guessBestVm(LOC, CPUS, MEM, null, TYPE) >> VM and: - spec.poolId == 'nf-pool-289d374ac1622e709cf863bce2570cab-Standard_X1' + spec.poolId == 'nf-pool-e3331cce25aa1563d6046b3de9ec2d93-Standard_X1' spec.metadata == [foo: 'bar'] } diff --git a/plugins/nf-azure/src/test/nextflow/cloud/azure/config/AzureConfigTest.groovy b/plugins/nf-azure/src/test/nextflow/cloud/azure/config/AzureConfigTest.groovy index 07a614944e..74e2a8b6e8 100644 --- a/plugins/nf-azure/src/test/nextflow/cloud/azure/config/AzureConfigTest.groovy +++ b/plugins/nf-azure/src/test/nextflow/cloud/azure/config/AzureConfigTest.groovy @@ -77,7 +77,7 @@ class AzureConfigTest extends Specification { cfg.batch().location == null cfg.batch().autoPoolMode == null cfg.batch().allowPoolCreation == null - cfg.batch().autoPoolOpts().vmType == 'Standard_D4_v3' + cfg.batch().autoPoolOpts().vmType == 'Standard_D4a_v4' cfg.batch().autoPoolOpts().vmCount == 1 cfg.batch().autoPoolOpts().maxVmCount == 3 cfg.batch().autoPoolOpts().scaleInterval == Duration.of('5 min') diff --git a/validation/azure.config b/validation/azure.config index 31574e6263..7e80bc53ed 100644 --- a/validation/azure.config +++ b/validation/azure.config @@ -25,7 +25,6 @@ azure { pools { 'nextflow-ci' { autoScale = true - vmType = 'Standard_D2_v3' } } } From 18b6934948026b1686a8ccd1a5e4737c6cf74987 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 21 Feb 2025 19:35:33 +0100 Subject: [PATCH 05/11] Update Platform token error message [ci skip] Signed-off-by: Paolo Di Tommaso --- .../nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy index d202420138..e81b2c33ae 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy @@ -383,7 +383,7 @@ class TowerClient implements TraceObserver { ? env.get('TOWER_ACCESS_TOKEN') : session.config.navigate('tower.accessToken', env.get('TOWER_ACCESS_TOKEN')) if( !token ) - throw new AbortOperationException("Missing personal access token -- Make sure there's a variable TOWER_ACCESS_TOKEN in your environment") + throw new AbortOperationException("Missing Seqera Platform access token -- Make sure there's a variable TOWER_ACCESS_TOKEN in your environment") return token } From 6a81c0159c614901957dba8b3a870296a8df718d Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Mon, 24 Feb 2025 04:51:28 -0500 Subject: [PATCH 06/11] Fix dynamic publish path (#5809) Signed-off-by: Ben Sherman --- .../src/main/groovy/nextflow/extension/PublishOp.groovy | 4 ++-- .../src/test/groovy/nextflow/script/OutputDslTest.groovy | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/extension/PublishOp.groovy b/modules/nextflow/src/main/groovy/nextflow/extension/PublishOp.groovy index 31d7b8be91..cc71dbb0b2 100644 --- a/modules/nextflow/src/main/groovy/nextflow/extension/PublishOp.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/extension/PublishOp.groovy @@ -138,8 +138,8 @@ class PublishOp { // if the resolved publish path is a string, resolve it // against the base output directory - if( resolvedPath instanceof String ) - return outputDir.resolve(resolvedPath) + if( resolvedPath instanceof CharSequence ) + return outputDir.resolve(resolvedPath.toString()) // if the resolved publish path is a closure, use the closure // to transform each published file and resolve it against diff --git a/modules/nextflow/src/test/groovy/nextflow/script/OutputDslTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/OutputDslTest.groovy index 12754d8350..16e9751bda 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/OutputDslTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/OutputDslTest.groovy @@ -56,7 +56,7 @@ class OutputDslTest extends Specification { when: dsl.target('bar') { - path('barbar') + path { v -> "${'barbar'}" } index { path 'index.csv' } From ed9da469680915b5fba209735880d3b93326cff3 Mon Sep 17 00:00:00 2001 From: Matthew Colpus Date: Mon, 24 Feb 2025 14:44:33 +0000 Subject: [PATCH 07/11] Fix Prevent S3 global option when using custom endpoints (#5779) Signed-off-by: Matthew Colpus Co-authored-by: Matthew Colpus --- .../nextflow/processor/TaskConfig.groovy | 14 ++++++++++---- .../nextflow/processor/TaskConfigTest.groovy | 18 +++++++++--------- .../cloud/aws/nio/S3FileSystemProvider.java | 5 ++++- 3 files changed, 23 insertions(+), 14 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/processor/TaskConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/processor/TaskConfig.groovy index 71cb92afb8..cea384f4ac 100644 --- a/modules/nextflow/src/main/groovy/nextflow/processor/TaskConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/processor/TaskConfig.groovy @@ -21,6 +21,7 @@ import static nextflow.processor.TaskProcessor.* import java.nio.file.Path import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j import nextflow.Const import nextflow.ast.NextflowDSLImpl import nextflow.exception.AbortOperationException @@ -40,6 +41,7 @@ import nextflow.util.MemoryUnit * * @author Paolo Di Tommaso */ +@Slf4j @CompileStatic class TaskConfig extends LazyMap implements Cloneable { @@ -392,10 +394,14 @@ class TaskConfig extends LazyMap implements Cloneable { for( String it : shell ) { if( !it ) throw new IllegalArgumentException("Directive `process.shell` cannot contain empty values - offending value: ${shell}") - if( !it || it.contains('\n') || it.contains('\r') ) - throw new IllegalArgumentException("Directive `process.shell` cannot contain new-line characters - offending value: ${shell}") - if( it.startsWith(' ') || it.endsWith(' ')) - throw new IllegalArgumentException("Directive `process.shell` cannot contain leading or tralining blanks - offending value: ${shell}") + if( !it || it.contains('\n') || it.contains('\r') ) { + log.warn1 "Directive `process.shell` cannot contain new-line characters - offending value: ${shell}" + break + } + if( it.startsWith(' ') || it.endsWith(' ')) { + log.warn "Directive `process.shell` cannot contain leading or tralining blanks - offending value: ${shell}" + break + } } return shell } diff --git a/modules/nextflow/src/test/groovy/nextflow/processor/TaskConfigTest.groovy b/modules/nextflow/src/test/groovy/nextflow/processor/TaskConfigTest.groovy index ace82766bc..d28dc432e6 100644 --- a/modules/nextflow/src/test/groovy/nextflow/processor/TaskConfigTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/processor/TaskConfigTest.groovy @@ -684,14 +684,14 @@ class TaskConfigTest extends Specification { then: thrown(IllegalArgumentException) - when: - config.validateShell(['bash\nthis\nthat']) - then: - thrown(IllegalArgumentException) - - when: - config.validateShell(['bash', ' -eu ']) - then: - thrown(IllegalArgumentException) +// when: +// config.validateShell(['bash\nthis\nthat']) +// then: +// thrown(IllegalArgumentException) +// +// when: +// config.validateShell(['bash', ' -eu ']) +// then: +// thrown(IllegalArgumentException) } } diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java b/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java index c7c4f74164..83ef032059 100644 --- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java +++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java @@ -835,7 +835,10 @@ protected S3FileSystem createFileSystem(URI uri, AwsConfig awsConfig) { ClientConfiguration clientConfig = createClientConfig(props); final String bucketName = S3Path.bucketName(uri); - final boolean global = bucketName!=null; + // do not use `global` flag for custom endpoint because + // when enabling that flag, it overrides S3 endpoints with AWS global endpoint + // see https://github.com/nextflow-io/nextflow/pull/5779 + final boolean global = bucketName!=null && !awsConfig.getS3Config().isCustomEndpoint(); final AwsClientFactory factory = new AwsClientFactory(awsConfig, globalRegion(awsConfig)); client = new S3Client(factory.getS3Client(clientConfig, global)); From b019b0bb7a3b65ce237839534019a1e0d06f9185 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 25 Feb 2025 11:22:42 -0600 Subject: [PATCH 08/11] Add Config parser v2 (and loader) (#4744) Signed-off-by: Ben Sherman Signed-off-by: Paolo Di Tommaso Co-authored-by: Chris Hakkaart Co-authored-by: Paolo Di Tommaso --- .github/workflows/build.yml | 2 +- build.gradle | 4 + docs/config.md | 41 +- docs/reference/env-vars.md | 5 + docs/updating-syntax.md | 6 + modules/nextflow/build.gradle | 1 + .../nextflow/config/ConfigBuilder.groovy | 9 +- .../nextflow/config/ConfigParser.groovy | 495 +------------ .../config/ConfigParserFactory.groovy | 47 ++ .../groovy/nextflow/config/PluginsDsl.groovy | 23 - .../config/{ => parser/v1}/ConfigBase.groovy | 3 +- .../config/parser/v1/ConfigParserV1.groovy | 500 +++++++++++++ .../{ => parser/v1}/ConfigTransform.groovy | 2 +- .../v1}/ConfigTransformImpl.groovy | 3 +- .../config/parser/v1/PluginsDsl.groovy | 38 + .../parser/v2/ClosureToStringVisitor.groovy | 103 +++ .../parser/v2/ClosureToStringXform.groovy | 47 ++ .../config/parser/v2/ConfigDsl.groovy | 226 ++++++ .../config/parser/v2/ConfigParserV2.groovy | 193 +++++ .../parser/v2/ConfigToGroovyVisitor.java | 112 +++ .../parser/v2/ConfigToGroovyXform.groovy | 45 ++ .../groovy/nextflow/scm/AssetManager.groovy | 4 +- .../groovy/nextflow/scm/ProviderConfig.groovy | 4 +- .../groovy/nextflow/scm/ProviderPath.groovy | 2 - .../src/test/groovy/FunctionalTests.groovy | 42 +- .../nextflow/config/ConfigBuilderTest.groovy | 2 +- .../v1/ConfigParserV1Test.groovy} | 176 ++--- .../parser/v2/ConfigParserV2Test.groovy | 664 ++++++++++++++++++ .../nextflow/script/ScriptRunnerTest.groovy | 20 +- ...included => config-labels-included.config} | 0 tests/config-labels.config | 2 +- tests/config-vars.config | 12 +- tests/profiles.config | 3 +- validation/test.sh | 21 +- 34 files changed, 2184 insertions(+), 673 deletions(-) create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy delete mode 100644 modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy rename modules/nextflow/src/main/groovy/nextflow/config/{ => parser/v1}/ConfigBase.groovy (98%) create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy rename modules/nextflow/src/main/groovy/nextflow/config/{ => parser/v1}/ConfigTransform.groovy (97%) rename modules/nextflow/src/main/groovy/nextflow/config/{ => parser/v1}/ConfigTransformImpl.groovy (98%) create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java create mode 100644 modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy rename modules/nextflow/src/test/groovy/nextflow/config/{ConfigParserTest.groovy => parser/v1/ConfigParserV1Test.groovy} (83%) create mode 100644 modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy rename tests/{config-labels.included => config-labels-included.config} (100%) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 3613344ae8..7502adc387 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -133,7 +133,7 @@ jobs: fail-fast: false matrix: java_version: [17, 23] - test_mode: ["test_integration", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"] + test_mode: ["test_integration", "test_parser_v2", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"] steps: - name: Checkout uses: actions/checkout@v4 diff --git a/build.gradle b/build.gradle index 03de4ffe27..af43ab13e5 100644 --- a/build.gradle +++ b/build.gradle @@ -85,6 +85,10 @@ allprojects { maven { url 'https://oss.sonatype.org/content/repositories/snapshots' } maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" } maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" } + maven { + url 'https://jitpack.io' + content { includeGroup 'com.github.nextflow-io.language-server' } + } } configurations { diff --git a/docs/config.md b/docs/config.md index 0818f2e3c1..75cbb5775e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -254,13 +254,12 @@ With the above configuration: ## Config profiles -Configuration files can contain the definition of one or more *profiles*. A profile is a set of configuration attributes that can be selected during pipeline execution by using the `-profile` command line option. +Configuration files can define one or more *profiles*. A profile is a set of configuration settings that can be selected during pipeline execution using the `-profile` command line option. -Configuration profiles are defined by using the special scope `profiles`, which group the attributes that belong to the same profile using a common prefix. For example: +Configuration profiles are defined in the `profiles` scope. For example: ```groovy profiles { - standard { process.executor = 'local' } @@ -276,41 +275,47 @@ profiles { process.container = 'cbcrg/imagex' docker.enabled = true } - } ``` -This configuration defines three different profiles: `standard`, `cluster`, and `cloud`, that each set different process -configuration strategies depending on the target runtime platform. The `standard` profile is used by default when no profile is specified. +The above configuration defines three profiles: `standard`, `cluster`, and `cloud`. Each profile provides a different configuration for a given execution environment. The `standard` profile is used by default when no profile is specified. -:::{tip} -Multiple configuration profiles can be specified by separating the profile names with a comma, for example: +Configuration profiles can be specified at runtime as a comma-separated list: ```bash nextflow run -profile standard,cloud ``` Config profiles are applied in the order in which they were defined in the config file, regardless of the order they are specified on the command line. + +:::{versionadded} 25.02.0-edge +When using the {ref}`strict config syntax `, profiles are applied in the order in which they are specified on the command line. ::: :::{danger} -When using the `profiles` feature in your config file, do NOT set attributes in the same scope both inside and outside a `profiles` context. For example: +When defining a profile in the config file, avoid using both the dot and block syntax for the same scope. For example: ```groovy -process.cpus = 1 - profiles { - foo { - process.memory = '2 GB' - } + foo { + process.memory = '2 GB' + process { + cpus = 2 + } + } +} +``` - bar { - process.memory = '4 GB' - } +Due to a limitation of the legacy config parser, the first setting will be overwritten by the second: + +```console +$ nextflow config -profile foo +process { + cpus = 2 } ``` -In the above example, the `process.cpus` attribute is not correctly applied because the `process` scope is also used in the `foo` and `bar` profiles. +This limitation can be avoided by using the {ref}`strict config syntax `. ::: ## Workflow handlers diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index 8bb17797bd..480a725470 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -182,6 +182,11 @@ The following environment variables control the configuration of the Nextflow ru ::: : Enable the use of Spack recipes defined by using the {ref}`process-spack` directive. (default: `false`). +`NXF_SYNTAX_PARSER` +: :::{versionadded} 25.02.0-edge + ::: +: Set to `'v2'` to use the {ref}`strict syntax ` for Nextflow config files (default: `'v1'`). + `NXF_TEMP` : Directory where temporary files are stored diff --git a/docs/updating-syntax.md b/docs/updating-syntax.md index ce0d3224e4..10b48e8033 100644 --- a/docs/updating-syntax.md +++ b/docs/updating-syntax.md @@ -487,8 +487,14 @@ The process `when` section is deprecated. Use conditional logic, such as an `if` The process `shell` section is deprecated. Use the `script` block instead. The VS Code extension provides syntax highlighting and error checking to help distinguish between Nextflow variables and Bash variables. +(updating-config-syntax)= + ### Configuration syntax +:::{versionadded} 25.02.0-edge +The strict config syntax can be enabled in Nextflow by setting `NXF_SYNTAX_PARSER=v2`. +::: + See {ref}`Configuration ` for a comprehensive description of the configuration language. Currently, Nextflow parses config files as Groovy scripts, allowing the use of scripting constructs like variables, helper functions, try-catch blocks, and conditional logic for dynamic configuration: diff --git a/modules/nextflow/build.gradle b/modules/nextflow/build.gradle index c005504dc3..75a398b364 100644 --- a/modules/nextflow/build.gradle +++ b/modules/nextflow/build.gradle @@ -20,6 +20,7 @@ compileGroovy { dependencies { api(project(':nf-commons')) api(project(':nf-httpfs')) + api 'com.github.nextflow-io.language-server:compiler:main-SNAPSHOT' api "org.apache.groovy:groovy:4.0.25" api "org.apache.groovy:groovy-nio:4.0.25" api "org.apache.groovy:groovy-xml:4.0.25" diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy index 76cada7d75..f8500d63f2 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy @@ -71,8 +71,6 @@ class ConfigBuilder { List parsedConfigFiles = [] - List parsedProfileNames - boolean showClosures boolean stripSecrets @@ -347,7 +345,7 @@ class ConfigBuilder { assert env != null final ignoreIncludes = options ? options.ignoreConfigIncludes : false - final slurper = new ConfigParser() + final slurper = ConfigParserFactory.create() .setRenderClosureAsString(showClosures) .setStripSecrets(stripSecrets) .setIgnoreIncludes(ignoreIncludes) @@ -384,9 +382,8 @@ class ConfigBuilder { } } - this.parsedProfileNames = new ArrayList<>(slurper.getProfileNames()) if( validateProfile ) { - checkValidProfile(slurper.getConditionalBlockNames()) + checkValidProfile(slurper.getProfiles()) } } @@ -421,7 +418,7 @@ class ConfigBuilder { log.debug "Applying config profile: `${profile}`" def allNames = profile.tokenize(',') - slurper.registerConditionalBlock('profiles', allNames) + slurper.setProfiles(allNames) def config = parse0(slurper,entry) validate(config,entry) diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy index 787f1da57d..9a51a02f81 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy @@ -18,503 +18,70 @@ package nextflow.config import java.nio.file.Path -import ch.artecat.grengine.Grengine -import com.google.common.hash.Hashing -import groovy.transform.PackageScope -import nextflow.ast.NextflowXform -import nextflow.exception.ConfigParseException -import nextflow.extension.Bolts -import nextflow.file.FileHelper -import nextflow.util.Duration -import nextflow.util.MemoryUnit -import org.codehaus.groovy.control.CompilerConfiguration -import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer -import org.codehaus.groovy.control.customizers.ImportCustomizer -import org.codehaus.groovy.runtime.InvokerHelper - -/* - * Copyright 2003-2013 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - - /** - * A ConfigSlurper that allows to include a config file into another. For example: - * - * 
- *     process {
- *         foo = 1
- *         that = 2
- *
- *         includeConfig( 'path/to/another/config/file' )
+ * Interface for Nextflow config parsers.
  *
- *     }
- *
- *
- * - * See http://naleid.com/blog/2009/07/30/modularizing-groovy-config-files-with-a-dash-of-meta-programming - * - * @author Paolo Di Tommaso - * - */ - -/** - *

- * ConfigSlurper is a utility class for reading configuration files defined in the form of Groovy - * scripts. Configuration settings can be defined using dot notation or scoped using closures - * - * 


- *   grails.webflow.stateless = true
- *    smtp {
- *        mail.host = 'smtp.myisp.com'
- *        mail.auth.user = 'server'
- *    }
- *    resources.URL = "http://localhost:80/resources"
- *
- * - *

Settings can either be bound into nested maps or onto a specified JavaBean instance. In the case - * of the latter an error will be thrown if a property cannot be bound. - * - * @author Graeme Rocher - * @author Andres Almiray - * @since 1.5 + * @author Ben Sherman */ -class ConfigParser { - private static final ENVIRONMENTS_METHOD = 'environments' - - private Map bindingVars = [:] - private Map paramVars = [:] - - private final Map> conditionValues = [:] - private final Stack> conditionalBlocks = new Stack>() - private final Set conditionalNames = new HashSet<>() - private final Set profileNames = new HashSet<>() - - private boolean ignoreIncludes - - private boolean renderClosureAsString - - private boolean stripSecrets - - private Grengine grengine - - ConfigParser() { - this('') - } - - /** - * Constructs a new IncludeConfigSlurper instance using the given environment - * @param env The Environment to use - */ - ConfigParser(String env) { - conditionValues[ENVIRONMENTS_METHOD] = [env] - } - - ConfigParser registerConditionalBlock(String blockName, String blockValue) { - if (blockName) { - if (!blockValue) { - conditionValues.remove(blockName) - } - else { - conditionValues[blockName] = [blockValue] - } - } - return this - } - - ConfigParser registerConditionalBlock(String blockName, List blockValues) { - if (blockName) { - if (!blockValues) { - conditionValues.remove(blockName) - } - else { - conditionValues[blockName] = blockValues - } - } - return this - } - - /** - * @return - * When a conditional block is registered this method returns the collection - * of block names visited during the parsing - */ - Set getConditionalBlockNames() { - Collections.unmodifiableSet(conditionalNames) - } +interface ConfigParser { /** - * Returns the profile names defined in the config file + * Toggle whether config include statements should be ignored. * - * @return The set of profile names. + * @param value */ - Set getProfileNames() { profileNames } - - private Grengine getGrengine() { - if( grengine ) { - return grengine - } - - // set the required base script - def config = new CompilerConfiguration() - config.scriptBaseClass = ConfigBase.class.name - if( stripSecrets ) - config.addCompilationCustomizers(new ASTTransformationCustomizer(StripSecretsXform)) - def params = [:] - if( renderClosureAsString ) - params.put('renderClosureAsString', true) - config.addCompilationCustomizers(new ASTTransformationCustomizer(params, ConfigTransform)) - config.addCompilationCustomizers(new ASTTransformationCustomizer(NextflowXform)) - // add implicit types - def importCustomizer = new ImportCustomizer() - importCustomizer.addImports( Duration.name ) - importCustomizer.addImports( MemoryUnit.name ) - config.addCompilationCustomizers(importCustomizer) - grengine = new Grengine(config) - } - - ConfigParser setRenderClosureAsString(boolean value) { - this.renderClosureAsString = value - return this - } - - ConfigParser setStripSecrets(boolean value) { - this.stripSecrets = value - return this - } + ConfigParser setIgnoreIncludes(boolean value) /** - * Sets any additional variables that should be placed into the binding when evaluating Config scripts - */ - ConfigParser setBinding(Map vars) { - this.bindingVars = vars - return this - } - - ConfigParser setParams(Map vars) { - // deep clone the map to prevent side-effect - // see https://github.com/nextflow-io/nextflow/issues/1923 - this.paramVars = Bolts.deepClone(vars) - return this - } - - - /** - * Creates a unique name for the config class in order to avoid collision - * with top level configuration scopes + * Toggle whether to strip secrets when rendering the config. * - * @param text - * @return + * @param value */ - private String createUniqueName(String text) { - def hash = Hashing - .murmur3_32() - .newHasher() - .putUnencodedChars(text) - .hash() - return "_nf_config_$hash" - } - - private Script loadScript(String text) { - (Script)getGrengine().load(text, createUniqueName(text)).newInstance() - } + ConfigParser setStripSecrets(boolean value) /** - * Parses a ConfigObject instances from an instance of java.util.Properties - * @param The java.util.Properties instance + * Toggle whether to render the source code of closures. + * + * @param value */ - ConfigObject parse(Properties properties) { - ConfigObject config = new ConfigObject() - for (key in properties.keySet()) { - def tokens = key.split(/\./) - - def current = config - def last - def lastToken - def foundBase = false - for (token in tokens) { - if (foundBase) { - // handle not properly nested tokens by ignoring - // hierarchy below this point - lastToken += "." + token - current = last - } else { - last = current - lastToken = token - current = current."${token}" - if (!(current instanceof ConfigObject)) foundBase = true - } - } + ConfigParser setRenderClosureAsString(boolean value) - if (current instanceof ConfigObject) { - if (last[lastToken]) { - def flattened = last.flatten() - last.clear() - flattened.each { k2, v2 -> last[k2] = v2 } - last[lastToken] = properties.get(key) - } - else { - last[lastToken] = properties.get(key) - } - } - current = config - } - return config - } /** - * Parse the given script as a string and return the configuration object + * Toggle whether to raise an error if a missing property is accessed. * - * @see ConfigParser#parse(groovy.lang.Script) + * @param value */ - ConfigObject parse(String text) { - return parse(loadScript(text)) - } + ConfigParser setStrict(boolean value) /** - * Parse the given script into a configuration object (a Map) - * (This method creates a new class to parse the script each time it is called.) - * @param script The script to parse - * @return A Map of maps that can be navigating with dot de-referencing syntax to obtain configuration entries + * Define variables which will be made available to the config script. + * + * @param vars */ - @Deprecated - ConfigObject parse(Script script) { - return parse(script, null) - } + ConfigParser setBinding(Map vars) /** - * Parses a Script represented by the given URL into a ConfigObject + * Define pipeline parameters which will be made available to the config script. * - * @param location The location of the script to parse - * @return The ConfigObject instance + * @param vars */ - @Deprecated - ConfigObject parse(URL location) { - return parse(loadScript(location.text), FileHelper.asPath(location.toURI())) - } - - ConfigObject parse(File file) { - return parse(file.toPath()) - } - - ConfigObject parse(Path path) { - return parse(loadScript(path.text), path) - } + ConfigParser setParams(Map vars) /** - * Parses the passed groovy.lang.Script instance using the second argument to allow the ConfigObject - * to retain an reference to the original location other Groovy script - * - * @param script The groovy.lang.Script instance - * @param location The original location of the Script as a URL - * @return The ConfigObject instance + * Parse a config object from the given source. */ - ConfigObject parse(Script _script, Path location) { - final script = (ConfigBase)_script - Stack currentConditionalBlock = new Stack() - def config = location ? new ConfigObject(location.toUri().toURL()) : new ConfigObject() - GroovySystem.metaClassRegistry.removeMetaClass(script.class) - def mc = script.class.metaClass - def prefix = "" - LinkedList stack = new LinkedList() - LinkedList profileStack = new LinkedList() - stack << [config: config, scope: [:]] - boolean withinProfile = false - - def pushStack = { co -> - stack << [config: co, scope: stack.last.scope.clone()] - } - def assignName = { name, co -> - def current = stack.last - current.config[name] = co - current.scope[name] = co - } - mc.getProperty = { String name -> - def current = stack.last - def result - if (current.config.get(name)) { - result = current.config.get(name) - } else if (current.scope.get(name)) { - result = current.scope[name] - } else { - try { - result = InvokerHelper.getProperty(this, name) - } catch (GroovyRuntimeException e) { - result = new ConfigObject() - assignName.call(name, result) - } - } - if( name=='params' && result instanceof Map && paramVars ) { - result.putAll(Bolts.deepMerge(result, paramVars)) - } - return result - } - - ConfigObject overrides = new ConfigObject() - mc.invokeMethod = { String name, args -> - def result - if (args.length == 1 && args[0] instanceof Closure) { - if( profileStack && profileStack.last == 'profiles' ) - profileNames.add(name) - - if (name in conditionValues.keySet()) { - try { - if( name == 'profiles' ){ - withinProfile=true - } - currentConditionalBlock.push(name) - conditionalBlocks.push([:]) - args[0].call() - } finally { - currentConditionalBlock.pop() - for (entry in conditionalBlocks.pop().entrySet()) { - def c = stack.last.config - (c != config? c : overrides).merge(entry.value) - } - if( name == 'profiles' ){ - withinProfile=false - } - } - } else if (currentConditionalBlock.size() > 0) { - String conditionalBlockKey = currentConditionalBlock.peek() - conditionalNames.add(name) - if (name in conditionValues[conditionalBlockKey]) { - def co = conditionalBlocks.peek()[conditionalBlockKey] - if( co == null ) { - co = new ConfigObject() - conditionalBlocks.peek()[conditionalBlockKey] = co - } - - pushStack.call(co) - try { - currentConditionalBlock.pop() - args[0].call() - } finally { - currentConditionalBlock.push(conditionalBlockKey) - } - stack.removeLast() - } - } - else if( name == 'plugins' ) { - if( stack.size()>1 ) - throw new ConfigParseException("Plugins definition is only allowed in config top-most scope") - // Implements `plugins` mini-dsl for plugins definition - def dsl = new PluginsDsl() - def clo = args[0] as Closure - clo.delegate = dsl - clo.resolveStrategy = Closure.DELEGATE_ONLY - clo.call() - assignName.call(name, dsl.plugins) - } - else { - def current = name=='profiles' || withinProfile ? stack.first : stack.last - def co - if (current.config.containsKey(name) && current.config.get(name) instanceof ConfigObject) { - co = current.config.get(name) - } - else if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { - co = current.scope.get(name).clone() - } - else { - co = new ConfigObject() - } - - profileStack.add(name) - assignName.call(name, co) - pushStack.call(co) - args[0].call() - stack.removeLast() - profileStack.removeLast() - - if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { - if( current.scope.get(name) != co) { - current.scope.get(name).merge(co) - } - } else { - current.scope.put(name,co) - } - } - } else if (args.length == 2 && args[1] instanceof Closure) { - try { - prefix = name + '.' - assignName.call(name, args[0]) - args[1].call() - } finally { prefix = "" } - } else { - MetaMethod mm = mc.getMetaMethod(name, args) - if (mm) { - result = mm.invoke(delegate, args) - } else { - throw new MissingMethodException(name, getClass(), args) - } - } - result - } - script.metaClass = mc - - def setProperty = { String name, value -> - assignName.call(prefix + name, value) - } - def binding = new ConfigBinding(setProperty) - if (this.bindingVars) { - binding.getVariables().putAll(this.bindingVars) - } - - // add the script file location into the binding - if( location ) { - script.setConfigPath(location) - } - - // disable include parsing when required - script.setIgnoreIncludes(ignoreIncludes) - script.setRenderClosureAsString(renderClosureAsString) - script.setStripSecrets(stripSecrets) - - // -- set the binding and run - script.binding = binding - script.run() - config.merge(overrides) - - return config - } + ConfigObject parse(String text) + ConfigObject parse(File file) + ConfigObject parse(Path path) /** - * Disable parsing of {@code includeConfig} directive - * - * @param value A boolean value, when {@code true} includes are disabled - * @return The {@link ConfigParser} object itself + * Set the profiles that should be applied. */ - ConfigParser setIgnoreIncludes(boolean value) { - this.ignoreIncludes = value - return this - } + ConfigParser setProfiles(List profiles) /** - * Since Groovy Script doesn't support overriding setProperty, we have to using a trick with the Binding to provide this - * functionality + * Get the set of available profiles. */ - @PackageScope - static class ConfigBinding extends Binding { - Closure callable + Set getProfiles() - ConfigBinding(Closure c) { - this.callable = c - } - - void setVariable(String name, Object value) { - callable(name, value) - } - } } - diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy new file mode 100644 index 0000000000..d61200e7d3 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy @@ -0,0 +1,47 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package nextflow.config + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.SysEnv +import nextflow.config.parser.v1.ConfigParserV1 +import nextflow.config.parser.v2.ConfigParserV2 + +/** + * Factory for creating an instance of {@link ConfigParser}. + * + * @author Ben Sherman + */ +@Slf4j +@CompileStatic +class ConfigParserFactory { + + static ConfigParser create() { + final parser = SysEnv.get('NXF_SYNTAX_PARSER', 'v1') + if( parser == 'v1' ) { + return new ConfigParserV1() + } + if( parser == 'v2' ) { + log.debug "Using config parser v2" + return new ConfigParserV2() + } + throw new IllegalStateException("Invalid NXF_SYNTAX_PARSER setting -- should be either 'v1' or 'v2'") + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy deleted file mode 100644 index e840df8028..0000000000 --- a/modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy +++ /dev/null @@ -1,23 +0,0 @@ -package nextflow.config - -import groovy.transform.CompileStatic - -/** - * Model a mini-dsl for plugins configuration - * - * @author Paolo Di Tommaso - */ -@CompileStatic -class PluginsDsl { - - private Set plugins = [] - - Set getPlugins() { plugins } - - void id( String plg ) { - if( !plg ) - throw new IllegalArgumentException("Plugin id cannot be empty or null") - plugins << plg - } - -} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBase.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigBase.groovy similarity index 98% rename from modules/nextflow/src/main/groovy/nextflow/config/ConfigBase.groovy rename to modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigBase.groovy index a97e23f19a..e970d1e80a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBase.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigBase.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import java.nio.file.NoSuchFileException import java.nio.file.Path @@ -22,6 +22,7 @@ import java.nio.file.Path import ch.artecat.grengine.Grengine import groovy.transform.Memoized import nextflow.SysEnv +import nextflow.config.StripSecretsXform import nextflow.exception.IllegalConfigException import nextflow.file.FileHelper import org.codehaus.groovy.control.CompilerConfiguration diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy new file mode 100644 index 0000000000..a259185a96 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy @@ -0,0 +1,500 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v1 + +import java.nio.file.Path + +import ch.artecat.grengine.Grengine +import com.google.common.hash.Hashing +import groovy.transform.PackageScope +import nextflow.ast.NextflowXform +import nextflow.config.ConfigParser +import nextflow.config.StripSecretsXform +import nextflow.exception.ConfigParseException +import nextflow.extension.Bolts +import nextflow.file.FileHelper +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import org.codehaus.groovy.control.CompilerConfiguration +import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer +import org.codehaus.groovy.control.customizers.ImportCustomizer +import org.codehaus.groovy.runtime.InvokerHelper + +/* + * Copyright 2003-2013 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + + +/** + * A ConfigSlurper that allows to include a config file into another. For example: + * + * 

+ *     process {
+ *         foo = 1
+ *         that = 2
+ *
+ *         includeConfig( 'path/to/another/config/file' )
+ *
+ *     }
+ *
+ *
+ * + * See http://naleid.com/blog/2009/07/30/modularizing-groovy-config-files-with-a-dash-of-meta-programming + * + * @author Paolo Di Tommaso + * + */ + +/** + *

+ * ConfigSlurper is a utility class for reading configuration files defined in the form of Groovy + * scripts. Configuration settings can be defined using dot notation or scoped using closures + * + * 


+ *   grails.webflow.stateless = true
+ *    smtp {
+ *        mail.host = 'smtp.myisp.com'
+ *        mail.auth.user = 'server'
+ *    }
+ *    resources.URL = "http://localhost:80/resources"
+ *
+ * + *

Settings can either be bound into nested maps or onto a specified JavaBean instance. In the case + * of the latter an error will be thrown if a property cannot be bound. + * + * @author Graeme Rocher + * @author Andres Almiray + * @since 1.5 + */ +class ConfigParserV1 implements ConfigParser { + private Map bindingVars = [:] + private Map paramVars = [:] + + private final Map> conditionValues = [:] + private final Stack> conditionalBlocks = new Stack>() + private final Set conditionalNames = new HashSet<>() + private final Set profileNames = new HashSet<>() + + private boolean ignoreIncludes + + private boolean renderClosureAsString + + private boolean stripSecrets + + private Grengine grengine + + @Override + ConfigParser setProfiles(List profiles) { + final blockName = 'profiles' + if (!profiles) { + conditionValues.remove(blockName) + } + else { + conditionValues[blockName] = profiles + } + return this + } + + @Override + Set getProfiles() { + Collections.unmodifiableSet(conditionalNames) + } + + private Grengine getGrengine() { + if( grengine ) { + return grengine + } + + // set the required base script + def config = new CompilerConfiguration() + config.scriptBaseClass = ConfigBase.class.name + if( stripSecrets ) + config.addCompilationCustomizers(new ASTTransformationCustomizer(StripSecretsXform)) + def params = [:] + if( renderClosureAsString ) + params.put('renderClosureAsString', true) + config.addCompilationCustomizers(new ASTTransformationCustomizer(params, ConfigTransform)) + config.addCompilationCustomizers(new ASTTransformationCustomizer(NextflowXform)) + // add implicit types + def importCustomizer = new ImportCustomizer() + importCustomizer.addImports( Duration.name ) + importCustomizer.addImports( MemoryUnit.name ) + config.addCompilationCustomizers(importCustomizer) + grengine = new Grengine(config) + } + + @Override + ConfigParser setRenderClosureAsString(boolean value) { + this.renderClosureAsString = value + return this + } + + @Override + ConfigParser setStrict(boolean value) { + // not supported + return this + } + + @Override + ConfigParser setStripSecrets(boolean value) { + this.stripSecrets = value + return this + } + + /** + * Sets any additional variables that should be placed into the binding when evaluating Config scripts + */ + @Override + ConfigParser setBinding(Map vars) { + this.bindingVars = vars + return this + } + + @Override + ConfigParser setParams(Map vars) { + // deep clone the map to prevent side-effect + // see https://github.com/nextflow-io/nextflow/issues/1923 + this.paramVars = Bolts.deepClone(vars) + return this + } + + + /** + * Creates a unique name for the config class in order to avoid collision + * with top level configuration scopes + * + * @param text + * @return + */ + private String createUniqueName(String text) { + def hash = Hashing + .murmur3_32() + .newHasher() + .putUnencodedChars(text) + .hash() + return "_nf_config_$hash" + } + + private Script loadScript(String text) { + (Script)getGrengine().load(text, createUniqueName(text)).newInstance() + } + + /** + * Parses a ConfigObject instances from an instance of java.util.Properties + * @param The java.util.Properties instance + */ + ConfigObject parse(Properties properties) { + ConfigObject config = new ConfigObject() + for (key in properties.keySet()) { + def tokens = key.split(/\./) + + def current = config + def last + def lastToken + def foundBase = false + for (token in tokens) { + if (foundBase) { + // handle not properly nested tokens by ignoring + // hierarchy below this point + lastToken += "." + token + current = last + } else { + last = current + lastToken = token + current = current."${token}" + if (!(current instanceof ConfigObject)) foundBase = true + } + } + + if (current instanceof ConfigObject) { + if (last[lastToken]) { + def flattened = last.flatten() + last.clear() + flattened.each { k2, v2 -> last[k2] = v2 } + last[lastToken] = properties.get(key) + } + else { + last[lastToken] = properties.get(key) + } + } + current = config + } + return config + } + + /** + * Parse the given script as a string and return the configuration object + * + * @see ConfigParser#parse(groovy.lang.Script) + */ + @Override + ConfigObject parse(String text) { + return parse(loadScript(text)) + } + + /** + * Parse the given script into a configuration object (a Map) + * (This method creates a new class to parse the script each time it is called.) + * @param script The script to parse + * @return A Map of maps that can be navigating with dot de-referencing syntax to obtain configuration entries + */ + @Deprecated + ConfigObject parse(Script script) { + return parse(script, null) + } + + /** + * Parses a Script represented by the given URL into a ConfigObject + * + * @param location The location of the script to parse + * @return The ConfigObject instance + */ + @Deprecated + ConfigObject parse(URL location) { + return parse(loadScript(location.text), FileHelper.asPath(location.toURI())) + } + + @Override + ConfigObject parse(File file) { + return parse(file.toPath()) + } + + @Override + ConfigObject parse(Path path) { + return parse(loadScript(path.text), path) + } + + /** + * Parses the passed groovy.lang.Script instance using the second argument to allow the ConfigObject + * to retain an reference to the original location other Groovy script + * + * @param script The groovy.lang.Script instance + * @param location The original location of the Script as a URL + * @return The ConfigObject instance + */ + ConfigObject parse(Script _script, Path location) { + final script = (ConfigBase)_script + Stack currentConditionalBlock = new Stack() + def config = location ? new ConfigObject(location.toUri().toURL()) : new ConfigObject() + GroovySystem.metaClassRegistry.removeMetaClass(script.class) + def mc = script.class.metaClass + def prefix = "" + LinkedList stack = new LinkedList() + LinkedList profileStack = new LinkedList() + stack << [config: config, scope: [:]] + boolean withinProfile = false + + def pushStack = { co -> + stack << [config: co, scope: stack.last.scope.clone()] + } + def assignName = { name, co -> + def current = stack.last + current.config[name] = co + current.scope[name] = co + } + mc.getProperty = { String name -> + def current = stack.last + def result + if (current.config.get(name)) { + result = current.config.get(name) + } else if (current.scope.get(name)) { + result = current.scope[name] + } else { + try { + result = InvokerHelper.getProperty(this, name) + } catch (GroovyRuntimeException e) { + result = new ConfigObject() + assignName.call(name, result) + } + } + if( name=='params' && result instanceof Map && paramVars ) { + result.putAll(Bolts.deepMerge(result, paramVars)) + } + return result + } + + ConfigObject overrides = new ConfigObject() + mc.invokeMethod = { String name, args -> + def result + if (args.length == 1 && args[0] instanceof Closure) { + if( profileStack && profileStack.last == 'profiles' ) + profileNames.add(name) + + if (name in conditionValues.keySet()) { + try { + if( name == 'profiles' ){ + withinProfile=true + } + currentConditionalBlock.push(name) + conditionalBlocks.push([:]) + args[0].call() + } finally { + currentConditionalBlock.pop() + for (entry in conditionalBlocks.pop().entrySet()) { + def c = stack.last.config + (c != config? c : overrides).merge(entry.value) + } + if( name == 'profiles' ){ + withinProfile=false + } + } + } else if (currentConditionalBlock.size() > 0) { + String conditionalBlockKey = currentConditionalBlock.peek() + conditionalNames.add(name) + if (name in conditionValues[conditionalBlockKey]) { + def co = conditionalBlocks.peek()[conditionalBlockKey] + if( co == null ) { + co = new ConfigObject() + conditionalBlocks.peek()[conditionalBlockKey] = co + } + + pushStack.call(co) + try { + currentConditionalBlock.pop() + args[0].call() + } finally { + currentConditionalBlock.push(conditionalBlockKey) + } + stack.removeLast() + } + } + else if( name == 'plugins' ) { + if( stack.size()>1 ) + throw new ConfigParseException("Plugins definition is only allowed in config top-most scope") + // Implements `plugins` mini-dsl for plugins definition + def dsl = new PluginsDsl() + def clo = args[0] as Closure + clo.delegate = dsl + clo.resolveStrategy = Closure.DELEGATE_ONLY + clo.call() + assignName.call(name, dsl.plugins) + } + else { + def current = name=='profiles' || withinProfile ? stack.first : stack.last + def co + if (current.config.containsKey(name) && current.config.get(name) instanceof ConfigObject) { + co = current.config.get(name) + } + else if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { + co = current.scope.get(name).clone() + } + else { + co = new ConfigObject() + } + + profileStack.add(name) + assignName.call(name, co) + pushStack.call(co) + args[0].call() + stack.removeLast() + profileStack.removeLast() + + if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { + if( current.scope.get(name) != co) { + current.scope.get(name).merge(co) + } + } else { + current.scope.put(name,co) + } + } + } else if (args.length == 2 && args[1] instanceof Closure) { + try { + prefix = name + '.' + assignName.call(name, args[0]) + args[1].call() + } finally { prefix = "" } + } else { + MetaMethod mm = mc.getMetaMethod(name, args) + if (mm) { + result = mm.invoke(delegate, args) + } else { + throw new MissingMethodException(name, getClass(), args) + } + } + result + } + script.metaClass = mc + + def setProperty = { String name, value -> + assignName.call(prefix + name, value) + } + def binding = new ConfigBinding(setProperty) + if (this.bindingVars) { + binding.getVariables().putAll(this.bindingVars) + } + + // add the script file location into the binding + if( location ) { + script.setConfigPath(location) + } + + // disable include parsing when required + script.setIgnoreIncludes(ignoreIncludes) + script.setRenderClosureAsString(renderClosureAsString) + script.setStripSecrets(stripSecrets) + + // -- set the binding and run + script.binding = binding + script.run() + config.merge(overrides) + + return config + } + + /** + * Disable parsing of {@code includeConfig} directive + * + * @param value A boolean value, when {@code true} includes are disabled + * @return The {@link ConfigParser} object itself + */ + @Override + ConfigParser setIgnoreIncludes(boolean value) { + this.ignoreIncludes = value + return this + } + + /** + * Since Groovy Script doesn't support overriding setProperty, we have to using a trick with the Binding to provide this + * functionality + */ + @PackageScope + static class ConfigBinding extends Binding { + Closure callable + + ConfigBinding(Closure c) { + this.callable = c + } + + void setVariable(String name, Object value) { + callable(name, value) + } + } +} + diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransform.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransform.groovy similarity index 97% rename from modules/nextflow/src/main/groovy/nextflow/config/ConfigTransform.groovy rename to modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransform.groovy index 4dc8164aa4..bb6430c9b5 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransform.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransform.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import java.lang.annotation.ElementType import java.lang.annotation.Retention diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransformImpl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransformImpl.groovy similarity index 98% rename from modules/nextflow/src/main/groovy/nextflow/config/ConfigTransformImpl.groovy rename to modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransformImpl.groovy index e6d8723388..65c5b9bf4a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransformImpl.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransformImpl.groovy @@ -14,11 +14,12 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import groovy.transform.CompileStatic import groovy.util.logging.Slf4j +import nextflow.config.ConfigClosurePlaceholder import org.codehaus.groovy.ast.ASTNode import org.codehaus.groovy.ast.AnnotationNode import org.codehaus.groovy.ast.ClassCodeVisitorSupport diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy new file mode 100644 index 0000000000..a46ebb935d --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy @@ -0,0 +1,38 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.config.parser.v1 + +import groovy.transform.CompileStatic + +/** + * Model a mini-dsl for plugins configuration + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class PluginsDsl { + + private Set plugins = [] + + Set getPlugins() { plugins } + + void id( String plg ) { + if( !plg ) + throw new IllegalArgumentException("Plugin id cannot be empty or null") + plugins << plg + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy new file mode 100644 index 0000000000..e3b9518cc7 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy @@ -0,0 +1,103 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import groovy.transform.CompileStatic +import nextflow.config.ConfigClosurePlaceholder +import org.codehaus.groovy.ast.ClassCodeVisitorSupport +import org.codehaus.groovy.ast.ClassNode +import org.codehaus.groovy.ast.expr.ArgumentListExpression +import org.codehaus.groovy.ast.expr.ClosureExpression +import org.codehaus.groovy.ast.expr.ConstantExpression +import org.codehaus.groovy.ast.expr.ConstructorCallExpression +import org.codehaus.groovy.ast.expr.Expression +import org.codehaus.groovy.ast.expr.MapExpression +import org.codehaus.groovy.ast.expr.MethodCallExpression +import org.codehaus.groovy.control.SourceUnit +/** + * AST transformation to render closure source text + * + * @author Ben Sherman + */ +@CompileStatic +class ClosureToStringVisitor extends ClassCodeVisitorSupport { + + protected SourceUnit sourceUnit + + ClosureToStringVisitor(SourceUnit sourceUnit) { + this.sourceUnit = sourceUnit + } + + @Override + protected SourceUnit getSourceUnit() { sourceUnit } + + @Override + void visitMethodCallExpression(MethodCallExpression methodCall) { + final name = methodCall.methodAsString + if( name != 'assign' ) { + super.visitMethodCallExpression(methodCall) + return + } + + final arguments = (ArgumentListExpression)methodCall.arguments + if( arguments.size() != 2 ) + return + + final arg = arguments.last() + if( arg instanceof MapExpression ) { + for( final entry : arg.mapEntryExpressions ) { + if( entry.valueExpression instanceof ClosureExpression ) + entry.valueExpression = closureToString(entry.valueExpression) + } + } + if( arg instanceof ClosureExpression ) { + final placeholder = closureToString(arg) + methodCall.arguments = new ArgumentListExpression(arguments[0], placeholder) + } + } + + protected Expression closureToString(Expression closure) { + final buffer = new StringBuilder() + readSource(closure, buffer) + final str = new ConstantExpression(buffer.toString()) + + final type = new ClassNode(ConfigClosurePlaceholder) + final args = new ArgumentListExpression(str) + return new ConstructorCallExpression(type, args) + } + + protected void readSource(Expression expr, StringBuilder buffer) { + final colBegin = Math.max(expr.getColumnNumber()-1, 0) + final colEnd = Math.max(expr.getLastColumnNumber()-1, 0) + final lineFirst = expr.getLineNumber() + final lineLast = expr.getLastLineNumber() + + for( int i=lineFirst; i<=lineLast; i++ ) { + def line = sourceUnit.source.getLine(i, null) + if( i==lineFirst ) { + def str = i==lineLast ? line.substring(colBegin,colEnd) : line.substring(colBegin) + buffer.append(str) + } + else { + def str = i==lineLast ? line.substring(0, colEnd) : line + buffer.append('\n') + buffer.append(str) + } + } + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy new file mode 100644 index 0000000000..dcd60f84e1 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy @@ -0,0 +1,47 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.lang.annotation.ElementType +import java.lang.annotation.Retention +import java.lang.annotation.RetentionPolicy +import java.lang.annotation.Target + +import groovy.transform.CompileStatic +import org.codehaus.groovy.ast.ASTNode +import org.codehaus.groovy.ast.ClassNode +import org.codehaus.groovy.control.CompilePhase +import org.codehaus.groovy.control.SourceUnit +import org.codehaus.groovy.transform.ASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformationClass + +@Retention(RetentionPolicy.SOURCE) +@Target(ElementType.METHOD) +@GroovyASTTransformationClass(classes = [ClosureToStringXformImpl]) +@interface ClosureToStringXform { + + @CompileStatic + @GroovyASTTransformation(phase = CompilePhase.CONVERSION) + class ClosureToStringXformImpl implements ASTTransformation { + @Override + void visit(ASTNode[] astNodes, SourceUnit sourceUnit) { + final clazz = (ClassNode)astNodes[1] + new ClosureToStringVisitor(sourceUnit).visitClass(clazz) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy new file mode 100644 index 0000000000..e0a1bcfc96 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy @@ -0,0 +1,226 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.nio.file.NoSuchFileException +import java.nio.file.Path +import java.nio.file.Paths + +import groovy.transform.CompileStatic +import groovy.transform.Memoized +import groovy.util.logging.Slf4j +import nextflow.SysEnv +import nextflow.exception.ConfigParseException +import nextflow.extension.Bolts +import nextflow.file.FileHelper +/** + * Builder DSL for Nextflow config files. + * + * @author Ben Sherman + * @author Paolo Di Tommaso + */ +@Slf4j +@CompileStatic +class ConfigDsl extends Script { + + private boolean ignoreIncludes + + private boolean renderClosureAsString + + private boolean strict + + private Path configPath + + private Map target = [:] + + void setIgnoreIncludes(boolean value) { + this.ignoreIncludes = value + } + + void setRenderClosureAsString(boolean value) { + this.renderClosureAsString = value + } + + void setStrict(boolean value) { + this.strict = value + } + + void setConfigPath(Path path) { + this.configPath = path + } + + void setParams(Map params) { + target.params = params + } + + Map getTarget() { + if( !target.params ) + target.remove('params') + return target + } + + Object run() {} + + @Override + def getProperty(String name) { + if( name == 'params' ) + return target.params + + try { + return super.getProperty(name) + } + catch( MissingPropertyException e ) { + if( strict ) + throw e + else + return null + } + } + + void append(List names, Object right) { + final values = (Set) navigate(names.init()).computeIfAbsent(names.last(), (k) -> new HashSet<>()) + values.add(right) + } + + void assign(List names, Object right) { + navigate(names.init()).put(names.last(), right) + } + + private Map navigate(List names) { + Map ctx = target + for( final name : names ) { + if( name !in ctx ) ctx[name] = [:] + ctx = ctx[name] as Map + } + return ctx + } + + void block(String name, Closure closure) { + block([name], closure) + } + + void block(List names, Closure closure) { + final delegate = new ConfigBlockDsl(this, names) + final cl = (Closure)closure.clone() + cl.setResolveStrategy(Closure.DELEGATE_FIRST) + cl.setDelegate(delegate) + cl.call() + } + + /** + * Get the value of an environment variable from the launch environment. + * + * @param name + */ + String env(String name) { + return SysEnv.get(name) + } + + void includeConfig(String includeFile) { + includeConfig([], includeFile) + } + + void includeConfig(List names, String includeFile) { + assert includeFile + + if( ignoreIncludes ) + return + + Path includePath = FileHelper.asPath(includeFile) + log.trace "Include config file: $includeFile [parent: $configPath]" + + if( !includePath.isAbsolute() && configPath ) + includePath = configPath.resolveSibling(includeFile) + + final configText = readConfigFile(includePath) + final config = new ConfigParserV2() + .setIgnoreIncludes(ignoreIncludes) + .setRenderClosureAsString(renderClosureAsString) + .setStrict(strict) + .setBinding(binding.getVariables()) + .parse(configText, includePath) + + final ctx = navigate(names) + ctx.putAll(Bolts.deepMerge(ctx, config)) + } + + /** + * Read the content of a config file. The result is cached to + * avoid multiple reads. + * + * @param includePath + */ + @Memoized + protected static String readConfigFile(Path includePath) { + try { + return includePath.getText() + } + catch (NoSuchFileException | FileNotFoundException ignored) { + throw new NoSuchFileException("Config file does not exist: ${includePath.toUriString()}") + } + catch (IOException e) { + throw new IOException("Cannot read config file include: ${includePath.toUriString()}", e) + } + } + + static class ConfigBlockDsl { + private ConfigDsl dsl + private List scope + + ConfigBlockDsl(ConfigDsl dsl, List scope) { + this.dsl = dsl + this.scope = scope + } + + void append(String name, Object right) { + dsl.append(scope, right) + } + + void assign(List names, Object right) { + dsl.assign(scope + names, right) + } + + void block(String name, Closure closure) { + dsl.block(scope + [name], closure) + } + + void withLabel(String label, Closure closure) { + if( !isWithinProcessScope() ) + throw new ConfigParseException("Process selectors are only allowed in the `process` scope (offending scope: `${scope.join('.')}`)") + dsl.block(scope + ["withLabel:${label}".toString()], closure) + } + + void withName(String selector, Closure closure) { + if( !isWithinProcessScope() ) + throw new ConfigParseException("Process selectors are only allowed in the `process` scope (offending scope: `${scope.join('.')}`)") + dsl.block(scope + ["withName:${selector}".toString()], closure) + } + + private boolean isWithinProcessScope() { + if( scope.size() == 1 ) + return scope.first() == 'process' + if( scope.size() == 3 ) + return scope.first() == 'profiles' && scope.last() == 'process' + return false + } + + void includeConfig(String includeFile) { + dsl.includeConfig(scope, includeFile) + } + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy new file mode 100644 index 0000000000..1b79af0ce6 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy @@ -0,0 +1,193 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.nio.file.Path + +import com.google.common.hash.Hashing +import groovy.transform.CompileStatic +import nextflow.ast.NextflowXform +import nextflow.config.ConfigParser +import nextflow.config.StripSecretsXform +import nextflow.config.parser.ConfigParserPluginFactory +import nextflow.extension.Bolts +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import org.codehaus.groovy.control.CompilerConfiguration +import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer +import org.codehaus.groovy.control.customizers.ImportCustomizer + +/** + * The parser for Nextflow config files. + * + * @author Ben Sherman + */ +@CompileStatic +class ConfigParserV2 implements ConfigParser { + + private Map bindingVars = [:] + + private Map paramVars = [:] + + private boolean ignoreIncludes = false + + private boolean renderClosureAsString = false + + private boolean strict = true + + private boolean stripSecrets + + private List appliedProfiles + + private Set parsedProfiles = [] + + private GroovyShell groovyShell + + @Override + ConfigParserV2 setProfiles(List profiles) { + this.appliedProfiles = profiles + return this + } + + @Override + Set getProfiles() { + return parsedProfiles + } + + @Override + ConfigParserV2 setIgnoreIncludes(boolean value) { + this.ignoreIncludes = value + return this + } + + @Override + ConfigParserV2 setRenderClosureAsString(boolean value) { + this.renderClosureAsString = value + return this + } + + @Override + ConfigParserV2 setStrict(boolean value) { + this.strict = value + return this + } + + @Override + ConfigParser setStripSecrets(boolean value) { + this.stripSecrets = value + return this + } + + @Override + ConfigParserV2 setBinding(Map vars) { + this.bindingVars = vars + return this + } + + @Override + ConfigParserV2 setParams(Map vars) { + // deep clone the map to prevent side-effect + // see https://github.com/nextflow-io/nextflow/issues/1923 + this.paramVars = Bolts.deepClone(vars) + return this + } + + /** + * Parse the given script as a string and return the configuration object + * + * @param text + * @param location + */ + @Override + ConfigObject parse(String text) { + parse(text, null) + } + + ConfigObject parse(String text, Path location) { + final groovyShell = getGroovyShell() + final script = (ConfigDsl) groovyShell.parse(text, uniqueClassName(text)) + if( location ) + script.setConfigPath(location) + script.setIgnoreIncludes(ignoreIncludes) + script.setRenderClosureAsString(renderClosureAsString) + if( location ) + script.setConfigPath(location) + + script.setBinding(new Binding(bindingVars)) + script.setParams(paramVars) + script.run() + + final result = Bolts.toConfigObject(script.getTarget()) + final profiles = (result.profiles ?: [:]) as ConfigObject + parsedProfiles.addAll(profiles.keySet()) + if( appliedProfiles ) { + for( final profile : appliedProfiles ) { + if( profile in profiles.keySet() ) + result.merge(profiles[profile] as ConfigObject) + } + result.remove('profiles') + } + + return result + } + + @Override + ConfigObject parse(File file) { + return parse(file.toPath()) + } + + @Override + ConfigObject parse(Path path) { + return parse(path.text, path) + } + + private GroovyShell getGroovyShell() { + if( groovyShell ) + return groovyShell + final classLoader = new GroovyClassLoader() + final config = new CompilerConfiguration() + config.setScriptBaseClass(ConfigDsl.class.getName()) + config.setPluginFactory(new ConfigParserPluginFactory()) + config.addCompilationCustomizers(new ASTTransformationCustomizer(ConfigToGroovyXform)) + if( stripSecrets ) + config.addCompilationCustomizers(new ASTTransformationCustomizer(StripSecretsXform)) + if( renderClosureAsString ) + config.addCompilationCustomizers(new ASTTransformationCustomizer(ClosureToStringXform)) + config.addCompilationCustomizers(new ASTTransformationCustomizer(NextflowXform)) + final importCustomizer = new ImportCustomizer() + importCustomizer.addImports( Duration.name ) + importCustomizer.addImports( MemoryUnit.name ) + config.addCompilationCustomizers(importCustomizer) + return groovyShell = new GroovyShell(classLoader, new Binding(), config) + } + + /** + * Creates a unique name for the config class in order to avoid collision + * with config DSL + * + * @param text + */ + private String uniqueClassName(String text) { + def hash = Hashing + .sipHash24() + .newHasher() + .putUnencodedChars(text) + .hash() + return "_nf_config_$hash" + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java new file mode 100644 index 0000000000..dd7d447f26 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java @@ -0,0 +1,112 @@ +/* + * Copyright 2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.config.parser.v2; + +import java.util.ArrayList; +import java.util.stream.Collectors; + +import nextflow.config.ast.ConfigAppendNode; +import nextflow.config.ast.ConfigAssignNode; +import nextflow.config.ast.ConfigBlockNode; +import nextflow.config.ast.ConfigIncludeNode; +import nextflow.config.ast.ConfigNode; +import nextflow.config.ast.ConfigVisitorSupport; +import org.codehaus.groovy.ast.VariableScope; +import org.codehaus.groovy.ast.expr.Expression; +import org.codehaus.groovy.ast.stmt.ReturnStatement; +import org.codehaus.groovy.ast.stmt.Statement; +import org.codehaus.groovy.control.SourceUnit; + +import static org.codehaus.groovy.ast.tools.GeneralUtils.*; + +/** + * Visitor to convert a Nextflow config AST into a + * Groovy AST which is executed against {@link ConfigDsl}. + * + * @author Ben Sherman + */ +public class ConfigToGroovyVisitor extends ConfigVisitorSupport { + + private SourceUnit sourceUnit; + + private ConfigNode moduleNode; + + public ConfigToGroovyVisitor(SourceUnit sourceUnit) { + this.sourceUnit = sourceUnit; + this.moduleNode = (ConfigNode) sourceUnit.getAST(); + } + + @Override + protected SourceUnit getSourceUnit() { + return sourceUnit; + } + + public void visit() { + if( moduleNode == null ) + return; + super.visit(moduleNode); + if( moduleNode.isEmpty() ) + moduleNode.addStatement(ReturnStatement.RETURN_NULL_OR_VOID); + } + + @Override + public void visitConfigAssign(ConfigAssignNode node) { + moduleNode.addStatement(transformConfigAssign(node)); + } + + protected Statement transformConfigAssign(ConfigAssignNode node) { + if( node instanceof ConfigAppendNode ) { + var name = node.names.get(0); + return stmt(callThisX("append", args(constX(name), node.value))); + } + var names = listX( + node.names.stream() + .map(name -> (Expression) constX(name)) + .collect(Collectors.toList()) + ); + return stmt(callThisX("assign", args(names, node.value))); + } + + @Override + public void visitConfigBlock(ConfigBlockNode node) { + moduleNode.addStatement(transformConfigBlock(node)); + } + + protected Statement transformConfigBlock(ConfigBlockNode node) { + var statements = new ArrayList(); + for( var stmt : node.statements ) { + if( stmt instanceof ConfigAssignNode can ) + statements.add(transformConfigAssign(can)); + else if( stmt instanceof ConfigBlockNode cbn ) + statements.add(transformConfigBlock(cbn)); + else if( stmt instanceof ConfigIncludeNode cin ) + statements.add(transformConfigInclude(cin)); + } + var code = block(new VariableScope(), statements); + var kind = node.kind != null ? node.kind : "block"; + return stmt(callThisX(kind, args(constX(node.name), closureX(code)))); + } + + @Override + public void visitConfigInclude(ConfigIncludeNode node) { + moduleNode.addStatement(transformConfigInclude(node)); + } + + protected Statement transformConfigInclude(ConfigIncludeNode node) { + return stmt(callThisX("includeConfig", args(node.source))); + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy new file mode 100644 index 0000000000..677ca16d11 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy @@ -0,0 +1,45 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.lang.annotation.ElementType +import java.lang.annotation.Retention +import java.lang.annotation.RetentionPolicy +import java.lang.annotation.Target + +import groovy.transform.CompileStatic +import org.codehaus.groovy.ast.ASTNode +import org.codehaus.groovy.control.CompilePhase +import org.codehaus.groovy.control.SourceUnit +import org.codehaus.groovy.transform.ASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformationClass + +@Retention(RetentionPolicy.SOURCE) +@Target(ElementType.METHOD) +@GroovyASTTransformationClass(classes = [ConfigToGroovyXformImpl]) +@interface ConfigToGroovyXform { + + @CompileStatic + @GroovyASTTransformation(phase = CompilePhase.CONVERSION) + class ConfigToGroovyXformImpl implements ASTTransformation { + @Override + public void visit(ASTNode[] astNodes, SourceUnit sourceUnit) { + new ConfigToGroovyVisitor(sourceUnit).visit() + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy index a1b7d57683..5aa65364e6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy @@ -28,7 +28,7 @@ import groovy.transform.ToString import groovy.transform.TupleConstructor import groovy.util.logging.Slf4j import nextflow.cli.HubOptions -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.config.Manifest import nextflow.exception.AbortOperationException import nextflow.exception.AmbiguousPipelineNameException @@ -455,7 +455,7 @@ class AssetManager { } if( text ) try { - def config = new ConfigParser().setIgnoreIncludes(true).parse(text) + def config = ConfigParserFactory.create().setIgnoreIncludes(true).setStrict(false).parse(text) result = (ConfigObject)config.manifest } catch( Exception e ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy index 63000e72f4..28064f9aaa 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy @@ -23,7 +23,7 @@ import groovy.transform.CompileStatic import groovy.transform.PackageScope import groovy.util.logging.Slf4j import nextflow.Const -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.exception.AbortOperationException import nextflow.exception.ConfigParseException import nextflow.file.FileHelper @@ -245,7 +245,7 @@ class ProviderConfig { @PackageScope static Map parse(String text) { - def slurper = new ConfigParser() + def slurper = ConfigParserFactory.create() slurper.setBinding(env) return slurper.parse(text) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy index 8d73d965fd..366f57e0db 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy @@ -34,9 +34,7 @@ import groovy.util.logging.Slf4j * project hosted in a source code repository * * @see nextflow.config.ConfigParser - * @see nextflow.config.ConfigBase * - * * @author Paolo Di Tommaso */ @EqualsAndHashCode diff --git a/modules/nextflow/src/test/groovy/FunctionalTests.groovy b/modules/nextflow/src/test/groovy/FunctionalTests.groovy index 018690cfff..14318032cd 100644 --- a/modules/nextflow/src/test/groovy/FunctionalTests.groovy +++ b/modules/nextflow/src/test/groovy/FunctionalTests.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.exception.AbortRunException import nextflow.processor.TaskProcessor import nextflow.util.MemoryUnit @@ -80,7 +80,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -116,7 +116,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -155,7 +155,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -196,7 +196,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -255,7 +255,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -282,7 +282,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -308,7 +308,7 @@ class FunctionalTests extends Dsl2Spec { workflow { foo() } ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -337,7 +337,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -389,7 +389,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -415,7 +415,7 @@ class FunctionalTests extends Dsl2Spec { workflow { bar() } ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -448,7 +448,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -479,7 +479,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -509,7 +509,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -535,7 +535,7 @@ class FunctionalTests extends Dsl2Spec { workflow { foo() } ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -564,7 +564,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -594,7 +594,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -626,7 +626,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -656,7 +656,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -685,7 +685,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -716,7 +716,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() then: diff --git a/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy index b3781a92ee..ead6b64d59 100644 --- a/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy @@ -1784,7 +1784,7 @@ class ConfigBuilderTest extends Specification { def 'should collect config files' () { given: - def slurper = new ConfigParser() + def slurper = ConfigParserFactory.create() def file1 = Files.createTempFile('test1', null) def file2 = Files.createTempFile('test2', null) def result = new ConfigObject() diff --git a/modules/nextflow/src/test/groovy/nextflow/config/ConfigParserTest.groovy b/modules/nextflow/src/test/groovy/nextflow/config/parser/v1/ConfigParserV1Test.groovy similarity index 83% rename from modules/nextflow/src/test/groovy/nextflow/config/ConfigParserTest.groovy rename to modules/nextflow/src/test/groovy/nextflow/config/parser/v1/ConfigParserV1Test.groovy index a128f0fd36..43fbf761cd 100644 --- a/modules/nextflow/src/test/groovy/nextflow/config/ConfigParserTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/config/parser/v1/ConfigParserV1Test.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import java.nio.file.Files import java.nio.file.NoSuchFileException @@ -25,6 +25,8 @@ import com.sun.net.httpserver.HttpExchange import com.sun.net.httpserver.HttpHandler import com.sun.net.httpserver.HttpServer import nextflow.SysEnv +import nextflow.config.ConfigBuilder +import nextflow.config.ConfigClosurePlaceholder import nextflow.exception.ConfigParseException import nextflow.util.Duration import nextflow.util.MemoryUnit @@ -35,7 +37,7 @@ import spock.lang.Specification * * @author Paolo Di Tommaso */ -class ConfigParserTest extends Specification { +class ConfigParserV1Test extends Specification { def 'should get an environment variable' () { given: @@ -45,7 +47,7 @@ class ConfigParserTest extends Specification { def CONFIG = ''' process.cpus = env('MAX_CPUS') ''' - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: config.process.cpus == '1' @@ -61,16 +63,16 @@ class ConfigParserTest extends Specification { id 'foo' id 'bar' id 'bar' - } - + } + process { cpus = 1 - mem = 2 + mem = 2 } ''' when: - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: config.plugins == ['foo','bar'] as Set @@ -84,12 +86,12 @@ class ConfigParserTest extends Specification { profiles { plugins { id 'foo' - } + } } ''' when: - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: def e = thrown(ConfigParseException) @@ -129,7 +131,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().parse(text) + def config = new ConfigParserV1().parse(text) then: config.process.name == 'alpha' config.process.resources.cpus == 4 @@ -141,20 +143,20 @@ class ConfigParserTest extends Specification { when: def buffer = new StringWriter() config.writeTo(buffer) - def str = buffer.toString() + def str = buffer.toString().replaceAll('\t', ' ') then: str == ''' process { - name='alpha' - resources { - disk='1TB' - cpus=4 - memory='8GB' - nested { - foo=1 - bar=2 - } - } + name='alpha' + resources { + disk='1TB' + cpus=4 + memory='8GB' + nested { + foo=1 + bar=2 + } + } } '''.stripIndent().leftTrim() @@ -193,7 +195,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().setBinding().parse(text) + def config = new ConfigParserV1().setBinding().parse(text) then: config.params.xxx == 'x' config.params.yyy == 'y' @@ -251,7 +253,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().setBinding([MIN: 1, MAX: 32]).parse(main) + def config = new ConfigParserV1().setBinding([MIN: 1, MAX: 32]).parse(main) then: config.profiles.proc1.cpus == 4 config.profiles.proc1.memory == '8GB' @@ -308,7 +310,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().parse(main) + def config = new ConfigParserV1().parse(main) then: config.process.name == 'foo' config.process.resources.cpus == 4 @@ -372,8 +374,8 @@ class ConfigParserTest extends Specification { """ when: - def config1 = new ConfigParser() - .registerConditionalBlock('profiles','slow') + def config1 = new ConfigParserV1() + .setProfiles(['slow']) .parse(configText) then: config1.workDir == '/my/scratch' @@ -382,8 +384,8 @@ class ConfigParserTest extends Specification { config1.process.disk == '100GB' when: - def config2 = new ConfigParser() - .registerConditionalBlock('profiles','fast') + def config2 = new ConfigParserV1() + .setProfiles(['fast']) .parse(configText) then: config2.workDir == '/fast/scratch' @@ -409,77 +411,19 @@ class ConfigParserTest extends Specification { b = 2 } } - - servers { - local { - x = 1 - } - test { - y = 2 - } - prod { - z = 3 - } - } ''' when: - def slurper = new ConfigParser().registerConditionalBlock('profiles','alpha') - slurper.parse(text) - then: - slurper.getConditionalBlockNames() == ['alpha','beta'] as Set - - when: - slurper = new ConfigParser().registerConditionalBlock('profiles','omega') + def slurper = new ConfigParserV1().setProfiles(['alpha']) slurper.parse(text) then: - slurper.getConditionalBlockNames() == ['alpha','beta'] as Set - - when: - slurper = new ConfigParser().registerConditionalBlock('servers','xxx') - slurper.parse(text) - then: - slurper.getConditionalBlockNames() == ['local','test','prod'] as Set - - when: - slurper = new ConfigParser().registerConditionalBlock('foo','bar') - slurper.parse(text) - then: - slurper.getConditionalBlockNames() == [] as Set - } - - def 'should return the profile names' () { - given: - def text = ''' - profiles { - alpha { - a = 1 - } - beta { - b = 2 - } - } - - servers { - local { - x = 1 - } - test { - y = 2 - } - prod { - z = 3 - } - } - ''' + slurper.getProfiles() == ['alpha','beta'] as Set when: - def slurper = new ConfigParser() + slurper = new ConfigParserV1().setProfiles(['omega']) slurper.parse(text) then: - slurper.getProfileNames() == ['alpha','beta'] as Set - slurper.getConditionalBlockNames() == [] as Set - + slurper.getProfiles() == ['alpha','beta'] as Set } def 'should disable includeConfig parsing' () { @@ -494,12 +438,12 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().setIgnoreIncludes(true).parse(text) + def config = new ConfigParserV1().setIgnoreIncludes(true).parse(text) then: config.manifest.description == 'some text ..' when: - new ConfigParser().parse(text) + new ConfigParserV1().parse(text) then: thrown(NoSuchFileException) @@ -512,7 +456,7 @@ class ConfigParserTest extends Specification { configFile.text = 'XXX.enabled = true' when: - new ConfigParser().parse(configFile) + new ConfigParserV1().parse(configFile) then: noExceptionThrown() @@ -525,7 +469,7 @@ class ConfigParserTest extends Specification { given: ConfigObject result - def CONFIG = ''' + def CONFIG = ''' str1 = 'hello' str2 = "${str1} world" closure1 = { "$str" } @@ -533,7 +477,7 @@ class ConfigParserTest extends Specification { ''' when: - result = new ConfigParser() + result = new ConfigParserV1() .parse(CONFIG) then: result.str1 instanceof String @@ -542,7 +486,7 @@ class ConfigParserTest extends Specification { result.map1.bar instanceof Closure when: - result = new ConfigParser() + result = new ConfigParserV1() .setRenderClosureAsString(false) .parse(CONFIG) then: @@ -552,7 +496,7 @@ class ConfigParserTest extends Specification { result.map1.bar instanceof Closure when: - result = new ConfigParser() + result = new ConfigParserV1() .setRenderClosureAsString(true) .parse(CONFIG) then: @@ -567,19 +511,19 @@ class ConfigParserTest extends Specification { def 'should handle extend mem and duration units' () { ConfigObject result - def CONFIG = ''' + def CONFIG = ''' mem1 = 1.GB mem2 = 1_000_000.toMemory() mem3 = MemoryUnit.of(2_000) time1 = 2.hours time2 = 60_000.toDuration() time3 = Duration.of(120_000) - flag = 10000 < 1.GB + flag = 10000 < 1.GB ''' when: - result = new ConfigParser() - .parse(CONFIG) + result = new ConfigParserV1() + .parse(CONFIG) then: result.mem1 instanceof MemoryUnit result.mem1 == MemoryUnit.of('1 GB') @@ -615,11 +559,11 @@ class ConfigParserTest extends Specification { folder.resolve('conf/remote.config').text = ''' process { - cpus = 4 + cpus = 4 memory = '10GB' } ''' - + when: def url = 'http://localhost:9900/nextflow.config' as Path def cfg = new ConfigBuilder().buildGivenFiles(url) @@ -652,7 +596,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: config.params.foo.bar == 'bar1' @@ -670,7 +614,7 @@ class ConfigParserTest extends Specification { } ''' and: - config = new ConfigParser().parse(CONFIG) + config = new ConfigParserV1().parse(CONFIG) then: config.params.foo.bar == 'bar1' @@ -725,7 +669,7 @@ class ConfigParserTest extends Specification { """ when: - def config1 = new ConfigParser() + def config1 = new ConfigParserV1() .parse(configText) then: config1.workDir == '/my/scratch' @@ -739,6 +683,26 @@ class ConfigParserTest extends Specification { } + def 'should apply profiles in the order they were defined' () { + given: + def CONFIG = ''' + profiles { + foo { + params.input = 'foo' + } + + bar { + params.input = 'bar' + } + } + ''' + + when: + def config = new ConfigParserV1().setProfiles(['bar', 'foo']).parse(CONFIG) + + then: + config.params.input == 'bar' + } static class ConfigFileHandler implements HttpHandler { diff --git a/modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy b/modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy new file mode 100644 index 0000000000..78f4228fcf --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy @@ -0,0 +1,664 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.nio.file.Files +import java.nio.file.NoSuchFileException +import java.nio.file.Path + +import com.sun.net.httpserver.Headers +import com.sun.net.httpserver.HttpExchange +import com.sun.net.httpserver.HttpHandler +import com.sun.net.httpserver.HttpServer +import nextflow.SysEnv +import nextflow.config.ConfigBuilder +import nextflow.config.ConfigClosurePlaceholder +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import spock.lang.Specification + +/** + * + * @author Paolo Di Tommaso + */ +class ConfigParserV2Test extends Specification { + + def 'should get an environment variable' () { + given: + SysEnv.push(MAX_CPUS: '1') + + when: + def CONFIG = ''' + process.cpus = env('MAX_CPUS') + ''' + def config = new ConfigParserV2().parse(CONFIG) + + then: + config.process.cpus == '1' + + cleanup: + SysEnv.pop() + } + + def 'should parse plugin ids' () { + given: + def CONFIG = ''' + plugins { + id 'foo' + id 'bar' + id 'bar' + } + + process { + cpus = 1 + mem = 2 + } + ''' + + when: + def config = new ConfigParserV2().parse(CONFIG) + + then: + config.plugins == ['foo','bar'] as Set + and: + config.process.cpus == 1 + } + + def 'should parse composed config files' () { + + given: + def folder = File.createTempDir() + def snippet1 = new File(folder,'config1.txt').absoluteFile + def snippet2 = new File(folder,'config2.txt').absoluteFile + + def text = """ + process { + name = 'alpha' + resources { + disk = '1TB' + includeConfig "$snippet1" + } + } + """ + + snippet1.text = """ + cpus = 4 + memory = '8GB' + nested { + includeConfig("$snippet2") + } + """ + + snippet2.text = ''' + foo = 1 + bar = 2 + ''' + + when: + def config = new ConfigParserV2().parse(text) + then: + config.process.name == 'alpha' + config.process.resources.cpus == 4 + config.process.resources.memory == '8GB' + config.process.resources.disk == '1TB' + config.process.resources.nested.foo == 1 + config.process.resources.nested.bar == 2 + + when: + def buffer = new StringWriter() + config.writeTo(buffer) + def str = buffer.toString().replaceAll('\t', ' ') + then: + str == ''' + process { + name='alpha' + resources { + disk='1TB' + cpus=4 + memory='8GB' + nested { + foo=1 + bar=2 + } + } + } + '''.stripIndent().leftTrim() + + cleanup: + folder?.deleteDir() + + } + + def 'should parse include config using dot properties syntax' () { + + given: + def folder = File.createTempDir() + def snippet1 = new File(folder,'config1.txt').absoluteFile + def snippet2 = new File(folder,'config2.txt').absoluteFile + + def text = """ + process.name = 'alpha' + includeConfig "$snippet1" + """ + + snippet1.text = """ + params.xxx = 'x' + + process.cpus = 4 + process.memory = '8GB' + + includeConfig("$snippet2") + """ + + snippet2.text = ''' + params.yyy = 'y' + process { disk = '1TB' } + process.resources.foo = 1 + process.resources.bar = 2 + ''' + + when: + def config = new ConfigParserV2().setBinding().parse(text) + then: + config.params.xxx == 'x' + config.params.yyy == 'y' + config.process.name == 'alpha' + config.process.cpus == 4 + config.process.memory == '8GB' + config.process.disk == '1TB' + config.process.resources.foo == 1 + config.process.resources.bar == 2 + + cleanup: + folder?.deleteDir() + + } + + def 'should parse multiple relative files' () { + + given: + def folder = File.createTempDir() + def main = new File(folder, 'main.config') + def folder1 = new File(folder, 'dir1') + def folder2 = new File(folder, 'dir2') + def folder3 = new File(folder, 'dir3') + folder1.mkdirs() + folder2.mkdirs() + folder3.mkdirs() + + main. text = ''' + profiles { + includeConfig 'dir1/config' + includeConfig 'dir2/config' + includeConfig 'dir3/config' + } + ''' + + new File(folder,'dir1/config').text = ''' + proc1 { + cpus = 4 + memory = '8GB' + } + ''' + + new File(folder, 'dir2/config').text = ''' + proc2 { + cpus = MIN + memory = '6GB' + } + ''' + + new File(folder, 'dir3/config').text = ''' + proc3 { + cpus = MAX + disk = '500GB' + } + ''' + + when: + def config = new ConfigParserV2().setBinding([MIN: 1, MAX: 32]).parse(main) + then: + config.profiles.proc1.cpus == 4 + config.profiles.proc1.memory == '8GB' + config.profiles.proc2.cpus == 1 + config.profiles.proc2.memory == '6GB' + config.profiles.proc3.cpus == 32 + config.profiles.proc3.disk == '500GB' + + cleanup: + folder?.deleteDir() + + } + + def 'should parse nested relative files' () { + + given: + def folder = File.createTempDir() + def main = new File(folder, 'main.config') + def folder1 = new File(folder, 'dir1/dir3') + def folder2 = new File(folder, 'dir2') + folder1.mkdirs() + folder2.mkdirs() + + main. text = """ + process { + name = 'foo' + resources { + disk = '1TB' + includeConfig "dir1/nextflow.config" + } + + commands { + includeConfig "dir2/nextflow.config" + } + } + """ + + new File(folder,'dir1/nextflow.config').text = """ + cpus = 4 + memory = '8GB' + nested { + includeConfig 'dir3/nextflow.config' + } + """ + + new File(folder, 'dir1/dir3/nextflow.config').text = ''' + alpha = 1 + delta = 2 + ''' + + new File(folder, 'dir2/nextflow.config').text = ''' + cmd1 = 'echo true' + cmd2 = 'echo false' + ''' + + when: + def config = new ConfigParserV2().parse(main) + then: + config.process.name == 'foo' + config.process.resources.cpus == 4 + config.process.resources.memory == '8GB' + config.process.resources.disk == '1TB' + + config.process.resources.nested.alpha == 1 + config.process.resources.nested.delta == 2 + + config.process.commands.cmd1 == 'echo true' + config.process.commands.cmd2 == 'echo false' + + cleanup: + folder?.deleteDir() + + } + + def 'should load selected profile configuration' () { + + given: + def folder = File.createTempDir() + def snippet1 = new File(folder,'config1.txt').absoluteFile + def snippet2 = new File(folder,'config2.txt').absoluteFile + + snippet1.text = ''' + process { + cpus = 1 + memory = '2GB' + disk = '100GB' + } + ''' + + snippet2.text = ''' + process { + cpus = 8 + memory = '20GB' + disk = '2TB' + } + ''' + + def configText = """ + workDir = '/my/scratch' + + profiles { + + standard {} + + slow { + includeConfig "$snippet1" + } + + fast { + workDir = '/fast/scratch' + includeConfig "$snippet2" + } + + } + """ + + when: + def config1 = new ConfigParserV2() + .setProfiles(['slow']) + .parse(configText) + then: + config1.workDir == '/my/scratch' + config1.process.cpus == 1 + config1.process.memory == '2GB' + config1.process.disk == '100GB' + + when: + def config2 = new ConfigParserV2() + .setProfiles(['fast']) + .parse(configText) + then: + config2.workDir == '/fast/scratch' + config2.process.cpus == 8 + config2.process.memory == '20GB' + config2.process.disk == '2TB' + + + cleanup: + folder.deleteDir() + + } + + def 'should return the set of parsed profiles' () { + + given: + def text = ''' + profiles { + alpha { + a = 1 + } + beta { + b = 2 + } + } + ''' + + when: + def slurper = new ConfigParserV2().setProfiles(['alpha']) + slurper.parse(text) + then: + slurper.getProfiles() == ['alpha','beta'] as Set + + when: + slurper = new ConfigParserV2().setProfiles(['omega']) + slurper.parse(text) + then: + slurper.getProfiles() == ['alpha','beta'] as Set + } + + def 'should ignore config includes when specified' () { + given: + def text = ''' + manifest { + description = 'some text ..' + } + + includeConfig 'this' + includeConfig 'that' + ''' + + when: + def config = new ConfigParserV2().setIgnoreIncludes(true).parse(text) + then: + config.manifest.description == 'some text ..' + + when: + new ConfigParserV2().parse(text) + then: + thrown(NoSuchFileException) + + } + + def 'should parse file named as a top config scope' () { + given: + def folder = File.createTempDir() + def configFile = new File(folder, 'XXX.config') + configFile.text = 'XXX.enabled = true' + + when: + new ConfigParserV2().parse(configFile) + then: + noExceptionThrown() + + cleanup: + folder?.deleteDir() + } + + def 'should access node metadata' () { + + given: + Map params + def CONFIG = ''' + params.str1 = 'hello' + params.str2 = "${params.str1} world" + params.closure1 = { "$str" } + params.map1 = [foo: 'hello', bar: { world }] + ''' + + when: + params = new ConfigParserV2() + .parse(CONFIG) + .params + then: + params.str1 instanceof String + params.str2 instanceof GString + params.closure1 instanceof Closure + params.map1.bar instanceof Closure + + when: + params = new ConfigParserV2() + .setRenderClosureAsString(false) + .parse(CONFIG) + .params + then: + params.str1 instanceof String + params.str2 instanceof GString + params.closure1 instanceof Closure + params.map1.bar instanceof Closure + + when: + params = new ConfigParserV2() + .setRenderClosureAsString(true) + .parse(CONFIG) + .params + then: + params.str1 == 'hello' + params.str2 == 'hello world' + params.closure1 instanceof ConfigClosurePlaceholder + params.closure1 == new ConfigClosurePlaceholder('{ "$str" }') + params.map1.foo == 'hello' + params.map1.bar == new ConfigClosurePlaceholder('{ world }') + + } + + def 'should handle extend mem and duration units' () { + ConfigObject result + def CONFIG = ''' + mem1 = 1.GB + mem2 = 1_000_000.toMemory() + mem3 = MemoryUnit.of(2_000) + time1 = 2.hours + time2 = 60_000.toDuration() + time3 = Duration.of(120_000) + flag = 10000 < 1.GB + ''' + + when: + result = new ConfigParserV2() + .parse(CONFIG) + then: + result.mem1 instanceof MemoryUnit + result.mem1 == MemoryUnit.of('1 GB') + result.mem2 == MemoryUnit.of(1_000_000) + result.mem3 == MemoryUnit.of(2_000) + result.time1 instanceof Duration + result.time1 == Duration.of('2 hours') + result.time2 == Duration.of(60_000) + result.time3 == Duration.of(120_000) + result.flag == true + } + + def 'should parse a config from an http server' () { + given: + def folder = Files.createTempDirectory('test') + folder.resolve('conf').mkdir() + + HttpServer server = HttpServer.create(new InetSocketAddress(9900), 0); + server.createContext("/", new ConfigFileHandler(folder)); + server.start() + + folder.resolve('nextflow.config').text = ''' + includeConfig 'conf/base.config' + includeConfig 'http://localhost:9900/conf/remote.config' + ''' + + folder.resolve('conf/base.config').text = ''' + params.foo = 'Hello' + params.bar = 'world!' + ''' + + folder.resolve('conf/remote.config').text = ''' + process { + cpus = 4 + memory = '10GB' + } + ''' + + when: + def url = 'http://localhost:9900/nextflow.config' as Path + def cfg = new ConfigBuilder().buildGivenFiles(url) + then: + cfg.params.foo == 'Hello' + cfg.params.bar == 'world!' + cfg.process.cpus == 4 + cfg.process.memory == '10GB' + + cleanup: + server?.stop(0) + folder?.deleteDir() + } + + def 'should not overwrite param values with nested values of with the same name' () { + given: + def CONFIG = ''' + params { + foo { + bar = 'bar1' + } + baz = 'baz1' + nested { + baz = 'baz2' + foo { + bar = 'bar2' + } + } + } + ''' + + when: + def config = new ConfigParserV2().parse(CONFIG) + + then: + config.params.foo.bar == 'bar1' + config.params.baz == 'baz1' + config.params.nested.baz == 'baz2' + config.params.nested.foo.bar == 'bar2' + + when: + CONFIG = ''' + params { + foo.bar = 'bar1' + baz = 'baz1' + nested.baz = 'baz2' + nested.foo.bar = 'bar2' + } + ''' + and: + config = new ConfigParserV2().parse(CONFIG) + + then: + config.params.foo.bar == 'bar1' + config.params.baz == 'baz1' + config.params.nested.baz == 'baz2' + config.params.nested.foo.bar == 'bar2' + } + + def 'should apply profiles in the order they are specified at runtime' () { + given: + def CONFIG = ''' + profiles { + foo { + params.input = 'foo' + } + + bar { + params.input = 'bar' + } + } + ''' + + when: + def config = new ConfigParserV2().setProfiles(['bar', 'foo']).parse(CONFIG) + + then: + config.params.input == 'foo' + } + + def 'should allow mixed use of dot and block syntax in a profile' () { + given: + def CONFIG = ''' + profiles { + foo { + process.memory = '2 GB' + process { + cpus = 2 + } + } + } + ''' + + when: + def config = new ConfigParserV2().setProfiles(['foo']).parse(CONFIG) + + then: + config.process.memory == '2 GB' + config.process.cpus == 2 + } + + static class ConfigFileHandler implements HttpHandler { + + Path folder + + ConfigFileHandler(Path folder) { + this.folder = folder + } + + void handle(HttpExchange request) throws IOException { + def path = request.requestURI.toString().substring(1) + def file = folder.resolve(path) + + Headers header = request.getResponseHeaders() + header.add("Content-Type", "text/plain") + request.sendResponseHeaders(200, file.size()) + + OutputStream os = request.getResponseBody(); + os.write(file.getBytes()); + os.close(); + } + } + +} diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy index 682337368d..bd2dc3e4bb 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy @@ -17,7 +17,7 @@ package nextflow.script import groovyx.gpars.dataflow.DataflowVariable -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.exception.AbortRunException import nextflow.exception.ProcessUnrecoverableException import nextflow.processor.TaskProcessor @@ -330,7 +330,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -378,7 +378,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -413,7 +413,7 @@ class ScriptRunnerTest extends Dsl2Spec { workflow { hola() } ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -451,7 +451,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -482,7 +482,7 @@ class ScriptRunnerTest extends Dsl2Spec { workflow { hola() } ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -530,7 +530,7 @@ class ScriptRunnerTest extends Dsl2Spec { } ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session) @@ -684,7 +684,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session).setScript(script).execute() @@ -721,7 +721,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session).setScript(script).execute() @@ -759,7 +759,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session).setScript(script).execute() diff --git a/tests/config-labels.included b/tests/config-labels-included.config similarity index 100% rename from tests/config-labels.included rename to tests/config-labels-included.config diff --git a/tests/config-labels.config b/tests/config-labels.config index 294fa3900a..51af5f4f80 100644 --- a/tests/config-labels.config +++ b/tests/config-labels.config @@ -32,6 +32,6 @@ profiles { } test3 { - includeConfig 'config-labels.included' + includeConfig 'config-labels-included.config' } } diff --git a/tests/config-vars.config b/tests/config-vars.config index a5bafc5237..f8af6330e8 100644 --- a/tests/config-vars.config +++ b/tests/config-vars.config @@ -18,19 +18,19 @@ * author Emilio Palumbo */ -l = [ - a: [1,2], - b: [3,4] -] +params { + a = [1,2] + b = [3,4] +} process { withName: foo { - ext.out = { l.a } + ext.out = { params.a } } withName: bar { - ext.out = { l.b } + ext.out = { params.b } } } diff --git a/tests/profiles.config b/tests/profiles.config index b74a3f87ef..95f0ac4ee9 100644 --- a/tests/profiles.config +++ b/tests/profiles.config @@ -16,8 +16,7 @@ echo = true -def x = 'delta' -includeConfig "${x}.config" +includeConfig "${'delta'}.config" profiles { diff --git a/validation/test.sh b/validation/test.sh index 7de964312a..b3c5d29fbe 100755 --- a/validation/test.sh +++ b/validation/test.sh @@ -11,11 +11,7 @@ export NXF_CMD=${NXF_CMD:-$(get_abs_filename ../launch.sh)} export NXF_ANSI_LOG=false export NXF_DISABLE_CHECK_LATEST=true -# -# Integration tests -# -if [[ $TEST_MODE == 'test_integration' ]]; then - +test_integration() { ( cd ../tests/ sudo bash cleanup.sh @@ -47,6 +43,21 @@ if [[ $TEST_MODE == 'test_integration' ]]; then $NXF_CMD run nextflow-io/rnaseq-nf -with-docker $OPTS -resume exit 0 +} + +# +# Integration tests +# +if [[ $TEST_MODE == 'test_integration' ]]; then + test_integration +fi + +# +# Integration tests (strict syntax) +# +if [[ $TEST_MODE == 'test_parser_v2' ]]; then + export NXF_SYNTAX_PARSER=v2 + test_integration fi # From 01db536cff93c041bda89fa7956440c1bd9c07ca Mon Sep 17 00:00:00 2001 From: Kevin Date: Tue, 25 Feb 2025 13:54:33 -0500 Subject: [PATCH 09/11] Fix broken link in docs (#5815) Signed-off-by: Kevin Galens --- docs/updating-syntax.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/updating-syntax.md b/docs/updating-syntax.md index 10b48e8033..ede255b773 100644 --- a/docs/updating-syntax.md +++ b/docs/updating-syntax.md @@ -550,6 +550,6 @@ There are two ways to preserve Groovy code: - Move the code to the `lib` directory - Create a plugin -Any Groovy code can be moved into the `lib` directory, which supports the full Groovy language. This approach is useful for temporarily preserving some Groovy code until it can be updated later and incorporated into a Nextflow script. See {ref}`` documentation for more information. +Any Groovy code can be moved into the `lib` directory, which supports the full Groovy language. This approach is useful for temporarily preserving some Groovy code until it can be updated later and incorporated into a Nextflow script. See {ref}`lib-directory` documentation for more information. For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`plugins-dev-page` for more information on how to develop plugins. From d8c8151bc58b41fd991ae48965071fb0d1b96c2b Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 25 Feb 2025 14:28:11 -0600 Subject: [PATCH 10/11] Update issue and PR templates (#5805) Signed-off-by: Ben Sherman --- .github/ISSUE_TEMPLATE/bug_report.md | 11 +++++------ .github/ISSUE_TEMPLATE/general_question.md | 15 ++------------- .github/ISSUE_TEMPLATE/new_feature.md | 16 +++++----------- .../pull_request_template.md | 13 ++----------- 4 files changed, 14 insertions(+), 41 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 5dd6c73320..be5f4c13a7 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,24 +1,23 @@ --- name: Bug report -about: Create a report to help us improve +about: Report a bug to help us improve --- ## Bug report -(Please follow this template replacing the text between parentheses with the requested information) +(Please follow this template by replacing the text between parentheses with the requested information) ### Expected behavior and actual behavior -(Give an brief description of the expected behavior -and actual behavior) +(Give a brief description of the expected behavior and actual behavior) ### Steps to reproduce the problem -(Provide a test case that reproduce the problem either with a self-contained script or GitHub repository) +(Provide a test case that reproduces the problem either with a self-contained script or GitHub repository) ### Program output -(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the `.nextflow.log` file.) +(Copy and paste the output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the `.nextflow.log` file.) ### Environment diff --git a/.github/ISSUE_TEMPLATE/general_question.md b/.github/ISSUE_TEMPLATE/general_question.md index 342c101efc..02546a28f8 100644 --- a/.github/ISSUE_TEMPLATE/general_question.md +++ b/.github/ISSUE_TEMPLATE/general_question.md @@ -1,19 +1,8 @@ --- name: General question -about: Need for help on Nextflow language and usage +about: Ask for help with Nextflow language and usage --- Hi! Thanks for using Nextflow. -If you need help about Nextflow scripting language, -configuration options and general Nextflow usage the better -channels to post this kind of questions are: - -* GitHub discussions: https://github.com/nextflow-io/nextflow/discussions -* Slack community chat: https://www.nextflow.io/slack-invite.html - - -Also you may also want to have a look at the patterns page -for common solutions to recurrent implementation problems: -http://nextflow-io.github.io/patterns/index.html - +If you need help using or developing Nextflow pipelines, the best place to ask questions is the [community forum](https://community.seqera.io/). diff --git a/.github/ISSUE_TEMPLATE/new_feature.md b/.github/ISSUE_TEMPLATE/new_feature.md index 9c0b9d007b..9de805a49d 100644 --- a/.github/ISSUE_TEMPLATE/new_feature.md +++ b/.github/ISSUE_TEMPLATE/new_feature.md @@ -1,22 +1,16 @@ --- name: New feature -about: Submit a new feature proposal +about: Propose a new feature or enhancement --- ## New feature -Hi! Thanks for using Nextflow and submitting the proposal -for a new feature or the enhancement of an existing functionality. +(Hi! Thanks for using Nextflow and for proposing a new feature or enhancement. Please replace this text with a brief description of your proposal.) -Please replace this text providing a short description of your -proposal. +## Use case -## Usage scenario +(What's the main use case and deployment scenario addressed by this proposal) -(What's the main usage case and the deployment scenario addressed by this proposal) - -## Suggest implementation +## Suggested implementation (Highlight the main building blocks of a possible implementation and/or related components) - - diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md index 363896fd8b..5bd4dd2e00 100644 --- a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md +++ b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md @@ -1,16 +1,7 @@ -Hi! Thanks for contributing to Nextflow project. +Hi! Thanks for contributing to Nextflow. -When submitting a Pull Request please make sure to not include -in the changeset any modification in these files: - -* `nextflow` -* `modules/nf-commons/src/main/nextflow/Const.groovy` - -Also, please sign-off the DCO [1] to certify you are the author of the contribution -and you adhere to Nextflow open source license [2] adding a `Signed-off-by` line to -the contribution commit message. For more details check [3]. +When submitting a Pull Request, please sign-off the DCO [1] to certify that you are the author of the contribution and you adhere to Nextflow's open source license [2] by adding a `Signed-off-by` line to the contribution commit message. See [3] for more details. 1. https://developercertificate.org/ 2. https://github.com/nextflow-io/nextflow/blob/master/COPYING 3. https://github.com/apps/dco - From ab358dc945e525f0c6bd7e41eaf927e1802d34ba Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 27 Feb 2025 08:17:17 -0600 Subject: [PATCH 11/11] Fail if glob does not exist with `checkIfExists: true` (#4224) Signed-off-by: Ben Sherman --- modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy | 6 +++++- .../nextflow/src/test/groovy/nextflow/NextflowTest.groovy | 6 ++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy b/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy index e7dc15884c..aee6368755 100644 --- a/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy @@ -135,7 +135,11 @@ class Nextflow { } // revolve the glob pattern returning all matches - return fileNamePattern(splitter, options) + final result = fileNamePattern(splitter, options) + if( glob && options?.checkIfExists && result.isEmpty() ) + throw new NoSuchFileException(path.toString()) + + return result } static files( Map options=null, def path ) { diff --git a/modules/nextflow/src/test/groovy/nextflow/NextflowTest.groovy b/modules/nextflow/src/test/groovy/nextflow/NextflowTest.groovy index f49db65779..4eda38c9a9 100644 --- a/modules/nextflow/src/test/groovy/nextflow/NextflowTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/NextflowTest.groovy @@ -348,6 +348,12 @@ class NextflowTest extends Specification { def e = thrown(NoSuchFileException) e.message == foo.toString() + when: + Nextflow.file("$folder/*.txt", checkIfExists: true) + then: + e = thrown(NoSuchFileException) + e.message == "$folder/*.txt" + cleanup: folder?.deleteDir() }