Code

Job arrays#3892

Code

Merged

pditommaso merged 113 commits intomasternextflow-io/nextflow:masterfrom 1477-job-array-executornextflow-io/nextflow:1477-job-array-executorCopy head branch name to clipboardMay 9, 2024

+2,068-280

Commits

Commits on Apr 21, 2023

• Add initial array executor

  Show description for 41ac662
  bentsherman

  committed
  41ac662

  Copy full SHA for 41ac662

• Add support for SLURM array jobs

  Show description for a0900ae
  bentsherman

  committed
  a0900ae

  Copy full SHA for a0900ae

• Fix failing test

  Show description for b5bad00
  bentsherman

  committed
  b5bad00

  Copy full SHA for b5bad00

• Document array executor

  Show description for 713e33e
  bentsherman

  committed
  713e33e

  Copy full SHA for 713e33e

• Use concurrent queues in array executor, add fallback for leftover tasks

  Show description for 76de0d4
  bentsherman

  committed
  76de0d4

  Copy full SHA for 76de0d4

• Move batching logic to array task polling monitor

  Show description for 1ef005c
  bentsherman

  committed
  1ef005c

  Copy full SHA for 1ef005c

• Cache status checks in array task handler

  Show description for 27728eb
  bentsherman

  committed
  27728eb

  Copy full SHA for 27728eb

Commits on Apr 24, 2023

• Finalize support for SLURM array jobs

  Show description for 343b331
  bentsherman

  committed
  343b331

  Copy full SHA for 343b331

Commits on Apr 25, 2023

• Merge branch 'master' into 1477-job-array-executor [ci skip]

  Show description for a70e6b4
  bentsherman

  committed
  a70e6b4

  Copy full SHA for a70e6b4

• Refactor array executor as process directive

  Show description for e5ca280
  bentsherman

  committed
  e5ca280

  Copy full SHA for e5ca280

• minor edits

  Show description for ed5089d
  bentsherman

  committed
  ed5089d

  Copy full SHA for ed5089d

• Support array jobs for LSF, PBS, SGE

  Show description for 5b36129
  bentsherman

  committed
  5b36129

  Copy full SHA for 5b36129

Commits on Apr 26, 2023

• Replace synchronized with atomics and locks

  Show description for d162948
  bentsherman

  committed
  d162948

  Copy full SHA for d162948

Commits on Apr 30, 2023

• Merge branch 'master' into 1477-job-array-executor

  pditommaso

  authored
  3e38ec0

  Copy full SHA for 3e38ec0

• Add array job support to AWS Batch

  Show description for 99e92d5
  bentsherman

  committed
  99e92d5

  Copy full SHA for 99e92d5

• Merge branch '1477-job-array-executor' of github.com:nextflow-io/nextflow into 1477-job-array-executor

  bentsherman

  committed
  33502e7

  Copy full SHA for 33502e7

• Fix bug in awsbatch + fusion + array job

  Show description for 75135d7
  bentsherman

  committed
  75135d7

  Copy full SHA for 75135d7

• Support grid + fusion + array jobs

  Show description for 2ee42ae
  bentsherman

  committed
  2ee42ae

  Copy full SHA for 2ee42ae

Commits on May 1, 2023

• Refactor ArrayTask* to TaskArray*

  Show description for 3d84594
  bentsherman

  committed
  3d84594

  Copy full SHA for 3d84594

• Reduce code duplication

  Show description for fc78470
  bentsherman

  committed
  fc78470

  Copy full SHA for fc78470

• minor edits

  Show description for 45a1c35
  bentsherman

  committed
  45a1c35

  Copy full SHA for 45a1c35

• Fix failing tests

  Show description for 16af92f
  bentsherman

  committed
  16af92f

  Copy full SHA for 16af92f

Commits on May 2, 2023

• Reduce code duplication

  Show description for 98cfb0e
  bentsherman

  committed
  98cfb0e

  Copy full SHA for 98cfb0e

• Reduce code duplication

  Show description for f02de40
  bentsherman

  committed
  f02de40

  Copy full SHA for f02de40

• Merge branch 'master' into 1477-job-array-executor

  abhi18av

  authored
  eac8099

  Copy full SHA for eac8099

• minor edits

  Show description for 603913f
  bentsherman

  committed
  603913f

  Copy full SHA for 603913f

• minor edits

  Show description for fe3a3d7
  bentsherman

  committed
  fe3a3d7

  Copy full SHA for fe3a3d7

• Use TaskRun subclass to submit array job in a generic manner

  Show description for aaaa35d
  bentsherman

  committed
  aaaa35d

  Copy full SHA for aaaa35d

Commits on May 3, 2023

• Minor changes

  Show description for 6945450
  bentsherman

  committed
  6945450

  Copy full SHA for 6945450

• Fix bugs in array submit script

  Show description for f95c7d0
  bentsherman

  committed
  f95c7d0

  Copy full SHA for f95c7d0

• Fix Fusion support

  Show description for 44f6825
  bentsherman

  committed
  44f6825

  Copy full SHA for 44f6825

• Replace array submitter with array handling logic in the task polling monitor

  Show description for 12612a3
  bentsherman

  committed
  12612a3

  Copy full SHA for 12612a3

• Fix task reporting in log observer

  Show description for b9dafe0
  bentsherman

  committed
  b9dafe0

  Copy full SHA for b9dafe0

• Fix missing method error

  Show description for ffc6d31
  bentsherman

  committed
  ffc6d31

  Copy full SHA for ffc6d31

Commits on May 7, 2023

• Merge branch 'master' into 1477-job-array-executor

  abhi18av

  authored
  5a9f6de

  Copy full SHA for 5a9f6de

Commits on May 24, 2023

• Add support for AWS SSE env variables

  Show description for 7ec397f
  pditommaso

  committed
  7ec397f

  Copy full SHA for 7ec397f

Commits on May 26, 2023

• Add support for google batch

  Show description for 324681b
  bentsherman

  committed
  324681b

  Copy full SHA for 324681b

• Merge branch 'master' into 1477-job-array-executor

  Show description for eb0750f
  bentsherman

  committed
  eb0750f

  Copy full SHA for eb0750f

• Fix static compilation errors

  Show description for d3d0377
  bentsherman

  committed
  d3d0377

  Copy full SHA for d3d0377

• Delete entire AWS Batch array job on workflow termination

  Show description for acea09f
  bentsherman

  committed
  acea09f

  Copy full SHA for acea09f

Commits on May 29, 2023

• Merge branch 'master' into 1477-job-array-executor

  Show description for 91b61a5
  bentsherman

  committed
  91b61a5

  Copy full SHA for 91b61a5

• Fix failing test

  Show description for d78daa8
  bentsherman

  committed
  d78daa8

  Copy full SHA for d78daa8

Commits on May 31, 2023

• Fix issues with google batch array jobs

  Show description for 75dc404
  bentsherman

  committed
  75dc404

  Copy full SHA for 75dc404

• Rename getSubmitCommand() to getLaunchCommand()

  Show description for 7c7a7b3
  bentsherman

  committed
  7c7a7b3

  Copy full SHA for 7c7a7b3

• Fix SLURM+Fusion+array jobs, cleanup

  Show description for 52f3195
  bentsherman

  committed
  52f3195

  Copy full SHA for 52f3195

• Remove unrelated changes

  Show description for c5e8b76
  bentsherman

  committed
  c5e8b76

  Copy full SHA for c5e8b76

• Submit retried tasks directly

  Show description for 14b2455
  bentsherman

  committed
  14b2455

  Copy full SHA for 14b2455

Commits on Jun 1, 2023

• Add unit tests

  Show description for 0de3c84
  bentsherman

  committed
  0de3c84

  Copy full SHA for 0de3c84

Commits on Jun 13, 2023

• Update docs [ci fast]

  Show description for 55d7941
  bentsherman

  committed
  55d7941

  Copy full SHA for 55d7941

• Fix issues with LSF array jobs

  Show description for eeeb12d
  bentsherman

  committed
  eeeb12d

  Copy full SHA for eeeb12d

Commits on Jun 14, 2023

• Add array index name to env whitelist

  Show description for fd10361
  bentsherman

  committed
  fd10361

  Copy full SHA for fd10361

Commits on Jun 15, 2023

• Disable non-native container run for task array

  Show description for 782dea2
  bentsherman

  committed
  782dea2

  Copy full SHA for 782dea2

Commits on Jun 21, 2023

• Fix incorrect index offset in array job script

  Show description for eb45069
  bentsherman

  committed
  eb45069

  Copy full SHA for eb45069

Commits on Jun 22, 2023

• Change SGE array index start to 1

  Show description for 50c300c
  bentsherman

  committed
  50c300c

  Copy full SHA for 50c300c

Commits on Jun 23, 2023

• minor edits

  Show description for c206a5c
  bentsherman

  committed
  c206a5c

  Copy full SHA for c206a5c

Commits on Jul 13, 2023

• Change SGE array index start to 1 (for real this time)

  Show description for d4ab783
  bentsherman

  committed
  d4ab783

  Copy full SHA for d4ab783

Commits on Jul 27, 2023

• Use Bolts.withLock

  Show description for 786a0b3
  bentsherman

  committed
  786a0b3

  Copy full SHA for 786a0b3

Commits on Sep 15, 2023

• Add array jobs to CRG executor

  Show description for 68ea23f
  bentsherman

  committed
  68ea23f

  Copy full SHA for 68ea23f

• Add array jobs to CRG executor

  Show description for 28dbe47
  bentsherman

  committed
  28dbe47

  Copy full SHA for 28dbe47

Commits on Sep 18, 2023

• Add array job to SGE parseJobId

  Show description for 6c7494a
  bentsherman

  committed
  6c7494a

  Copy full SHA for 6c7494a

Commits on Jan 24, 2024

• Merge branch 'master' into 1477-job-array-executor

  Show description for dfca759
  bentsherman

  committed
  dfca759

  Copy full SHA for dfca759

• Refactor "array job" -> "job array"

  Show description for 1929773
  bentsherman

  committed
  1929773

  Copy full SHA for 1929773

Commits on Apr 21, 2024

• Merge branch 'master' into 1477-job-array-executor

  Show description for fcdd6b7
  pditommaso

  committed
  fcdd6b7

  Copy full SHA for fcdd6b7

• Fix intellij compiler errors [ci fast]

  Show description for 50c8f6f
  pditommaso

  committed
  50c8f6f

  Copy full SHA for 50c8f6f

Commits on Apr 22, 2024

• Apply suggestions from review

  Show description for c5a12ec
  bentsherman

  committed
  c5a12ec

  Copy full SHA for c5a12ec

Commits on Apr 23, 2024

• Fix failing tests

  Show description for 9897a6c
  bentsherman

  committed
  9897a6c

  Copy full SHA for 9897a6c

• Stability improvements

  Show description for c15e5ec
  bentsherman

  committed
  c15e5ec

  Copy full SHA for c15e5ec

Commits on Apr 24, 2024

• Improve construction of job array

  Show description for ba176e0
  bentsherman

  committed
  ba176e0

  Copy full SHA for ba176e0

• Add process directives to wrapper script for Fusion

  Show description for 584c961
  bentsherman

  committed
  584c961

  Copy full SHA for 584c961

Commits on Apr 26, 2024

• Merge branch 'master' into 1477-job-array-executor

  pditommaso

  committed
  7d8aa5c

  Copy full SHA for 7d8aa5c

• Prefer try/finally idiom to avoid closure [ci fast]

  Show description for 49f7fbd
  pditommaso

  committed
  49f7fbd

  Copy full SHA for 49f7fbd

• Apply suggestions from review

  Show description for 99a1ec3
  bentsherman

  committed
  99a1ec3

  Copy full SHA for 99a1ec3

Commits on Apr 29, 2024

• Fix bug with job array context

  Show description for bf04e3f
  bentsherman

  committed
  bf04e3f

  Copy full SHA for bf04e3f

Commits on May 1, 2024

• Merge branch 'master' into 1477-job-array-executor

  pditommaso

  committed
  67f76a1

  Copy full SHA for 67f76a1

• Add integration tests

  Show description for 16dfb25
  pditommaso

  committed
  16dfb25

  Copy full SHA for 16dfb25

• Restore private method access (#4961) [ci fast]

  Show description for 0e88bf7
  pditommaso

  authored
  0e88bf7

  Copy full SHA for 0e88bf7

• Minor changes

  Show description for 11aeaae
  pditommaso

  committed
  11aeaae

  Copy full SHA for 11aeaae

• Add aws batch integration test

  Show description for 519c93a
  pditommaso

  committed
  519c93a

  Copy full SHA for 519c93a

• Revert re-use of NXF_CHDIR in job array script

  Show description for ade72e2
  bentsherman

  committed
  ade72e2

  Copy full SHA for ade72e2

• Add unit tests for grid executors

  Show description for 8c13ec3
  bentsherman

  committed
  8c13ec3

  Copy full SHA for 8c13ec3

• Add unit test for crg executor

  Show description for b7e829d
  bentsherman

  committed
  b7e829d

  Copy full SHA for b7e829d

• Fix TaskArrayRun access to private methods

  Show description for 6f6e5a8
  pditommaso

  committed
  6f6e5a8

  Copy full SHA for 6f6e5a8

Commits on May 2, 2024

• Fix typ

  Show description for 4bb0e98
  pditommaso

  committed
  4bb0e98

  Copy full SHA for 4bb0e98

• Add integration tests

  Show description for beddbbd
  pditommaso

  committed
  beddbbd

  Copy full SHA for beddbbd

• Use idiomatic name for task array dir var

  Show description for 1713167
  pditommaso

  committed
  1713167

  Copy full SHA for 1713167

• Merge branch 'master' into 1477-job-array-executor

  pditommaso

  committed
  5137589

  Copy full SHA for 5137589

• Use TaskArrayRun as return type

  Show description for bf859bd
  pditommaso

  committed
  bf859bd

  Copy full SHA for bf859bd

• Promote status update methods private

  Show description for a4518a3
  pditommaso

  committed
  a4518a3

  Copy full SHA for a4518a3

• Minor change

  Show description for 56eb754
  pditommaso

  committed
  56eb754

  Copy full SHA for 56eb754

• Refactor task name in task meta comment

  Show description for 8afc6a9
  pditommaso

  committed
  8afc6a9

  Copy full SHA for 8afc6a9

Commits on May 3, 2024

• Force new test

  Show description for b42ccbb
  pditommaso

  committed
  b42ccbb

  Copy full SHA for b42ccbb

• Disable job log when submitting job arrays to grid schedulers

  Show description for 7e411fd
  bentsherman

  committed
  7e411fd

  Copy full SHA for 7e411fd

• Move misplaced test

  Show description for a77cfd5
  bentsherman

  committed
  a77cfd5

  Copy full SHA for a77cfd5

Commits on May 4, 2024

• Fix collectFile saving to GCS with sort: false (#4965)

  Show description for 653064f
  

  bentsherman

  authored and

  pditommaso

  committed
  653064f

  Copy full SHA for 653064f

• Fix script error text alignment (#4681)

  Show description for ea1cc2c
  

  2 peopleauthored and

  pditommaso

  committed
  ea1cc2c

  Copy full SHA for ea1cc2c

• Use for instead eachLine in error formatting [ci fast]

  Show description for 7367687
  pditommaso

  committed
  7367687

  Copy full SHA for 7367687

• Update aws.md to include Cluster access (#4951) [ci skip]

  Show description for be536f8
  

  Kartstig

  authored and

  pditommaso

  committed
  be536f8

  Copy full SHA for be536f8

• Add Wave and Fusion info to workflow metadata (#4945)

  Show description for d899ebc
  

  3 peoplecommitted
  d899ebc

  Copy full SHA for d899ebc

• Strip auth secret from logs

  Show description for 11d4a87
  pditommaso

  committed
  11d4a87

  Copy full SHA for 11d4a87

Commits on May 5, 2024

• [ci skip] empty

  Show description for a9d5d86
  pditommaso

  committed
  a9d5d86

  Copy full SHA for a9d5d86

Commits on May 6, 2024

• Job array refactor (#4973)

  Show description for 174bdfb
  pditommaso

  authored
  174bdfb

  Copy full SHA for 174bdfb

• Merge branch 'master' into 1477-job-array-executor

  Show description for 12656bc
  bentsherman

  authored
  12656bc

  Copy full SHA for 12656bc

Commits on May 7, 2024

• Strengthen aws batch deletion logi

  Show description for 4e41436
  pditommaso

  committed
  4e41436

  Copy full SHA for 4e41436

• Strengthen google batch deletion logic

  Show description for e8b701d
  pditommaso

  committed
  e8b701d

  Copy full SHA for e8b701d

• Merge branch 'master' into 1477-job-array-executor

  pditommaso

  authored
  3291f6c

  Copy full SHA for 3291f6c

• minor edits

  Show description for b7ce0a1
  bentsherman

  committed
  b7ce0a1

  Copy full SHA for b7ce0a1

• Update google batch logging to select task logs

  Show description for 3e29adc
  bentsherman

  committed
  3e29adc

  Copy full SHA for 3e29adc

• Fix race condition in LogsCheckpoint

  Show description for 247b721
  bentsherman

  committed
  247b721

  Copy full SHA for 247b721

Commits on May 8, 2024

• Revert "Fix race condition in LogsCheckpoint"

  Show description for e086674
  pditommaso

  committed
  e086674

  Copy full SHA for e086674

• Resolve conflicts

  Show description for cbbbc46
  pditommaso

  committed
  cbbbc46

  Copy full SHA for cbbbc46

Commits on May 9, 2024

• Restore grid handler names [ci fast]

  Show description for 6b8bb49
  pditommaso

  committed
  6b8bb49

  Copy full SHA for 6b8bb49

• Revert submit naming

  Show description for 912f90d
  pditommaso

  committed
  912f90d

  Copy full SHA for 912f90d

• Revert volatile change

  Show description for 4b45aa0
  pditommaso

  committed
  4b45aa0

  Copy full SHA for 4b45aa0