From 8e03b8312b9a3ff73ab6702fb51e36174e66a6ae Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Mon, 3 Mar 2025 12:51:53 -0600 Subject: [PATCH 1/3] Document plugin filesystems Signed-off-by: Ben Sherman --- docs/developer/plugins.md | 32 ++++++++++++++++++++++++++++++++ docs/working-with-files.md | 2 ++ 2 files changed, 34 insertions(+) diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md index 98044a4ee1..d48459d009 100644 --- a/docs/developer/plugins.md +++ b/docs/developer/plugins.md @@ -151,6 +151,38 @@ process foo { Refer to the source code of Nextflow's built-in executors to see how to implement the various components of an executor. You might be able to implement most of your executor by simply reusing existing code. ::: +### Filesystems + +Plugins can define custom filesystems, which are used by Nextflow to interact with external storage systems through a single interface (see {ref}`remote-files`). + +To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem: + +```groovy +import java.nio.file.spi.FileSystemProvider + +class MyFileSystemProvider extends FileSystemProvider { + + @Override + String getScheme() { + return 'myfs' + } + + // ... +} +``` + +You can then use this filesystem in your pipeline: + +```nextflow +input = file('myfs://path/to/input/file.txt') +``` + +Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. + +:::{tip} +Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3. +::: + ### Functions :::{versionadded} 22.09.0-edge diff --git a/docs/working-with-files.md b/docs/working-with-files.md index 392d9ae849..8135226e39 100644 --- a/docs/working-with-files.md +++ b/docs/working-with-files.md @@ -226,6 +226,8 @@ myDir.eachFile { item -> In general, you should not need to manually copy files, because Nextflow will automatically stage files in and out of the task environment based on the definition of process inputs and outputs. Ideally, any operation which transforms files should be encapsulated in a process, in order to leverage Nextflow's staging capabilities as much as possible. +(remote-files)= + ## Remote files Nextflow works with many types of remote files and objects using the same interface as for local files. The following protocols are supported: From da2e098186ae8dc756c30a777592aa338e4e0045 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 4 Mar 2025 08:42:49 -0600 Subject: [PATCH 2/3] Update docs/developer/plugins.md Signed-off-by: Ben Sherman --- docs/developer/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md index d48459d009..3ebf6cde47 100644 --- a/docs/developer/plugins.md +++ b/docs/developer/plugins.md @@ -177,7 +177,7 @@ You can then use this filesystem in your pipeline: input = file('myfs://path/to/input/file.txt') ``` -Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. +See [this guide](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more details. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. :::{tip} Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3. From 9294e59dc45d019213ff19afabb0597736b88034 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 4 Mar 2025 14:39:29 -0600 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Chris Hakkaart Signed-off-by: Ben Sherman --- docs/developer/plugins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md index 3ebf6cde47..e82fee95a7 100644 --- a/docs/developer/plugins.md +++ b/docs/developer/plugins.md @@ -153,7 +153,7 @@ Refer to the source code of Nextflow's built-in executors to see how to implemen ### Filesystems -Plugins can define custom filesystems, which are used by Nextflow to interact with external storage systems through a single interface (see {ref}`remote-files`). +Plugins can define custom filesystems that can be used by Nextflow to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem: @@ -177,7 +177,7 @@ You can then use this filesystem in your pipeline: input = file('myfs://path/to/input/file.txt') ``` -See [this guide](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more details. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. :::{tip} Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3.