Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document plugin filesystems #5841

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/developer/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,38 @@ process foo {
Refer to the source code of Nextflow's built-in executors to see how to implement the various components of an executor. You might be able to implement most of your executor by simply reusing existing code.
:::

### Filesystems

Plugins can define custom filesystems, which are used by Nextflow to interact with external storage systems through a single interface (see {ref}`remote-files`).

To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem:

```groovy
import java.nio.file.spi.FileSystemProvider

class MyFileSystemProvider extends FileSystemProvider {

@Override
String getScheme() {
return 'myfs'
}

// ...
}
```

You can then use this filesystem in your pipeline:

```nextflow
input = file('myfs://path/to/input/file.txt')
```

Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems.

:::{tip}
Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3.
:::

### Functions

:::{versionadded} 22.09.0-edge
Expand Down
2 changes: 2 additions & 0 deletions docs/working-with-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,8 @@ myDir.eachFile { item ->

In general, you should not need to manually copy files, because Nextflow will automatically stage files in and out of the task environment based on the definition of process inputs and outputs. Ideally, any operation which transforms files should be encapsulated in a process, in order to leverage Nextflow's staging capabilities as much as possible.

(remote-files)=

## Remote files

Nextflow works with many types of remote files and objects using the same interface as for local files. The following protocols are supported:
Expand Down