Create a module

Creating a VDSL3 module is as simple as adding { type: nextflow } to the platforms section in the Viash config. Luckily, our previous example already contained such an entry:

Bash

functionality:
  name: example_bash
  description: A minimal example component.
  arguments:
    - type: file
      name: --input
      example: file.txt
      required: true
    - type: file
      name: --output
      direction: output
      example: output.txt
      required: true
  resources:
    - type: bash_script
      path: script.sh
platforms:
  - type: docker
    image: "bash:4.0"
  - type: native
  - type: nextflow

C#

functionality:
  name: example_csharp
  description: A minimal example component.
  arguments:
    - type: file
      name: --input
      example: file.txt
      required: true
    - type: file
      name: --output
      direction: output
      example: output.txt
      required: true
  resources:
    - type: csharp_script
      path: script.csx
platforms:
  - type: docker
    image: "ghcr.io/data-intuitive/dotnet-script:1.3.1"
  - type: native
  - type: nextflow

JavaScript

functionality:
  name: example_js
  description: A minimal example component.
  arguments:
    - type: file
      name: --input
      example: file.txt
      required: true
    - type: file
      name: --output
      direction: output
      example: output.txt
      required: true
  resources:
    - type: javascript_script
      path: script.js
platforms:
  - type: docker
    image: "node:19-bullseye-slim"
  - type: native
  - type: nextflow

Python

functionality:
  name: example_python
  description: A minimal example component.
  arguments:
    - type: file
      name: --input
      example: file.txt
      required: true
    - type: file
      name: --output
      direction: output
      example: output.txt
      required: true
  resources:
    - type: python_script
      path: script.py
platforms:
  - type: docker
    image: "python:3.10-slim"
  - type: native
  - type: nextflow

R

functionality:
  name: example_r
  description: A minimal example component.
  arguments:
    - type: file
      name: --input
      example: file.txt
      required: true
    - type: file
      name: --output
      direction: output
      example: output.txt
      required: true
  resources:
    - type: r_script
      path: script.R
platforms:
  - type: docker
    image: "eddelbuettel/r2u:22.04"
  - type: native
  - type: nextflow

Scala

functionality:
  name: example_scala
  description: A minimal example component.
  arguments:
    - type: file
      name: --input
      example: file.txt
      required: true
    - type: file
      name: --output
      direction: output
      example: output.txt
      required: true
  resources:
    - type: scala_script
      path: script.scala
platforms:
  - type: docker
    image: "sbtscala/scala-sbt:eclipse-temurin-19_36_1.7.2_2.13.10"
  - type: native
  - type: nextflow

Build the VDSL3 module

We will now turn the Viash component into a VDSL3 module. By default, the viash build command will select the first platform in the list of platforms. To select the nextflow platform, use the --platform nextflow argument, or -p nextflow for short.

Bash

viash build config.vsh.yaml -o target -p nextflow

This will generate a Nextflow module in the target/ directory:

tree target

target
├── main.nf
└── nextflow.config

0 directories, 2 files

This main.nf file is both a standalone Nextflow pipeline and a module which can be used as part of another pipeline.

Tip

You can also use the viash ns build command to build all of the platforms in one go. Give it a try! More information in the following section.

Module as a standalone pipeline

When VDSL3 modules are used as a standalone pipeline, you need to specify the input parameters and a --publish_dir parameter, as Nextflow will automatically choose the parameter names of the output files.

You can run the executable by providing a value for --input and --publish_dir:

nextflow run target/main.nf --input config.vsh.yaml --publish_dir output/

N E X T F L O W  ~  version 22.10.6
Launching `target/main.nf` [astonishing_lumiere] DSL2 - revision: 9e6b8c6325
[-        ] process > example_bash:example_bash_p... [  0%] 0 of 1
input: [run, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/config.vsh.yaml]]

executor >  local (1)
[cd/a08e71] process > example_bash:example_bash_p... [100%] 1 of 1 ✔
input: [run, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/config.vsh.yaml]]
output: [run, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/cd/a08e71dc9a07855e9b42e328540d66/run.example_bash.output.txt]

This results in the following output:

tree output

output
└── run.example_bash.output.txt

0 directories, 1 file

The pipeline help can be shown by passing the --help parameter (Output not shown).

nextflow run target/main.nf --help

Passing a parameter list

Every VDSL3 can accept a list of parameters to populate a Nextflow channel with.

For example, we create a set of input files which we want to process in parallel.

touch sample1.txt sample2.txt sample3.txt sample4.txt

Next, we create a YAML file param_list.yaml containing an id and an input value for each parameter entry.

- id: sample1
  input: /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample1.txt
- id: sample2
  input: /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample2.txt
- id: sample3
  input: /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample3.txt
- id: sample4
  input: /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample4.txt

You can run the pipeline on the list of parameters using the --param_list parameter.

nextflow run target/main.nf --param_list param_list.yaml --publish_dir output2

N E X T F L O W  ~  version 22.10.6
Launching `target/main.nf` [peaceful_brown] DSL2 - revision: 9e6b8c6325
[-        ] process > example_bash:example_bash_p... -
input: [sample4, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample4.txt]]
input: [sample3, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample3.txt]]
input: [sample2, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample2.txt]]
input: [sample1, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample1.txt]]

executor >  local (2)
[db/88a291] process > example_bash:example_bash_p... [  0%] 0 of 4
input: [sample4, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample4.txt]]
input: [sample3, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample3.txt]]
input: [sample2, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample2.txt]]
input: [sample1, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample1.txt]]

executor >  local (4)
[db/88a291] process > example_bash:example_bash_p... [ 50%] 2 of 4
input: [sample4, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample4.txt]]
input: [sample3, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample3.txt]]
input: [sample2, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample2.txt]]
input: [sample1, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample1.txt]]
output: [sample4, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/65/77903ea3fcc7ef30855b42bd75ce3d/sample4.example_bash.output.txt]
output: [sample3, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/db/88a29170bf06b7d68e309d976e4fb0/sample3.example_bash.output.txt]

executor >  local (4)
[59/be3e92] process > example_bash:example_bash_p... [100%] 4 of 4 ✔
input: [sample4, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample4.txt]]
input: [sample3, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample3.txt]]
input: [sample2, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample2.txt]]
input: [sample1, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample1.txt]]
output: [sample4, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/65/77903ea3fcc7ef30855b42bd75ce3d/sample4.example_bash.output.txt]
output: [sample3, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/db/88a29170bf06b7d68e309d976e4fb0/sample3.example_bash.output.txt]
output: [sample2, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/5b/242eb46388318c40495de034472350/sample2.example_bash.output.txt]
output: [sample1, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/59/be3e92d2f9dbfe9024dffab1e0702c/sample1.example_bash.output.txt]

executor >  local (4)
[59/be3e92] process > example_bash:example_bash_p... [100%] 4 of 4 ✔
input: [sample4, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample4.txt]]
input: [sample3, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample3.txt]]
input: [sample2, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample2.txt]]
input: [sample1, [output:$id.$key.output.txt, input:/tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/sample1.txt]]
output: [sample4, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/65/77903ea3fcc7ef30855b42bd75ce3d/sample4.example_bash.output.txt]
output: [sample3, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/db/88a29170bf06b7d68e309d976e4fb0/sample3.example_bash.output.txt]
output: [sample2, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/5b/242eb46388318c40495de034472350/sample2.example_bash.output.txt]
output: [sample1, /tmp/RtmpPMbhZR/create-a-module2a69694757d/bash/work/59/be3e92d2f9dbfe9024dffab1e0702c/sample1.example_bash.output.txt]

This results in the following outputs:

tree output2

output2
├── sample1.example_bash.output.txt
├── sample2.example_bash.output.txt
├── sample3.example_bash.output.txt
└── sample4.example_bash.output.txt

0 directories, 4 files

Tip

Instead of a YAML, you can also pass a JSON or a CSV to the --param_list parameter.

Module as part of a pipeline

This module can also be used as part of a Nextflow pipeline. Below is a short preview of what this looks like.

import { example_bash } from "target/main.nf"

Channel.fromList([
  ["sample1", file("sample1.txt")],
  ["sample2", file("sample2.txt")],
  ["sample3", file("sample3.txt")]
])
  | view { it -> "input: $it" }
  | example_bash
  | view { it -> "output: $it" }

We will discuss building pipelines with VDSL3 modules in more detail in Create a pipeline.

Improvements over standard Nextflow modules

• No need to write any Nextflow Groovy code, just your script and the Viash config.
• VDSL3 module are also standalone pipelines.
• Help documentation is automatically generated.
• Standardized interface for passing parameter lists.
• Automatically uses the Docker platform’s container.