Nextflow

When targeting the Nextflow platform, a module is generated that can be run standalone or used in a pipeline.

Requirements

This example targets the Nextflow platform, which requires you to install Nextflow and Docker. Follow the instructions on NextFlow’s homepage to install it on your system if you haven’t done so already. You can find instructions on how to install Docker for your OS on the Viash installation page.

Preparing a component for Nextflow

Unlike components that target the native and Docker platform, a component targeting Nextflow requires you to keep some things in mind besides editing your config file.

Creating the component

This guide builds upon the hello_world component from the Docker component creation guide. You can either follow with that guide first or simply download the component below that contains the component in all supported languages. Simply rename the language folder you want to use to hello_world and use that as your working directory.

Download hello_world_docker.zip

Create a new folder named src and drag the component files inside of that, this is to keep your source and target files neatly separated. Your directory structure should look like this now:

hello world
└── src
    └── config.vsh.yaml
    └── script

Adapting the script

Components that target the Nextflow platform don’t support writing to standard output when run in a pipeline. Instead, their output should be redirected to a file. To do that, change the contents of your script file as below:

Change this:

echo "Hello $par_input"

To this:

echo "Hello $par_input" > $par_output

Change this:

Console.WriteLine("Hello " + par.input);

To this:

File.WriteAllText(par.output, "Hello " + par.input);

Change this:

console.log('Hello' + par['input']);

To this:

const fs = require('fs');
fs.writeFileSync(par['output'], 'Hello ' + par['input']);

Change this:

print("Hello", par["input"])

To this:

file = open(par["output"], 'a+')
file.write("Hello " + par["input"])
file.close()

Change this:

println("Hello " + par.input.get)

To this:

import java.io._

val writer = new FileWriter(par.output.get)
try { writer.append("Hello " + par.input.get) }
finally { writer.close }        

Change this:

cat("Hello ", par$input)

To this:

cat("Hello", par$input, file=par$output, sep=" ", append=TRUE)

Adjusting the config

Add --id argument

Nextflow modules need an --id argument in order to work, so add that to the config file below the --input argument:

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
...

Add a file output argument

Now that the script outputs a file, its output path should be added as an argument to the config file. To do that, add yet another argument below the --id argument named --output:

- type: file
  name: --output
  default: "output"

Add Nextflow platform

To add support for the Nextflow platform as a build target, you need to add nextflow to the platforms section of your Viash config file. To do this, add these lines to the bottom of your config file:

- type: nextflow
  variant: vdsl3

This will add Nextflow as a build target, with VDSL3 set as the pipeline scripting variant, which adds new features and quality of life changes.

By default, Nextflow will use the Docker image specified in the docker platform section. For example:

platforms:
  - type: docker
    image: bash:4.0

If you don’t add docker as a build target or want to override the image, you can use the directives dictionary to specify a base image to use:

platforms:
  - type: nextflow
    variant: vdsl3
    directives:
      container: ubuntu:20.04

Here’s the full config after making these changes:

Download config.vsh.yaml

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
  - type: file
    name: --output
    direction: output
    default: "output"
  resources:
  - type: bash_script
    path: script.sh
platforms:
  - type: native
  - type: docker
    image: bash:4.0
  - type: nextflow
    variant: vdsl3

Here’s the full config after making these changes:

Download config.vsh.yaml

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
  - type: file
    name: --output
    direction: output
    default: "output"
  resources:
  - type: csharp_script
    path: script.csx
platforms:
  - type: native
  - type: docker
    image: "dataintuitive/dotnet-script:1.2.1"
    setup:
      - type: apk
        packages: [ bash ]
  - type: nextflow
    variant: vdsl3

Here’s the full config after making these changes:

Download config.vsh.yaml

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
  - type: file
    name: --output
    direction: output
    default: "output"
  resources:
  - type: javascript_script
    path: script.js
platforms:
  - type: native
  - type: docker
    image: node:15-buster
  - type: nextflow
    variant: vdsl3

Here’s the full config after making these changes:

Download config.vsh.yaml

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
  - type: file
    name: --output
    direction: output
    default: "output"
  resources:
  - type: python_script
    path: script.py
platforms:
  - type: native
  - type: docker
    image: "python:3.8"
  - type: nextflow
    variant: vdsl3

Here’s the full config after making these changes:

Download config.vsh.yaml

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
  - type: file
    name: --output
    direction: output
    default: "output"
  resources:
  - type: scala_script
    path: script.scala
platforms:
  - type: native
  - type: docker
    image: hseeberger/scala-sbt
  - type: nextflow
    variant: vdsl3

Here’s the full config after making these changes:

Download config.vsh.yaml

functionality:
  name: hello_world
  description: A minimal example component.
  arguments:
  - type: string
    name: --input
    default: "World"
  - type: string
    name: --id
    default: "hello_world"
  - type: file
    name: --output
    direction: output
    default: "output"
  resources:
  - type: r_script
    path: script.R
platforms:
  - type: native
  - type: docker
    image: "rocker/tidyverse:4.0.4"
  - type: nextflow
    variant: vdsl3

Running

In contrast to native and Docker based components, a Nextflow based component cannot be ran by using the viash run command. Components targeting Nextflow require you to use viash build first to generate a Nextflow module and run that module using nextflow run. See the Nextflow Build Target page for more information on how to generate a Nextflow module and run it as a standalone pipeline.