Import a VDSL3 module

A VDSL3 module is a Nextflow module generated by Viash. See the guide for a more in-depth explanation on how to create Nextflow workflows with VDSL3 modules.

Importing a VDSL3 module

After building a VDSL3 module from a component, the VDSL3 module can be imported just like any other Nextflow module.

Example:

include { mymodule } from 'target/nextflow/mymodule/main.nf'

VDSL3 module interface

VDSL3 modules are actually workflows which take one channel and emit one channel. It expects the channel events to be tuples containing an ‘id’ and a ‘state’: [id, state], where id is a unique String and state is a Map[String, Object]. The resulting channel then consists of tuples [id, new_state].

Example:

workflow {
  Channel.fromList([
    ["myid", [input: file("in.txt")]]
  ])
    | mymodule
}
Note

If the input tuple has more than two elements, the elements after the second element are passed through to the output tuple. That is, an input tuple [id, input, ...] will result in a tuple [id, output, ...] after running the module. For example, an input tuple ["foo", [input: file("in.txt")], "bar"] will result in an output tuple ["foo", [output: file("out.txt")], "bar"].

Customizing VDSL3 modules on the fly

Usually, Nextflow processes are quite static objects. For example, changing its directives can be quite tricky.

The .run() function is a unique feature for every VDSL3 module which allows dynamically altering the behaviour of a module from within the pipeline. For example, we use it to set the publishDir directive to "output/" so the output of that step in the pipeline will be stored as output.

Example:

workflow {
  Channel.fromList([
    ["myid", [input: file("in.txt")]]
  ])
    | mymodule.run(
      args: [k: 10],
      directives: [cpus: 4, memory: "16 GB"]
    )
}

Arguments of .run()

  • key (String): A unique key used to trace the process and help make names of output files unique. Default: the name of the Viash component.

  • args (Map[String, Object]): Argument overrides to be passed to the module.

  • directives (Map[String, Object]): Custom directives overrides. See the Nextflow documentation for a list of available directives.

  • auto (Map[String, Boolean]): Whether to apply certain automated processing steps. Default values are inherited from the Viash config.

  • auto.simplifyInput: If true, if the input tuple is a single file and if the module only has a single input file, the input file will be passed the module accordingly. Default: true (inherited from Viash config).

  • auto.simplifyOutput: If true, if the output tuple is a single file and if the module only has a single output file, the output map will be transformed into a single file. Default: true (inherited from Viash config).

  • auto.publish: If true, the output files will be published to the params.publishDir folder. Default: false (inherited from Viash config).

  • auto.transcript: If true, the module’s transcript will be published to the params.transcriptDir folder. Default: false (inherited from Viash config).

  • map (Function): Apply a map over the incoming tuple. Example: { tup -> [ tup[0], [input: tup[1].output] ] + tup.drop(2) }. Default: null.

  • mapId (Function): Apply a map over the ID element of a tuple (i.e. the first element). Example: { id -> id + "_foo" }. Default: null.

  • mapData (Function): Apply a map over the data element of a tuple (i.e. the second element). Example: { data -> [ input: data.output ] }. Default: null.

  • mapPassthrough (Function): Apply a map over the passthrough elements of a tuple (i.e. the tuple excl. the first two elements). Example: { pt -> pt.drop(1) }. Default: null.

  • filter (Function): Filter the channel. Example: { tup -> tup[0] == "foo" }. Default: null.

  • fromState: Fetch data from the state and pass it to the module without altering the current state. fromState should be null, List[String], Map[String, String] or a function.

    • If it is null, the state will be passed to the module as is.
    • If it is a List[String], the data will be the values of the state at the given keys.
    • If it is a Map[String, String], the data will be the values of the state at the given keys, with the keys renamed according to the map.
    • If it is a function, the tuple ([id, state]) in the channel will be passed to the function, and the result will be used as the data.

    Example: { id, state -> [input: state.fastq_file] } Default: null

  • toState: Determine how the state should be updated after the module has been run. toState should be null, List[String], Map[String, String] or a function.

    • If it is null, the state will be replaced with the output of the module.
    • If it is a List[String], the state will be updated with the values of the data at the given keys.
    • If it is a Map[String, String], the state will be updated with the values of the data at the given keys, with the keys renamed according to the map.
    • If it is a function, a tuple ([id, output, state]) will be passed to the function, and the result will be used as the new state.

    Example: { id, output, state -> state + [counts: state.output] } Default: { id, output, state -> output }

  • debug: Whether or not to print debug messages. Default: false.