Viash 0.8.0

Nextflow workflows and dependencies

New Release
Author

Viash Team

Published

October 23, 2023

What’s new?

Nextflow workflows definitions are picked up by Viash and assembled into a functional Nextflow workflow, reducing the amount of boilerplate code needed to be written by the user. It also adds a new runIf argument to the NextflowPlatform which allows for conditional execution of modules. We added new ‘dependencies’ functionality to allow for more advanced functionality to be offloaded and re-used in components and workflows.

Full changelog

BREAKING CHANGES

  • NextflowPlatform: Changed the default value of auto.simplifyOutput from true to false (#522, PR #518). With simplifyOutput set to true, the resulting Map could be simplified into a File or a List[File] depending on the number of outputs. To replicate the previous behaviour, add the following config mod to _viash.yaml:

    config_mods: |
  .platforms[.type == 'nextflow'].auto.simplifyOutput := true

  • VDSL3Helper.nf: Removed from the Viash jar file (PR #518). Its functions have been moved to WorkflowHelper.nf.

  • DataflowHelper.nf: Added deprecation warning to functions from this file (PR #518).

  • preprocessInputs() in WorkflowHelper.nf: Added deprecation warning to preprocessInputs() because this function causes a synchronisation event (PR #518).

  • author.props: Removed deprecated props field (PR #536). Deprecated since 0.7.4.

NEW FUNCTIONALITY

  • dependencies: Add dependencies and repositories to functionality (PR #509). The new functionality allows specifying dependencies and where to retrieve (repositories) them in a component, and subsequentially allows advanced functionality to be offloaded and re-used in scripts and projects. This is alike e.g. npm, pip and many others. A big difference is that we aim to provide the needed boilerplate code to ease the usage of the dependencies in scripts, workflows and pipelines. Note that the dependency is required to be a built Viash component or project and not a random file or code project found externally. This is needed to provide the necessary background information to correctly link dependencies into a component.

  • NextflowScript & NextflowPlatform: Merged code for merging the main.nf files for VDSL3 components and wrapped Nextflow workflows (PR #518). By aligning the codebase for these two, wrapped Nextflow workflows are more similar to VDSL3 components. For example, you can override the behaviour of a wrapped Nextflow workflow using the .run() method. Status of a workflows .run() arguments:

    • Works as intended: auto.simplifyInput, auto.simplifyOutput, fromState, toState, map, mapData, mapPassthrough, filter, auto.publish = "state"
    • Does not work (yet): auto.transcript, auto.publish = true, directives, debug.

    In a next PR, each of the dependencies will have their values overridden by the arguments of the .run.

  • NextflowPlatform: The data passed to the input of a component and produced as output by the component are now validated against the arguments defined in the Viash config (PR #518).

  • NextflowPlatform: Use stageAs to allow duplicate filenames to be used automatigically (PR #518).

  • NextflowPlatform: When wrapping Nextflow workflows, throw an error if the IDs of the output channel doesn’t match the IDs of the input channel (PR #518). If they don’t, the workflow should store the original ID of the input tuple in the in the _meta.join_id field inside the state as follows: Example input event: ["id", [input: file(...)]], Example output event: ["newid", [output: file(...), _meta: [join_id: "id"]]]

  • NextflowPlatform: Added new .run() argument runIf - a function that determines whether the module should be run or not (PR #553). If the runIf closure evaluates to true, then the module will be run. Otherwise it will be passed through without running.

MAJOR CHANGES

  • WorkflowHelper.nf: The workflow helper was split into different helper files for each of the helper functions (PR #518). For now, these helper files are pasted together to recreate the WorkflowHelper.nf. In Viash development environments, don’t forget to run ./configure to start using the updated Makefile.

  • NextflowPlatform: Set default tag to "$id" (#521, PR #518).

  • NextflowPlatform: Refactoring of helper functions (PR #557).

    • Cleaned up processConfig(): Removed support for functionality.inputs and functionality.outputs
    • Cleaned up processConfig(): Removed support for .functionality.argument_groups[].argument containing a list of argument ids as opposed to the arguments themselves.
    • Rewrote --param_list parser.
    • Removed unused function applyConfig() and applyConfigToOneParamSet().
    • Refactored channelFromParams() to make use of new helper functions.
    • Removed deprecated paramsToChannel(), paramsToList(), viashChannel().
    • Deprecated preprocessInputs() – use the wrapped Viash Nextflow functionality instead.
    • Refactored preprocessInputs() to make use of new helper functions.
    • Deprecated run arguments map, mapData, mapPassthrough, renameKeys.

MINOR CHANGES

  • NextflowPlatform: Throw error when unexpected keys are passed to the .run() method (#512, PR #518).

  • Testbenches: Add testbenches for the new dependencies functionality and other small coverage improvements (PR #524).

  • NextflowPlatform: Use moduleDir instead of projectDir to determine the resource directory.

  • NextflowPlatform: Rename internal VDSL3 variables to be more consistent with regular Viash component variables and avoid naming clashes (PR #553).

DOCUMENTATION

  • Minor fixes to VDSL3 reference documentation (PR #508).

BUG FIXES

  • WorkflowHelper.nf: Only set default values of output files which are not already set, and if the output file argument is not required (PR #514).

  • NextflowPlatform: When using fromState and toState, do not throw an error when the state or output is missing an optional argument (PR #515).

  • export cli_autocomplete: Fix output script format and hide --loglevel and --colorize (PR #544). Masked arguments are usable but might not be very useful to always display in help messages.