Viash 0.6.0

Major release that includes functionality improvements, testing options and several bug fixes

New Release
Author

Viash Team

Published

September 7, 2022

What’s new?

This is a major release that contains a lot of new features and changes. Here are a few highlights:

  • VDSL3 is now the default Nextflow platform
  • The inputs and outputs are now deprecated, use the more flexible argument groups instead
  • Arguments can now be added directly to argument groups instead of referencing arguments by name
  • The number of processes and amount of memory that are allowed to be used can now be included in the config
  • A new viash ns exec command has been added for executing arbitrary commands on all Viash components

See the full changelog below for a breakdown of all changes.

Full changelog

BREAKING CHANGES

  • NextflowPlatform: variant: vdsl3 is now the default NextflowPlatform. variant: legacy has been deprecated.

  • Functionality: Fields .inputs and .outputs has been deprecated. Please use .argument_groups instead (#186). Before:

    functionality:
      inputs:
        - name: "--foo"
      outputs:
        - name: "--bar"

    Now:

    functionality:
      argument_groups:
        - name: Inputs
          arguments:
            - name: "--foo"
              type: file
        - name: Outputs
          arguments:
            - name: "--bar"
              type: file
              direction: output
  • Passing strings to an argument group’s arguments has been deprecated. Please simply copy the argument itself into the argument group (#186). Before:

    functionality:
      arguments:
        - name: "--foo"
          type: file
        - name: "--bar"
          type: file
          direction: output
      argument_groups:
        - name: Inputs
          arguments: [ foo ]
        - name: Outputs
          arguments: [ bar ]

    Now:

    functionality:
      argument_groups:
        - name: Inputs
          arguments:
            - name: "--foo"
              type: file
        - name: Outputs
          arguments:
            - name: "--bar"
              type: file
              direction: output

NEW FUNCTIONALITY

  • Allow setting the number of processes and memory limit from within the Viash config, as well as a list of required commands. Example:

    functionality:
    name: foo
    requirements:
      n_proc: 10
      memory: 10G
      commands: [ bash, r, perl ]

    You can override the default requirements at runtime:

    • ./foo ---n_proc 4 ---memory 100pb (for NativePlatform or DockerPlatform)
    • By adding process.cpus = 4 and process.memory "100 PB" to a nextflow.config (for NextflowPlatform)

    This results the following meta variables to be injected into a script:

    • meta_n_proc (in Bash) or meta["n_proc"] (in any other language): Number of processes the script is allowed to spawn.
    • meta_memory_b (in Bash) or meta["memory_b"] (in any other language): Amount of memory the script is allowed to allocate, in bytes.
    • meta_memory_kb (in Bash) or meta["memory_kb"] (in any other language): Same but in kilobytes, rounded up.
    • meta_memory_mb (in Bash) or meta["memory_mb"] (in any other language): Same but in megabytes, rounded up.
    • meta_memory_gb (in Bash) or meta["memory_gb"] (in any other language): Same but in gigabytes, rounded up.
    • meta_memory_tb (in Bash) or meta["memory_tb"] (in any other language): Same but in terabytes, rounded up.
    • meta_memory_pb (in Bash) or meta["memory_pb"] (in any other language): Same but in petabytes, rounded up.
  • viash ns exec: Added a command for executing arbitrary commands for all found Viash components. The syntax of this command is inspired by find . -exec echo {} \;.

    The following fields are automatically replaced:

    • {} | {path}: path to the config file
    • {abs-path}: absolute path to the config file
    • {dir}: path to the parent directory of the config file
    • {abs-dir}: absolute path to the directory of the config file
    • {main-script}: path to the main script (if any)
    • {abs-main-script}: absolute path to the main script (if any)
    • {functionality-name}: name of the component

    A command suffixed by \; (or nothing) will execute one command for each of the Viash components, whereas a command suffixed by + will execute one command for all Viash components.

  • ConfigMod: Added a del(...) config mod to be able to delete a value from the yaml. Example: del(.functionality.version).

MAJOR CHANGES

  • Folder structure: Adjusted the folder structure to correctly reflect the the namespace change of viash from com.dataintuitive.viash to io.viash.

  • Functionality: Reworked the enabled field from boolean to a status field which can have the following statusses: enabled, disabled and deprecated. When parsing a config file which has the status field set to deprecated a warning message is displayed on stderr. Backwards for enabled is provided where enabled: true => status: enabled and enabled: false => status: false. The enabled field is marked deprecated.

MINOR CHANGES

  • Resources: Handle edge case when no resources are specified in the vsh.yaml config file and display a warning message.

  • BashWrapper: Add a warning when an argument containing flags (e.g. --foo) is not recognized and will be handled as a positional argument as this is likely a mistake.

  • Functionality: Add check to verify there are no double argument names or short names in the config vsh.yaml declarations.

  • BashWrapper: Add check to verify a parameter isn’t declared twice on the CLI, except in the case multiple: true is declared as then it’s a valid use case.

  • BashWrapper: For int min/max checking: use native bash functionality so there is no dependency to bc. For double min/max checking: add fallback code to use awk in case bc is not present on the system (most likely to happen when running tests in a docker container).

  • viash ns list/viash config view: Allow viewing the post-processed argument groups by passing the --parse_argument_groups parameter.

TESTING

  • ConfigMod: Added unit tests for condition config mods.

  • MainTestDockerSuite: Derive config alternatives from the base vsh.yaml instead of adding the changes in separate files. This both reduces file clutter and prevents having to change several files when there are updates in the config format.

  • GitTest: Added unit tests for Git helper (#216).

BUG FIXES

  • csharp_script, javascript_script, python_script, r_script, scala_script: Make meta fields for memory and n_proc optional.

  • NextflowVdsl3Platform: Don’t generate an error when --publish_dir is not defined and -profile no_publish is used.

  • Viash run: Viash now properly returns the exit code from the executed script.

  • Git: Fix incorrect metadata when git repository is empty (#216).