Welcome to Viash!

Viash is your go-to script wrapper for building data pipelines from modular software components. All you need is your trusty script and a metadata file to embark on this journey.

Check out some of Viash’s key features:

Requirements

This guide assumes you’ve already installed Viash, Docker and Nextflow.

Quickstart example project

To get up and running fast, we provide a template project for you to use. It contains three components from the same package as well as 2 custom operators from vsh-pipeline-operators, which are combined into a Nextflow pipeline as follows:

graph TD
   A(file?.tsv) --> X[vsh_flatten] 
   X --file1.tsv--> B1[/remove_comments/] --> C1[/take_column/] --> Y
   X --file2.tsv--> B2[/remove_comments/] --> C2[/take_column/] --> Y
   Y[vsh_toList] --> D[/combine_columns/]
   D --> E(output)

This pipeline takes one or more TSV files as input and stores its output in an output folder.

Step 1: Get the template

First create a new repository by clicking the “Use this template” button. If you can’t see the “Use this template” button, log into GitHub first.

Next, clone the repository using the following command.

git clone https://github.com/youruser/my_first_pipeline.git && cd my_first_pipeline
Note

At the moment of writing, the branch to use should be 0_8_2.

Your new repository should contain the following files:

tree my_first_pipeline
.
├── CHANGELOG.md
├── LICENSE.md
├── README.md
├── README.qmd
├── _viash.yaml
├── main.nf
├── resources_test
│   ├── file1.tsv
│   └── file2.tsv
└── src
    └── template
        ├── combine_columns
        │   ├── config.vsh.yaml
        │   └── script.R
        ├── remove_comments
        │   ├── config.vsh.yaml
        │   ├── script.sh
        │   └── test.sh
        ├── take_column
        │   ├── config.vsh.yaml
        │   └── script.py
        └── workflow
            ├── config.vsh.yaml
            ├── main.nf
            └── nextflow.config

Step 2: Build the Viash components

With Viash you can turn the components in src/ into Dockerized Nextflow modules by running:

viash ns build --setup cachedbuild --parallel
Output
temporaryFolder: /tmp/viash_hub_repo9628939319909400755 uri: https://viash-hub.com/data-intuitive/vsh-pipeline-operators.git
Cloning into '.'...
checkout out: List(git, checkout, tags/v0.2.0, --, .) 0 
Exporting workflow (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/workflow
Exporting take_column (template) =docker=> /home/runner/work/website/website/guide/_viash_project_template/target/docker/template/take_column
Exporting remove_comments (template) =docker=> /home/runner/work/website/website/guide/_viash_project_template/target/docker/template/remove_comments
Exporting combine_columns (template) =docker=> /home/runner/work/website/website/guide/_viash_project_template/target/docker/template/combine_columns
[notice] Building container 'ghcr.io/viash-io/viash_project_template/template/combine_columns:0.2.3' with Dockerfile
[notice] Building container 'ghcr.io/viash-io/viash_project_template/template/take_column:0.2.3' with Dockerfile
[notice] Building container 'ghcr.io/viash-io/viash_project_template/template/remove_comments:0.2.3' with Dockerfile
Exporting combine_columns (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/combine_columns
Exporting take_column (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/take_column
Exporting remove_comments (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/remove_comments
All 7 configs built successfully

This command not only transforms the Viash components in src/ to Nextflow modules but it also builds the containers when appropriate (starting from the Docker cache when available using the cachedbuild argument). Once everything is built, a new target directory has been created containing the executables and modules grouped per platform:

ls -l
Output
total 68
-rw-r--r-- 1 runner docker  1020 Mar 14 14:42 CHANGELOG.md
-rw-r--r-- 1 runner docker 32219 Mar 14 14:42 LICENSE.md
-rw-r--r-- 1 runner docker  1094 Mar 14 14:42 README.md
-rw-r--r-- 1 runner docker  1125 Mar 14 14:42 README.qmd
-rw-r--r-- 1 runner docker   417 Mar 14 14:42 _viash.yaml
-rw-r--r-- 1 runner docker   245 Mar 14 14:42 main.nf
-rw-r--r-- 1 runner docker   222 Mar 14 14:42 nextflow.config
drwxr-xr-x 2 runner docker  4096 Mar 14 14:42 resources_test
drwxr-xr-x 3 runner docker  4096 Mar 14 14:42 src
drwxr-xr-x 5 runner docker  4096 Mar 14 14:42 target

Step 3: Run the pipeline

Now run the pipeline with Nextflow:

nextflow run . \
  -main-script template/workflow/main.nf \
  -with-docker \
  --input resources_test/file*.tsv \
  --publish_dir output
Output
N E X T F L O W  ~  version 23.10.1
Launching `target/nextflow/template/workflow/main.nf` [marvelous_davinci] DSL2 - revision: 35d7b44269
[e4/d9df2c] Submitted process > workflow:run_wf:remove_comments:processWf:remove_comments_process (run)
[7a/7b49c6] Submitted process > workflow:run_wf:take_column:processWf:take_column_process (run)
[a3/2f8931] Submitted process > workflow:run_wf:combine_columns:processWf:combine_columns_process (run)
[ee/fb9588] Submitted process > workflow:publishStatesSimpleWf:publishStatesProc (run)

This will run the different stages of the workflow , with the final result result being stored in a file named run.combine_columns.output in the output directory output:

cat output/run.combine_columns.output
Output
"1" 0.11
"2" 0.23
"3" 0.35
"4" 0.47

What’s next?

Congratulations, you’ve reached the end of this quickstart tutorial, and we’re excited for you to delve deeper into the world of Viash! Our comprehensive guide and reference documentation is here to help you explore various topics, such as:

So, get ready to enhance your skills and create outstanding solutions with Viash!