Welcome to Viash!

Viash is your go-to script wrapper for building data pipelines from modular software components. All you need is your trusty script and a metadata file to embark on this journey.

Check out some of Viash’s key features:

Requirements

This guide assumes you’ve already installed Viash, Docker and Nextflow.

Quickstart example project

To get up and running fast, we provide a template project for you to use. It contains three components from the same package as well as 2 custom operators from vsh-pipeline-operators, which are combined into a Nextflow pipeline as follows:

graph TD
   A(file?.tsv) --> X[vsh_flatten] 
   X --file1.tsv--> B1[/remove_comments/] --> C1[/take_column/] --> Y
   X --file2.tsv--> B2[/remove_comments/] --> C2[/take_column/] --> Y
   Y[vsh_toList] --> D[/combine_columns/]
   D --> E(output)

This pipeline takes one or more TSV files as input and stores its output in an output folder.

Step 1: Get the template

First create a new repository by clicking the “Use this template” button. If you can’t see the “Use this template” button, log into GitHub first.

Next, clone the repository using the following command.

git clone https://github.com/youruser/my_first_pipeline.git && cd my_first_pipeline
Note

At the moment of writing, the branch to use should be 0_8_2.

Your new repository should contain the following files:

tree my_first_pipeline
.
├── CHANGELOG.md
├── LICENSE.md
├── README.md
├── README.qmd
├── _viash.yaml
├── main.nf
├── resources_test
│   ├── file1.tsv
│   └── file2.tsv
└── src
    └── template
        ├── combine_columns
        │   ├── config.vsh.yaml
        │   └── script.R
        ├── remove_comments
        │   ├── config.vsh.yaml
        │   ├── script.sh
        │   └── test.sh
        ├── take_column
        │   ├── config.vsh.yaml
        │   └── script.py
        └── workflow
            ├── config.vsh.yaml
            ├── main.nf
            └── nextflow.config

Step 2: Build the Viash components

With Viash you can turn the components in src/ into Dockerized Nextflow modules by running:

viash ns build --setup cachedbuild --parallel
Output 
temporaryFolder: /tmp/viash_hub_repo12598496069126012426 uri: https://viash-hub.com/data-intuitive/vsh-pipeline-operators.git
Cloning into '.'...
checkout out: List(git, checkout, tags/v0.2.0, --, .) 0 
Exporting remove_comments (template) =docker=> /home/runner/work/website/website/guide/_viash_project_template/target/docker/template/remove_comments
Exporting combine_columns (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/combine_columns
Exporting take_column (template) =docker=> /home/runner/work/website/website/guide/_viash_project_template/target/docker/template/take_column
Exporting take_column (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/take_column
Exporting workflow (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/workflow
Exporting combine_columns (template) =docker=> /home/runner/work/website/website/guide/_viash_project_template/target/docker/template/combine_columns
[notice] Building container 'ghcr.io/viash-io/viash_project_template/template/take_column:0.2.3' with Dockerfile
[notice] Building container 'ghcr.io/viash-io/viash_project_template/template/remove_comments:0.2.3' with Dockerfile
[notice] Building container 'ghcr.io/viash-io/viash_project_template/template/combine_columns:0.2.3' with Dockerfile
Exporting remove_comments (template) =nextflow=> /home/runner/work/website/website/guide/_viash_project_template/target/nextflow/template/remove_comments
All 7 configs built successfully

This command not only transforms the Viash components in src/ to Nextflow modules but it also builds the containers when appropriate (starting from the Docker cache when available using the cachedbuild argument). Once everything is built, a new target directory has been created containing the executables and modules grouped per platform:

ls -l
Output 
total 68
-rw-r--r-- 1 runner docker  1020 Apr 26 17:22 CHANGELOG.md
-rw-r--r-- 1 runner docker 32219 Apr 26 17:22 LICENSE.md
-rw-r--r-- 1 runner docker  1094 Apr 26 17:22 README.md
-rw-r--r-- 1 runner docker  1125 Apr 26 17:22 README.qmd
-rw-r--r-- 1 runner docker   417 Apr 26 17:22 _viash.yaml
-rw-r--r-- 1 runner docker   245 Apr 26 17:22 main.nf
-rw-r--r-- 1 runner docker   222 Apr 26 17:22 nextflow.config
drwxr-xr-x 2 runner docker  4096 Apr 26 17:22 resources_test
drwxr-xr-x 3 runner docker  4096 Apr 26 17:22 src
drwxr-xr-x 5 runner docker  4096 Apr 26 17:22 target

Step 3: Run the pipeline

Now run the pipeline with Nextflow:

nextflow run . \
  -main-script template/workflow/main.nf \
  -with-docker \
  --input resources_test/file*.tsv \
  --publish_dir output
Output 
N E X T F L O W  ~  version 23.10.1
Launching `target/nextflow/template/workflow/main.nf` [mad_lovelace] DSL2 - revision: 0601439cf4
[d2/29b72c] Submitted process > workflow:run_wf:remove_comments:processWf:remove_comments_process (run)
[a8/8100fa] Submitted process > workflow:run_wf:take_column:processWf:take_column_process (run)
[a5/70c498] Submitted process > workflow:run_wf:combine_columns:processWf:combine_columns_process (run)
[26/8340a3] Submitted process > workflow:publishStatesSimpleWf:publishStatesProc (run)

This will run the different stages of the workflow , with the final result result being stored in a file named run.combine_columns.output in the output directory output:

cat output/run.combine_columns.output
Output 
"1" 0.11
"2" 0.23
"3" 0.35
"4" 0.47

What’s next?

Congratulations, you’ve reached the end of this quickstart tutorial, and we’re excited for you to delve deeper into the world of Viash! Our comprehensive guide and reference documentation is here to help you explore various topics, such as:

So, get ready to enhance your skills and create outstanding solutions with Viash!