Par and meta variables

A showcase of the available variables and meta-variables.

When running a Viash component with viash run, Viash will wrap your script into a Bash executable. In doing so, it strips away the “Viash placeholder” code block and replaces it by a bit of code to your script for reading any parameter values at runtime.

Recognizing the Viash placeholder code block

Recall what the script of the previous example looks like:

Bash

#!/bin/bash

## VIASH START
par_input=path/to/file.txt
par_output=output.txt
## VIASH END

# copy file
echo "Copying '$par_input' to '$par_output'."
cp -r "$par_input" "$par_output"

C#

using System.IO;

// VIASH START
var par = new {
  input = "path/to/file.txt",
  output = "output.txt"
};
// VIASH END

// copy file
Console.WriteLine($"Copying '{par.input}' to '{par.output}'.");
File.Copy(par.input, par.output, true);

JavaScript

const fs = require('fs');

// VIASH START
let par = {
  'input': 'path/to/file.txt',
  'output': 'output.txt'
};
// VIASH END

// copy file
console.log(`Copying '${par['input']}' to '${par['output']}'`)
fs.copyFile(par['input'], par['output'], (err) => {
  if (err) throw err;
});

Python

import shutil

## VIASH START
par = {
  'input': 'file.txt',
  'output': 'output.txt'
}
## VIASH END

# copy file
print(f"Copying '{par['input']}' to '{par['output']}'.")
shutil.copyfile(par['input'], par['output'])

R

## VIASH START
par <- list(
  "input" = 'file.txt',
  "output" = 'output.txt'
)
## VIASH END

# copy file
cat("Copying '", par$input, "' to '", par$output, "'.\n", sep = "")
file.copy(par$input, par$output)

Scala

import java.nio.file.StandardCopyOption.REPLACE_EXISTING
import java.nio.file.Files
import java.nio.file.Paths

// VIASH START
case class ViashPar(input: String, output: String)
val par = ViashPar(
  "path/to/file.txt",
  "output.txt"
)
// VIASH END

// copy file
println(s"Copying '${par.input}' to '${par.output}'.")
val fileIn = Paths.get(par.input)
val fileOut = Paths.get(par.output)
Files.copy(fileIn, fileOut, REPLACE_EXISTING)

A “Viash placeholder” code block is the section between the VIASH START and VIASH END comments.

What happens at runtime

By passing arguments to the component, Viash will add your parameter values to your script by replacing the Viash placeholder code block. If no such code block exists yet, the parameters are inserted at the top of the file.

The resulting code block will contain two maps (or dictionaries): par and meta. The par map contains the parameter values specified by the user, and meta contains additional information on the current runtime environment. Note that for Bash scripts, the par and meta maps are flattened into separate environment variables.

Previewing the par and meta objects

To get insight into how par and meta are defined, you can run viash config inject to replace the current parameter placeholder with an auto-generated parameter placeholder.

Warning

This will change the contents of your script!

Bash

Running viash config inject effectively changes the contents of the script.

viash config inject config.vsh.yaml

The updated script.sh now contains the following code:

#!/bin/bash

## VIASH START
# The following code has been auto-generated by Viash.
par_input='file.txt'
par_output='output.txt'
meta_name='example_bash'
meta_functionality_name='example_bash'
meta_resources_dir='/tmp/viash_inject_example_bash1210728389008820256'
meta_executable='/tmp/viash_inject_example_bash1210728389008820256/example_bash'
meta_config='/tmp/viash_inject_example_bash1210728389008820256/.config.vsh.yaml'
meta_temp_dir='/tmp'
meta_cpus='123'
meta_memory_b='123'
meta_memory_kb='123'
meta_memory_mb='123'
meta_memory_gb='123'
meta_memory_tb='123'
meta_memory_pb='123'
meta_memory_kib='123'
meta_memory_mib='123'
meta_memory_gib='123'
meta_memory_tib='123'
meta_memory_pib='123'

## VIASH END

# copy file
echo "Copying '$par_input' to '$par_output'."
cp -r "$par_input" "$par_output"

C#

Running viash config inject effectively changes the contents of the script.

viash config inject config.vsh.yaml

The updated script.csx now contains the following code:

using System.IO;

// VIASH START
// The following code has been auto-generated by Viash.
var par = new {
  input = @"file.txt",
  output = @"output.txt"
};
var meta = new {
  name = @"example_csharp",
  functionality_name = @"example_csharp",
  resources_dir = @"/tmp/viash_inject_example_csharp17936768336908624138",
  executable = @"/tmp/viash_inject_example_csharp17936768336908624138/example_csharp",
  config = @"/tmp/viash_inject_example_csharp17936768336908624138/.config.vsh.yaml",
  temp_dir = @"/tmp",
  cpus = Convert.ToInt32(@"123"),
  memory_b = Convert.ToInt64(@"123"),
  memory_kb = Convert.ToInt64(@"123"),
  memory_mb = Convert.ToInt64(@"123"),
  memory_gb = Convert.ToInt64(@"123"),
  memory_tb = Convert.ToInt64(@"123"),
  memory_pb = Convert.ToInt64(@"123"),
  memory_kib = Convert.ToInt64(@"123"),
  memory_mib = Convert.ToInt64(@"123"),
  memory_gib = Convert.ToInt64(@"123"),
  memory_tib = Convert.ToInt64(@"123"),
  memory_pib = Convert.ToInt64(@"123")
};
var dep = new {
  
};

// VIASH END

// copy file
Console.WriteLine($"Copying '{par.input}' to '{par.output}'.");
File.Copy(par.input, par.output, true);

JavaScript

Running viash config inject effectively changes the contents of the script.

viash config inject config.vsh.yaml

The updated script.js now contains the following code:

const fs = require('fs');

// VIASH START
// The following code has been auto-generated by Viash.
let par = {
  'input': String.raw`file.txt`,
  'output': String.raw`output.txt`
};
let meta = {
  'name': String.raw`example_js`,
  'functionality_name': String.raw`example_js`,
  'resources_dir': String.raw`/tmp/viash_inject_example_js17154298320491473089`,
  'executable': String.raw`/tmp/viash_inject_example_js17154298320491473089/example_js`,
  'config': String.raw`/tmp/viash_inject_example_js17154298320491473089/.config.vsh.yaml`,
  'temp_dir': String.raw`/tmp`,
  'cpus': parseInt(String.raw`123`),
  'memory_b': parseInt(String.raw`123`),
  'memory_kb': parseInt(String.raw`123`),
  'memory_mb': parseInt(String.raw`123`),
  'memory_gb': parseInt(String.raw`123`),
  'memory_tb': parseInt(String.raw`123`),
  'memory_pb': parseInt(String.raw`123`),
  'memory_kib': parseInt(String.raw`123`),
  'memory_mib': parseInt(String.raw`123`),
  'memory_gib': parseInt(String.raw`123`),
  'memory_tib': parseInt(String.raw`123`),
  'memory_pib': parseInt(String.raw`123`)
};
let dep = {
  
};

// VIASH END

// copy file
console.log(`Copying '${par['input']}' to '${par['output']}'`)
fs.copyFile(par['input'], par['output'], (err) => {
  if (err) throw err;
});

Python

Running viash config inject effectively changes the contents of the script.

viash config inject config.vsh.yaml

The updated script.py now contains the following code:

import shutil

## VIASH START
# The following code has been auto-generated by Viash.
par = {
  'input': r'file.txt',
  'output': r'output.txt'
}
meta = {
  'name': r'example_python',
  'functionality_name': r'example_python',
  'resources_dir': r'/tmp/viash_inject_example_python11500170140718865543',
  'executable': r'/tmp/viash_inject_example_python11500170140718865543/example_python',
  'config': r'/tmp/viash_inject_example_python11500170140718865543/.config.vsh.yaml',
  'temp_dir': r'/tmp',
  'cpus': int(r'123'),
  'memory_b': int(r'123'),
  'memory_kb': int(r'123'),
  'memory_mb': int(r'123'),
  'memory_gb': int(r'123'),
  'memory_tb': int(r'123'),
  'memory_pb': int(r'123'),
  'memory_kib': int(r'123'),
  'memory_mib': int(r'123'),
  'memory_gib': int(r'123'),
  'memory_tib': int(r'123'),
  'memory_pib': int(r'123')
}
dep = {
  
}

## VIASH END

# copy file
print(f"Copying '{par['input']}' to '{par['output']}'.")
shutil.copyfile(par['input'], par['output'])

R

Running viash config inject effectively changes the contents of the script.

viash config inject config.vsh.yaml

The updated script.R now contains the following code:

## VIASH START
# The following code has been auto-generated by Viash.
# treat warnings as errors
.viash_orig_warn <- options(warn = 2)

par <- list(
  "input" = 'file.txt',
  "output" = 'output.txt'
)
meta <- list(
  "name" = 'example_r',
  "functionality_name" = 'example_r',
  "resources_dir" = '/tmp/viash_inject_example_r4247542873219032659',
  "executable" = '/tmp/viash_inject_example_r4247542873219032659/example_r',
  "config" = '/tmp/viash_inject_example_r4247542873219032659/.config.vsh.yaml',
  "temp_dir" = '/tmp',
  "cpus" = as.integer('123'),
  "memory_b" = bit64::as.integer64('123'),
  "memory_kb" = bit64::as.integer64('123'),
  "memory_mb" = bit64::as.integer64('123'),
  "memory_gb" = bit64::as.integer64('123'),
  "memory_tb" = bit64::as.integer64('123'),
  "memory_pb" = bit64::as.integer64('123'),
  "memory_kib" = bit64::as.integer64('123'),
  "memory_mib" = bit64::as.integer64('123'),
  "memory_gib" = bit64::as.integer64('123'),
  "memory_tib" = bit64::as.integer64('123'),
  "memory_pib" = bit64::as.integer64('123')
)
dep <- list(
  
)


# restore original warn setting
options(.viash_orig_warn)
rm(.viash_orig_warn)

## VIASH END

# copy file
cat("Copying '", par$input, "' to '", par$output, "'.\n", sep = "")
file.copy(par$input, par$output)

Scala

Running viash config inject effectively changes the contents of the script.

viash config inject config.vsh.yaml

The updated script.scala now contains the following code:

import java.nio.file.StandardCopyOption.REPLACE_EXISTING
import java.nio.file.Files
import java.nio.file.Paths

// VIASH START
// The following code has been auto-generated by Viash.
case class ViashPar(
  input: Option[String],
  output: Option[String]
)
val par = ViashPar(
  Some("""file.txt"""),
  Some("""output.txt""")
)
case class ViashMeta(
  name: Option[String],
  functionality_name: Option[String],
  resources_dir: Option[String],
  executable: Option[String],
  config: Option[String],
  temp_dir: Option[String],
  cpus: Option[Int],
  memory_b: Option[Long],
  memory_kb: Option[Long],
  memory_mb: Option[Long],
  memory_gb: Option[Long],
  memory_tb: Option[Long],
  memory_pb: Option[Long],
  memory_kib: Option[Long],
  memory_mib: Option[Long],
  memory_gib: Option[Long],
  memory_tib: Option[Long],
  memory_pib: Option[Long]
)
val meta = ViashMeta(
  Some("""example_scala"""),
  Some("""example_scala"""),
  Some("""/tmp/viash_inject_example_scala6130220668793971010"""),
  Some("""/tmp/viash_inject_example_scala6130220668793971010/example_scala"""),
  Some("""/tmp/viash_inject_example_scala6130220668793971010/.config.vsh.yaml"""),
  Some("""/tmp"""),
  Some("""123""".toInt),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong),
  Some("""123""".toLong)
)
case class ViashDep(
  
)
val dep = ViashDep(
  
)

// VIASH END

// copy file
println(s"Copying '${par.input}' to '${par.output}'.")
val fileIn = Paths.get(par.input)
val fileOut = Paths.get(par.output)
Files.copy(fileIn, fileOut, REPLACE_EXISTING)

Runtime parameters in par

The par object (or par_ environment variables in Bash) will contain argument values passed at runtime. For example, passing --input foo.txt will result in a par["input"] being equal to "foo.txt".

Tip

Try adding more arguments with different file types to see what effect this has on the resulting placeholder.

Meta variables in meta

Meta-variables offer information on the runtime environment which you can use from within your script.

• cpus (integer): The maximum number of (logical) cpus a component is allowed to use. By default, this value will be undefined.

• config (string): Path to the processed Viash config YAML. This file is usually called .config.vsh.yaml and resides next to the wrapped executable (see below). This YAML file is useful for doing some runtime introspection of the component for writing generic unit tests.

• executable (string): The executable being used at runtime; that is, the wrapped script. This variable is used in unit tests.

• functionality_name (string): The name of the component, useful for logging.

• memory_* (long): The maximum amount of memory a component is allowed to allocate. The following denominations are provided: memory_b, memory_kb, memory_mb, memory_gb, memory_tb, memory_pb. By default, this value will be undefined.

• resources_dir (string): Path to where the resources are stored. See the previous section on how to define resources.

• temp_dir (string): A temporary directory in which your script is allowed to create new temporary files / directories. By default, this will be set to the VIASH_TEMP environment variable. When the VIASH_TEMP variable is undefined, /tmp is used instead.

cpus (integer)

This field specifies the maximum number of (logical) cpus a component is allowed to use. This is useful when parallellizing your component in such a way that integrates very nicely with pipeline frameworks such as Nextflow. Below is an example usage of the cpus meta-variable.

Bash

#!/bin/bash

## VIASH START
par_input="path/to/file.txt"
par_output="output.txt"
meta_cpus=10
## VIASH END

# Pass number of cores to the popular_software_tool. Set the default to 1.
./popular_software_tool --ncores ${meta_cpus:-1}

C#

No example available yet.

JavaScript

No example available yet.

Python

from multiprocessing import Pool

## VIASH START
par = {}
meta = {"cpus": 1}
## VIASH END

def my_fun(x):
    return x + "!"
my_data = ["hello", "world"]

with Pool(processes=meta.get("cpus", 1)) as pool:
    out = pool.map(my_fun, my_data)

R

library(furrr)

## VIASH START
par <- list()
meta <- list(
  cpus = 1L
)
## VIASH END

if (is.null(meta$cpus)) meta$cpus <- 1
plan(multisession, workers = meta$cpus)

my_data <- c("hello", "world")
out = future_map(
  my_data, 
  function(x) {
    paste0(x, "!")
  }
)

Scala

import scala.collection.parallel._
import java.util.concurrent.ForkJoinPool

// VIASH START
// ...
// VIASH END

val pc = mutable.ParArray(1, 2, 3)
val numCores = meta.cores.getOrElse(1)
pc.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(numCores))
pc map { _ + 1 }

You can set the number of cores in your component using any of the following approaches:

# as a parameter of viash run
viash run config.vsh.yaml --cpus 10 -- <my component arguments>

# as a parameter of viash test
viash test config.vsh.yaml --cpus 10

# or as a parameter of the executable
viash build config.vsh.yaml -o output
output/my_executable ---cpus 10
#                     ↑ notice the triple dash