Unit Testing

To ensure that your components and namespaces works as expected during its development cycle, writing one or more tests is essential.
Viash supports unit testing, which is a software testing method by which individual units of source code are tested to determine whether they output data as expected.

Unit testing a single component

To write a unit test for a component, you need two things: a definition in the config file and a script that runs the executable and verifies its output.

This tutorial uses a sample component named md_url_checker to explain how to write a unit test for Viash. To follow along, start by creating a new folder named “testing” on your machine.
Now create a new file named Testfile.md and add the following contents:

# Test File

This is a simple markdown file with some hyperlinks to test if the component works correctly.
Some links to websites:

- [Google](https://www.google.com)
- [Reddit](https://www.reddit.com)
- [A broken link](http://microsoft.com/random-link)

Links that are relative to [viash.io](http://www.viash.io):

- You can [install viash here](/documentation/installation).
- It all starts with a script and a [config file](/documentation/reference/config/overview.html).

Next, either download or copy the text for the config file and script below and add them to the same folder:

Download config.vsh.yaml
Contents of config.vsh.yaml
functionality:
  name: md_url_checker
  description: Check if URLs in a markdown are reachable and create a text report with the results.
  arguments:                     
  - type: file
    name: --inputfile
    description: The input markdown file.
    required: true
    must_exist: true
  - type: string                           
    name: --domain
    description: The domain URL that gets inserted before any relative URLs. For example, "/documentation/intro" could be replaced with "https://my-website/documentation/intro" to create a valid URL.
  - type: file                           
    name: --output
    description: The path of the output text file that will contain the report.
    default: "output.txt"
    direction: output
  resources:
  - type: bash_script
    path: script.sh
  test_resources:
  - type: bash_script
    path: test.sh
  - path: Testfile.md
platforms:
  - type: native
  - type: docker
    image: bash:latest
    setup:
      - type: apk
        packages: [ curl ]
Download script.sh
Contents of script.sh
#!/usr/bin/env bash

### 1 ###

## VIASH START

par_inputfile="Testfile.md"
par_domain="https://viash.io"
par_output="output.txt"

## VIASH END

amount_of_errors=0

echo "Extracting URLs"

### 2 ###

# Extract the titles and URLs from the markdown file with sed and put them into arrays
readarray -t title_array <<<$(sed -rn 's@^.*\[(.*)\]\((.*)\).*$@\1@p' $par_inputfile)
readarray -t url_array <<<$(sed -rn 's@^.*\[(.*)\]\((.*)\).*$@\2@p' $par_inputfile)

# Get length of array
amount_of_urls=$(echo "${#url_array[@]}")

echo "Checking $amount_of_urls URLs"

# Clear file
>$par_output

### 3 ###

# Iterate over the array of titles and check each URL
for ((n = 0; n < ${#title_array[*]}; n++)); do
    title="${title_array[n]}"
    url="${url_array[n]}"

    ### 4 ###

    # If an URL doesn't start with 'http', add the domain before it
    if [[ $url != http* ]]; then
        url="$par_domain${url_array[n]}"
    fi

    echo "$(($n + 1)): $url"

    echo -e "Link name: $title" >>$par_output
    echo -e "URL: $url" >>$par_output

    ### 5 ###

    # Do a cURL and get the status code from the last response after following any redirects
    status_code=$(curl -ILs --max-redirs 5 $url | tac | grep -m1 HTTP)
    expected_code="200"

    # Check if status code obtained via cURL contains the expected code
    if [[ $status_code == *$expected_code* ]]; then
        echo "OK"
        echo -e "Status: OK, can be reached." >>$par_output
    else
        echo $status_code
        echo -e "Status: ERROR! URL cannot be reached. Status code: $status_code" >>$par_output
        amount_of_errors=$(($amount_of_errors + 1))
    fi

    echo -e "---" >>$par_output
done

echo ""
echo "$par_inputfile has been checked and a report named $par_output has been generated.
$amount_of_errors of $amount_of_urls URLs could not be resolved."

Configuration

To see where to define your unit tests, open up config.vsh.yaml file and take a look at the end of the functionality dictionary, between the path: script.sh and platforms: lines:

  test_resources:
  - type: bash_script
    path: test.sh
  - path: Testfile.md

The test_resources dictionary contains a reference to a test script and all of the files that need to be copied over in order to complete a test:

  • The type signifies what scripting language is used for performing the unit test, which doesn’t need to be the same language as the main script. The path points to the test script.
  • Every file path added straight into the tests dictionary will be copied over next to the temporary test directory. Any files that are necessary for the test to work correctly can be added here.

In the case of this example, test.sh will be the test script and Testfile.md is necessary as an input markdown file for the script to function.

Test script

To write the test itself, create a new file named test.sh in the testing folder and add this as its content:

set -ex # Exit the script when one of the checks fail. Output all commands.

# Check 1
echo ">>> Checking whether output is correct"

# Run md_url_checker component with its required inputs and output the results to test-output.txt
"./$meta_functionality_name" --inputfile Testfile.md > test-output.txt

[[ ! -f test-output.txt ]] && echo "Test output file could not be found!" && exit 1 # Check if test-output.txt exists
grep -q '1: https://www.google.com' test-output.txt # Did the script find the URL?
grep -q 'HTTP/2 404' test-output.txt  # Did the web request return a 404 for the page that doesn't exist?

# Check 2
echo ">>> Checking whether an output file was created correctly"

[[ ! -f output.txt ]] && echo "Output file could not be found!" && exit 1 # Check if output.txt exists
grep -q 'URL: https://www.google.com' output.txt # Was the URL written correctly in the report?
grep -q 'Status: ERROR! URL cannot be reached. Status code: HTTP/2 404' output.txt # Was the error written correctly in the report?
grep -q 'Link name: install viash here' output.txt # Was link name written correctly in the report?

echo ">>> Test finished successfully!"
exit 0 # Exit with a 0 code to note a success 
Tip

As you can see above, you can use the meta variable $meta_functionality_name to automatically get the name of the component (and the generated executable). This way, you can more reuse parts of your test scripts.

Tip

This script uses grep to search for files and strings. Make sure to read the comments to understand what is happening.

The bash script above is just a minimal example of how you can write a test. You can use any of the supported languages and your favorite testing framework if that’s your preference. The most important part is the last line, which uses a 0 exit code to signal a success.

Running a test

To run all tests defined in a config file, use the viash test command:

viash test config.vsh.yaml

Viash will now automatically build an executable and place it alongside the other defined resources in a temporary working directory. The output should look like this:

Running tests in temporary directory: '/tmp/viash_test_md_url_checker5828773777380597444'
====================================================================
+/tmp/viash_test_md_url_checker5828773777380597444/build_executable/md_url_checker ---setup
====================================================================
+/tmp/viash_test_md_url_checker5828773777380597444/test_test.sh/test.sh
>>> Checking whether output is correct
+ echo '>>> Checking whether output is correct'
+ ./md_url_checker --inputfile Testfile.md
+ [[ ! -f test-output.txt ]]
+ grep -q '1: https://www.google.com' test-output.txt
+ grep -q 'HTTP/2 404' test-output.txt
>>> Checking whether an output file was created correctly
+ echo '>>> Checking whether an output file was created correctly'
+ [[ ! -f output.txt ]]
+ grep -q 'URL: https://www.google.com' output.txt
+ grep -q 'Status: ERROR! URL cannot be reached. Status code: HTTP/2 404' output.txt
+ grep -q 'Link name: install viash here' output.txt
+ echo '>>> Test finished successfully!'
>>> Test finished successfully!
+ exit 0
====================================================================
[32mSUCCESS! All 1 out of 1 test scripts succeeded![0m
Cleaning up temporary directory

If the test succeeds, Viash writes the full output to the shell and deletes the temporary files. If there are any issues, the script stops and an error message will appear in red. In this case, you can inspect the temporary files to troubleshoot.

Tip

You can pass the --keep true argument to viash test to prevent viash from automatically deleting the temporary files in case of a success: viash test --keep true config.vsh.yaml.