The Value of Testing

Writing automated tests is one of the most valuable things you can do as a developer but the practice is often resisted, rejected, or pushed off repeatedly. It’s understandable. Testing code is hard. It’s extra work with no visible impact to the end user. I get it.

It’s cruel irony that tests are easiest to write at a project’s beginning which is also when they are easiest to ignore. A single developer can store all the behavior of small projects in their head at one time. That developer becomes the living test suite. Unfortunately, the cost of introducing tests increases exponentially the more untested code is written. By the time developers accept that they should write some tests, making the project’s code actually testable can be a monstrous undertaking. Effort aside, making untestable code testable can actually introduce new bugs.

If writing tests can cause bugs, takes more effort, and produces no tangible value to the end user, why are they so important?

Because modern software is never “done” – there’s always more to do. The next developer that cracks open a project will have none of the experience of the original team and no understanding of what to look out for. Automated tests are the OSHA-approved guard rails that keep code working and prevent injuries (er, bugs).

It was critical for us to make testing Wasmflow components as easy as possible.

Testing Wasmflow Components

Testing any Wasmflow component is as simple as gathering three things:

• The component name.
• The inputs to send.
• The values to assert on the output.

That’s all you need to start and you define your tests in YAML.

A test for a basic echo component – a component that echos out what it receives – would look like this:

- component: echo
  inputs:
    input: "Hello world"
  outputs:
    - port: output
      payload:
        value: "Hello world"

We can define such an echo component in a flow definition:

version: 1
components:
  echo:
    flow:
      - <>.input -> <>.output

Run the test with the wasmflow command:

$ wasmflow test manifest.wafl tests/echo.yaml

# wasmflow test for : tests/echo.yaml
1..3
# echo
ok 1 echo:output[output]
ok 2 echo:payload
ok 3 echo:total_outputs

You might have seen this output before without knowing it’s something standard. It’s TAP output, the Test Anything Protocol. Wasmflow uses TAP for its test results.

The Test Anything Protocol

The Test Anything Protocol (TAP) is a spec that defines a standard test output format. It’s a way of decoupling the test runner from the tools that process the output; visualizers, CI, reporting, et al. Anything that produces TAP output can be fed into anything that can read TAP.

To see how this works, install tap via npm:

$ npm install -g tap

Now pipe wasmflow test through tap. Remember to add a trailing dash (-) to tap to make it read from STDIN:

$ wasmflow test manifest.wafl tests/echo.yaml | tap -

 PASS  TAP
 ✓ echo:output[output]

 PASS  TAP
 ✓ echo:payload

 PASS  TAP
 ✓ echo:total_outputs


  🌈 SUMMARY RESULTS 🌈


Suites:   0 of 0 completed
Asserts:  3 passed, of 3
Time:   57.172ms
>

TAP is great because the same tests can have nice, readable output during development and terse, processable output for CI and build environments.

Handling Randomness in tests

Managing unpredictable behavior is a major struggle for test writers. Randomness can be anywhere, IDs, durations, dice rolls, critical hits, anywhere. Wasmflow encourages maximum determinism by generating a random seed for every transaction and every flow so components can generate reliably reproducibly random values.

The seed is delivered via the <core> component along with timestamp, the time of the invocation.

To use the seed, pipe the seed into a component that needs one:

version: 1
components:
  random-id:
    instances:
      UUID: wafl::rand::uuid
    flow:
      - <core>.seed -> UUID.seed
      - UUID.output -> <>.output

Run the above manifest with wasmflow invoke. The -o flag tells wasmflow to print the payload values directly.

$ wasmflow invoke random.wafl random-id -o

6db109e8-a07e-1442-1ad3-490bbb6a90ee

Repeated runs give you new UUIDs, as you’d expect.

$ for i in $(seq 1 10); do \
    wasmflow invoke random.wafl random-id -o; \
  done

b1c30125-6522-197b-c033-a12a51aed489
46820f40-af31-b646-6d80-225cbce3d540
55e8ab44-88d3-7002-caa3-5aba48a5542d
2ce4c34f-5ab7-020c-98f7-f4cf26120d19
17221536-ed2b-26f1-dbe0-2ef379c93c0d
b7be579f-7e54-37d7-ef97-bc2691a97202
9cc5eecd-0faa-0730-639a-a32eb80bc1a8
7fa54435-89ca-00c1-6479-71227978a3f5
eab7fbdf-1f28-e89b-e9ca-19aa7f053844
8fb1f62b-67f5-05ca-8a20-06320166fc2a

We don’t need to test that the UUID generator is generating valid UUIDs. That’s a unit test and falls within the scope of the UUID library in the UUID component. For the sake of our testing, we just want to know what IDs to use so we can write our tests without accounting for randomness.

Pass an explicit seed to wasmflow and see the behavior change (or not change…). Use the -s option to pass in a static seed. The seed here is 90:

$ wasmflow invoke random.wafl random-id -o -s 90

150563a2-79d4-6cc4-2299-0bd2bd15ca5b

And again for effect:

$ for i in $(seq 1 10); do wasmflow invoke random.wafl random-id -o -s 90; done

150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b

To write tests with a static seed, add a seed field to each test:

- component: random-id
  seed: 90
  inputs: {}
  outputs:
    - port: output
      payload:
        value: 150563a2-79d4-6cc4-2299-0bd2bd15ca5b

Now our tests execute predictably every single run:

$ wasmflow test random.wafl tests/random.yaml

1..3 # wasmflow test for : tests/random.yaml
# random-id
ok 1 random-id:output[output]
ok 2 random-id:payload
ok 3 random-id:total_outputs

Wrap up

Tests are a critical part of long-lived software and Wasmflow makes them trivial to set up and run. Every Wasmflow component gets tested the same way regardless of whether it’s from a manifest like above, a WebAssembly module, a microservice, or anything else Wasmflow supports.

For more information on Wasmflow, head to wasmflow.com or its Github repository github.com/wasmflow/wasmflow.