Testing Wasmflow Components
August 10, 2022
The Value of Testing
Writing automated tests is one of the most valuable things you can do as a developer but the practice is often resisted, rejected, or pushed off repeatedly. It’s understandable. Testing code is hard. It’s extra work with no visible impact to the end user. I get it.
It’s cruel irony that tests are easiest to write at a project’s beginning which is also when they are easiest to ignore. A single developer can store all the behavior of small projects in their head at one time. That developer becomes the living test suite. Unfortunately, the cost of introducing tests increases exponentially the more untested code is written. By the time developers accept that they should write some tests, making the project’s code actually testable can be a monstrous undertaking. Effort aside, making untestable code testable can actually introduce new bugs.
If writing tests can cause bugs, takes more effort, and produces no tangible value to the end user, why are they so important?
Because modern software is never “done” – there’s always more to do. The next developer that cracks open a project will have none of the experience of the original team and no understanding of what to look out for. Automated tests are the OSHA-approved guard rails that keep code working and prevent injuries (er, bugs).
It was critical for us to make testing Wasmflow components as easy as possible.
Testing Wasmflow Components
Testing any Wasmflow component is as simple as gathering three things:
- The component name.
- The inputs to send.
- The values to assert on the output.
That’s all you need to start and you define your tests in YAML.
A test for a basic echo
component – a component that echos out what it receives – would look like this:
- component: echo
inputs:
input: "Hello world"
outputs:
- port: output
payload:
value: "Hello world"
We can define such an echo component in a flow definition:
version: 1
components:
echo:
flow:
- <>.input -> <>.output
Wasmflow components are all the same. This is an example of a flow-based component written as YAML, but testing a WebAssembly or any other component would act the exact same.
Run the test with the wasmflow
command:
$ wasmflow test manifest.wafl tests/echo.yaml
# wasmflow test for : tests/echo.yaml
1..3
# echo
ok 1 echo:output[output]
ok 2 echo:payload
ok 3 echo:total_outputs
You might have seen this output before without knowing it’s something standard. It’s TAP output, the Test Anything Protocol. Wasmflow uses TAP for its test results.
The Test Anything Protocol
The Test Anything Protocol (TAP) is a spec that defines a standard test output format. It’s a way of decoupling the test runner from the tools that process the output; visualizers, CI, reporting, et al. Anything that produces TAP output can be fed into anything that can read TAP.
To see how this works, install tap
via npm
:
$ npm install -g tap
Now pipe wasmflow test
through tap
. Remember to add a trailing dash (-
) to tap
to make it read from STDIN
:
$ wasmflow test manifest.wafl tests/echo.yaml | tap -
PASS TAP
✓ echo:output[output]
PASS TAP
✓ echo:payload
PASS TAP
✓ echo:total_outputs
🌈 SUMMARY RESULTS 🌈
Suites: 0 of 0 completed
Asserts: 3 passed, of 3
Time: 57.172ms
>
TAP is great because the same tests can have nice, readable output during development and terse, processable output for CI and build environments.
Handling Randomness in tests
Managing unpredictable behavior is a major struggle for test writers. Randomness can be anywhere, IDs, durations, dice rolls, critical hits, anywhere. Wasmflow encourages maximum determinism by generating a random seed for every transaction and every flow so components can generate reliably reproducibly random values.
The seed
is delivered via the <core>
component along with timestamp
, the time of the invocation.
Time is another source is randomness that makes programs difficult to predict, troubleshoot, and test. Wasmflow makes a single timestamp available to all components, the time the invocation started. This is usually good enough for timestamp-related needs and eliminates another source of indeterminism.
To use the seed, pipe the seed
into a component that needs one:
version: 1
components:
random-id:
instances:
UUID: wafl::rand::uuid
flow:
- <core>.seed -> UUID.seed
- UUID.output -> <>.output
Run the above manifest with wasmflow invoke
. The -o
flag tells wasmflow
to print the payload values directly.
$ wasmflow invoke random.wafl random-id -o
6db109e8-a07e-1442-1ad3-490bbb6a90ee
Repeated runs give you new UUIDs, as you’d expect.
$ for i in $(seq 1 10); do \
wasmflow invoke random.wafl random-id -o; \
done
b1c30125-6522-197b-c033-a12a51aed489
46820f40-af31-b646-6d80-225cbce3d540
55e8ab44-88d3-7002-caa3-5aba48a5542d
2ce4c34f-5ab7-020c-98f7-f4cf26120d19
17221536-ed2b-26f1-dbe0-2ef379c93c0d
b7be579f-7e54-37d7-ef97-bc2691a97202
9cc5eecd-0faa-0730-639a-a32eb80bc1a8
7fa54435-89ca-00c1-6479-71227978a3f5
eab7fbdf-1f28-e89b-e9ca-19aa7f053844
8fb1f62b-67f5-05ca-8a20-06320166fc2a
We don’t need to test that the UUID generator is generating valid UUIDs. That’s a unit test and falls within the scope of the UUID library in the UUID component. For the sake of our testing, we just want to know what IDs to use so we can write our tests without accounting for randomness.
Pass an explicit seed to wasmflow and see the behavior change (or not change…). Use the -s
option to pass in a static seed. The seed here is 90
:
$ wasmflow invoke random.wafl random-id -o -s 90
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
And again for effect:
$ for i in $(seq 1 10); do wasmflow invoke random.wafl random-id -o -s 90; done
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
150563a2-79d4-6cc4-2299-0bd2bd15ca5b
To write tests with a static seed, add a seed
field to each test:
- component: random-id
seed: 90
inputs: {}
outputs:
- port: output
payload:
value: 150563a2-79d4-6cc4-2299-0bd2bd15ca5b
Now our tests execute predictably every single run:
$ wasmflow test random.wafl tests/random.yaml
1..3 # wasmflow test for : tests/random.yaml
# random-id
ok 1 random-id:output[output]
ok 2 random-id:payload
ok 3 random-id:total_outputs
Wrap up
Tests are a critical part of long-lived software and Wasmflow makes them trivial to set up and run. Every Wasmflow component gets tested the same way regardless of whether it’s from a manifest like above, a WebAssembly module, a microservice, or anything else Wasmflow supports.
For more information on Wasmflow, head to wasmflow.com or its Github repository github.com/wasmflow/wasmflow.