How BXM Was Tested With Two Million Damaged Inputs, Sixteen Processes and a Killed Run

Thu, 22 Oct 2026 00:00:00 +0000

Each BXM tool makes one main promise. Canon promises that every value has exactly one encoding. Memo promises that it never hands you a wrong or half-restored output. Most of the testing went into trying to break those two promises, and the results are recorded in the repository’s test/results folder by a script anyone can run again.

Canon

The property behind Canon is short to state: if the decoder accepts a byte string, encoding the decoded value gives back exactly the same bytes. If that holds, no value can have a second accepted spelling.

Replaying 13 Commits Through Memo Cut Test Time From 228 to 79 Seconds and Turned Up Two Flaky Tests

Thu, 22 Oct 2026 00:00:00 +0000

BXM’s repository has two packages with their own test suites: Canon, tested with cargo test, and Memo, tested with go test. Most commits touch one of them, or neither. A docs change or a new JavaScript build script doesn’t change either suite’s result, yet a plain CI job runs both every time.

So we replayed the repository’s own history. All 13 commits from 0.1.0 to the end of 0.3.0 were checked out one after another, and both suites ran at each one. That happened twice: once plain, and once with each suite behind Memo. Canon’s suite takes canon/** and the spec’s test vectors as inputs. Memo’s takes memo/**. Memo needs a declared output, so each suite writes its log to a file and that file is the output.

Testing on BXM.net

How BXM Was Tested With Two Million Damaged Inputs, Sixteen Processes and a Killed Run

Canon

Replaying 13 Commits Through Memo Cut Test Time From 228 to 79 Seconds and Turned Up Two Flaky Tests