BXM Canon
BXM Canon is a binary encoding with exactly one valid encoding for every value. Same data in, same bytes out, on any machine. The decoder refuses everything else, so “are these two the same?” becomes “are these bytes equal?”.
The Problem It Solves
Sign a JSON object on one machine and check the signature on another, and it can fail even though nothing changed. One side wrote the keys in another order. Or 1.0 came out as 1. Or a string used an escape the other side didn’t. Same data, different bytes, broken hash.
The usual fix is a canonicalization step bolted on after the fact. Every language has to get every rule exactly right, and most bugs in this area are two implementations quietly disagreeing about one corner case.
How Canon Works
Canon fixes every choice in the format itself:
- Maps are written with their keys in byte order. Writing them in any other order, or with a key twice, is an error.
- Numbers use the shortest possible length prefix. A longer one that means the same number is an error.
- Integers and floats are different types.
1and1.0have different encodings, and neither is silently turned into the other. - Floats are the 8 IEEE bytes. NaN, infinity and negative zero aren’t values, because each of them has more than one reasonable meaning.
- Strings must be valid UTF-8. Canon doesn’t normalize Unicode for you, so two strings that look alike but differ in code points stay different.
The result is a format with one rule that matters: if the decoder accepts a byte string, encoding the decoded value gives back those exact bytes. That rule was tested on 2,000,000 random and damaged inputs. How it was tested.
Values
Canon carries what JSON carries, plus raw bytes:
| Type | Example in Canon text | Bytes |
|---|---|---|
| null | null |
00 |
| false, true | true |
02 |
| integer (64-bit) | -1 |
03 01 |
| float (64-bit) | 1.0 |
04 3f f0 00 00 00 00 00 00 |
| bytes | h"00ff" |
05 02 00 ff |
| string | "a" |
06 01 61 |
| array | [1,"a"] |
07 02 03 02 06 01 61 |
| map | {"b":1,"a":2} |
08 02 01 61 03 04 01 62 03 02 |
Canon text is ordinary JSON with three rules: a number without . or e is an integer, a repeated key is an error, and bytes are written as h"…" with hex digits.
What You Get
-
bxm-canon, a command-line tool to encode, decode, check, hash, diff and normalize. Reference. -
A Rust library, the one implementation everything else is built on.
-
A C API in the release archive, as a static and a shared library with
bxm_canon.h. Any language with a C foreign function interface can use the same code. -
The spec and conformance vectors: 30 values with their exact bytes and 30 byte strings that must be rejected, each with its reason. Another implementation that passes them agrees with this one.
-
A JavaScript package,
bxm-canon, since 0.2.0: the same Rust core compiled to WebAssembly, for Node 18+ and browsers, with TypeScript types and no dependencies. It passes the same conformance vectors. Install and use. -
A Python package,
bxm-canon, since 0.3.0: the same core through PyO3, with wheels for Linux, macOS and Windows and Python 3.9 and later. It encodes Python values directly, keepingint,floatandbytesapart. Install and use.
Speed
On a 1.1 MB document of 8,000 records, Canon encoded at about 1,100 MB/s and checked bytes at about 800 MB/s, where checking validates every rule without building the value in memory. Decoding into a full value tree ran at about 130 MB/s, most of it spent allocating strings and maps. The same document was 1.9 MB as JSON. Measured on a 2-CPU Linux machine, single thread.
Where It Fits
- Hashing structured data for caches, content addressing and deduplication
- Signing records, receipts, audit logs and attestations
- Comparing configuration or state across machines and languages
- Storing values where “same data” must mean “same file”