Hashing 100,000 JSON Lines Found No Duplicates, Hashing Them With Canon Found 9,900
Event pipelines get the same event twice. A producer times out, retries, and the retry goes through a different service that writes JSON its own way. The usual dedupe is a hash of each line, and it quietly misses all of these.
We generated a stream of 100,000 JSON lines from a fixed seed, 13.1 MB in all. 90,000 are distinct purchase events. The other 10,000 are resends of earlier events, written by two other producers. One sorts the keys and escapes non-ASCII, so “Zürich” arrives as Z\u00fcrich. The other reverses the key order and adds spaces. For a twist, 100 of the resends come from a producer that writes the quantity as a float: 2.0 where the original said 2.
| Method | Distinct records | Duplicates found |
|---|---|---|
| SHA-256 of each line as received | 100,000 | 0 |
| Canon hash of each parsed line | 90,100 | 9,900 |
| SHA-256 of Python’s sorted-key JSON | 90,100 | 9,900 |
Hashing the lines found nothing, since not one resend matched its original byte for byte. Canon found every resend whose data was the same. The 100 float resends stayed separate, because to Canon 2 and 2.0 are different values. That’s a choice, not an accident. If one producer writes quantities as floats, you want to know, and the fix belongs in that producer.
Then why not sorted-key JSON?
In this test it did exactly as well, and we put it in the table on purpose. Inside one Python program, json.dumps(sort_keys=True) is a fine dedupe key.
It stops being fine when the keys are made in more than one place. Python, JavaScript, Go and Java each write JSON with their own spacing, escaping and float formatting. That’s case study one again. A Canon hash comes out the same from the command, the C library and the Python and JavaScript packages, because they all run the same code. So a dedupe table filled by a JavaScript service and checked by a Python job still agrees, as long as both keep ints and floats the way the schema says. Python does that on its own. JavaScript has one number type, so it needs the one step case study one shows.
Speed
On a 2-CPU Linux machine, in one run, parsing each line with json.loads and taking its Canon hash with the Python package took 0.53 s for all 100,000 lines, about 187,000 records a second. The sorted-key JSON route took 0.80 s in the same run. Hashing the raw lines took 0.11 s and found nothing.
Run it yourself
The script is examples/case-studies/03-duplicates/run.py in the repository, and the recorded output is test/results/case-studies/03-duplicates.txt. It needs only the Python package.
More examples are on the use cases page.