“Tokenizing a 1.2MB file, which was seen passing json to sqlite, is approximately 12,000x slower with this pattern.”
Zig rewrote the libc string functions in pure Zig. The new strspn measures the entire string before it does anything. Call it in a tokenizer loop and 40,000 JSON records take minutes. He only saw it on Linux, because that is where zig cc built his SQLite. Rewriting boring old C has a price. This time it was 12,000x.