I Wasted Weeks Hand Optimizing Assembly Because I Benchmarked on Random Data

“When I adjusted my benchmark to primarily generate small numbers, the speedup vanished entirely.”

A developer spent two weeks hand-optimizing a VarInt encoder in assembly, got impressive benchmark numbers, then discovered the optimization only helped on random 64-bit integers. Real world data is mostly small numbers like zip codes and page counts. The Java implementation it was supposed to beat performed fine on realistic inputs. The whole thing got rolled back after A/B testing showed zero production improvement. A perfect cautionary tale about how benchmarks that don’t model actual workloads are just expensive entertainment.