Benchmarking is hard to do right. Recently Jason Switzer found an old benchmark of the smart match operator written by Michael Schwern. Jason Switcher comments:
If I were to publish a paper with such a gaping hole like that, it would never be taken serious. In each test, he’s generating a random number and a random character and storing the result! That’s not part of the test. The $needles should each be generated outside the timing loop. This is adding tons of additional instructions that are not considered part of the test. These results are basically useless and should be redone.
After doing his test, Jason concludes: “Those results are astonishing!”. Yes, it clearly shows that the grep solutions are 25% faster than using first. This is astonishing and surprising — and probably warrants an explanation.
Jason’s methodology consists of generating a single random array and needle for each piece of code he wants to benchmark. Running his benchmark on my machine shows that first is 15 times faster than smart match and about 28 times faster than grep. Of course, my random arrays might have been more biased against ‘first’ and randomly placing the needle at the start.
So Jason’s test are not to be taken seriously either. Even when benchmarking with random data, we have to benchmark all our code pieces against the same data and not just on one data set.
Update: I’ve been running some more benchmarks in the background. It seems that the general pattern is that ‘first’ from List::Util and any from List::MoreUtils is about 25% slower than grep, but with some spikes where ‘first’/'any’ is blazingly fast. Is being a native opcode that much faster than XS-code?