wc -l的性能

Question

I ran the following command :

time for i in {1..100}; do find / -name "*.service" | wc -l; done

got a 100 lines of the result then :

real 0m35.466s user 0m15.688s sys 0m14.552s

I then ran the following command :

time for i in {1..100}; do find / -name "*.service" | awk 'END{print NR}'; done

got a 100 lines of the result then :

real 0m35.036s user 0m15.848s sys 0m14.056s

I precise I already ran find / -name "*.service" just before so it was cached for both commands.

I expected wc -l to be faster. Why is it not ?

Answer 1

other's have mentioned that you're probably timing find , not wc or awk . still, there may be interesting differences to explore between wc and awk in their various flavors.

here are the results I get:

Mac OS 10.10.5 awk    0.16m lines/second
GNU awk/gawk 4.1.4    4.4m  lines/second
Mac OS 10.10.5 wc     6.8m  lines/second
GNU wc 8.27          11m    lines/second

i didn't use find , but instead used wc -l or `awk 'END{print NR}' on a large text file (66k lines) in a loop.

i varied the order of the commands and didn't find any deviations large enough to change the rankings i reported.

LC_CTYPE=C had no measurable effect on any of these.

conclusions

1. don't use mac builtin command line tools except for trivial amounts of data.

2. GNU wc is faster than GNU awk at counting lines.

i use MacPorts GNU binaries. it would be interesting to see how Homebrew binaries compare. (i'm guessing they'd lose.)

Answer 2

Three things:

1. Such a small difference is usually not significant:

    0m35.466s - 0m35.036s = 0m0.43s or 1.2% 

2. Yet wc -l is faster (10x) than awk 'END{print NR}' .

    % time seq 100000000 | awk 'END{print NR}' > /dev/null real 0m13.624s user 0m14.656s sys 0m1.047s % time seq 100000000 | wc -l > /dev/null real 0m1.604s user 0m2.413s sys 0m0.623s 

3. My guess is that the hard drive cache holds the find results, so after the first run with wc -l , most of the reads needed for find are in the cache. Presumably the difference in times between the initial find with disk reads and the second find with cache reads, would be greater than the difference in run times between awk and wc .

   One way to test this is to reboot, which clears the hard disk cache, then run the two tests again, but in the reverse order , so that awk is run first. I'd expect that the first-run awk would be even slower than the first-run wc , and the second-run wc would be faster than the second-run awk .