如何在 Raku 中模拟 wc -l

Question

In perl 5, you can emulate wc -l using oneliner:

perl -lnE 'END {say $.}' test.txt

How to implement this functionality on Raku

If you try to implement this:

raku -e 'say "test.txt".IO.open.lines.elems'

it turns out to be slow and uses a lot of memory

Information for reproduce:

$ wget http://eforexcel.com/wp/wp-content/uploads/2017/07/1500000%20Sales%20Records.zip
$ unzip "1500000 Sales Records.zip"
$ mv "1500000 Sales Records.csv" part.txt
$ for i in `seq 1 10`; do cat part.txt >> test.txt ; done
$ du -sh test.txt
1.8G    test.txt

$ time wc -l test.txt
15000000 test.txt

real    0m0,350s
user    0m0,143s
sys     0m0,205s

$ time perl -lnE 'END { say $. }' test.txt
15000001

real    0m1,981s
user    0m1,719s
sys     0m0,256s

$ time raku -e 'say "test.txt".IO.open.lines.elems'
15000001

real    2m51,852s
user    0m25,129s
sys     0m6,378s

# Using swap (maximum uses 2.2G swap):
# Before `raku -e ''`

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15009        1695       12604         107         708       12917
Swap:          7583           0        7583

# After `raku -e ''`

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15009         752       13923          72         332       13899
Swap:          7583         779        6804

# Swap not used
$ time raku -ne '++$ andthen END .say' test.txt
15000001

real    1m44,906s
user    2m14,165s
sys     0m0,653s

$ raku -v
This is Rakudo version 2019.11 built on MoarVM version 2019.11
implementing Perl 6.d.

Answer 1

One option that's still likely to be pretty slow compared to perl but worth comparing:

raku -ne '++$ andthen END .say' test.txt

The l command line option is redundant.

$ is an anonymous state scalar.

andthen tests that its lhs is defined, and if so, sets that value as the topic ( $_ ) and then evaluates its rhs.

END is similar to perl 's END . Note that it returns Nil to the andthen but that doesn't matter here because we're using the END 's statement for its side-effect.

Several things will impact this code's speed. Some things I can think of:

• Compiler startup overhead. Ignoring any modules being used, the raku compiler Rakudo has a startup overhead of about a tenth of a second on typical hardware compared to a fairly negligible one for perl .

• The notion of a "line". In perl , the default notion of line processing is reading a series of bytes, some of which represent a line end. In raku , the default notion of line processing is reading a UTF-8 string, some of which represents line ends. Thus perl only incurs the reading overhead of an ASCII (or Extended ASCII) decoder whereas raku incurs the reading overhead of a UTF-8 decoder.

• Compiler optimizations. perl is generally optimized to the max. It wouldn't surprise me if perl -lnE 'END {say $.}' test.txt takes advantage of some clever optimizations. In contrast, work on Rakudo optimization is still in its early days relatively speaking.

The only things I think anyone can do about the first and last of the three points I've mentioned above are to wait N years and/or contribute to the compiler's improvement.

There will be a way to work around raku's UTF-8-by-default. Perhaps something like the following is already doable and significantly faster than raku's default, at least ignoring the overhead of using a module called foo :

raku -Mfoo -ne '++$ andthen END .say' test.txt

where module foo switches the default encoding for file I/O to ASCII or whatever from the available encodings .

I haven't checked that this is actually doable in current Rakudo but would be surprised if were not.