bash'while read line'效率与大文件

Question

I was using a while loop to process a task,

which read records from a big file about 10 million lines.

I found that the processing become more and more slower as time goes by.

and I make a simulated script with 1 million lines as blow, which reveal the problem.

but I still don't know why, how does the read command work?

seq 1000000 > seq.dat
while read s;
do
    if [ `expr $s % 50000` -eq 0 ];then
        echo -n $( expr `date +%s` - $A) ' ';
        A=`date +%s`;
    fi
done < seq.dat

The terminal outputs the time interval:

98 98 98 98 98 97 98 97 98 101 106 112 121 121 127 132 135 134

at about 50,000 lines,the processing become slower obviously.

Answer 1

Using your code, I saw the same pattern of increasing times (right from the beginning!). If you want faster processing, you should rewrite using shell internal features. Here's my bash version:

tabChar="   "  # put a real tab char here, of course
seq 1000000 > seq.dat
while read s;
do
    if (( ! ( s % 50000 ) )) ;then
        echo $s "${tabChar}" $( expr `date +%s` - $A) 
        A=$(date +%s);
    fi
done < seq.dat

edit fixed bug, output indicated each line was being processed, now only every 50000'th line gets the timing treatment. Doah!

was

  if ((  s % 50000 )) ;then

fixed to

  if (( ! ( s % 50000 ) )) ;then

output now echo ${.sh.version} = Version JM 93t+ 2010-05-24

50000
100000   1
150000   0
200000   1
250000   0
300000   1
350000   0
400000   1
450000   0
500000   1
550000   0
600000   1
650000   0
700000   1
750000   0

output bash

50000    480
100000   3
150000   2
200000   3
250000   3
300000   2
350000   3
400000   3
450000   2
500000   2
550000   3
600000   2
650000   2
700000   3
750000   3
800000   2
850000   2
900000   3
950000   2
800000   1
850000   0
900000   1
950000   0
1e+06    1

As to why your original test case is taking so long ... not sure. I was surprised to see both the time for each test cyle AND the increase in time. If you really need to understand this, you may need to spend time instrumenting more test stuff. Maybe you'd see something running truss or strace (depending on your base OS).

I hope this helps.

Answer 2

Read is a comparatively slow process, as the author of "Learning the Korn Shell" points out *. (Just above Section 7.2.2.1.) There are other programs, such as awk or sed that have been highly optimized to do what is essentially the same thing: read from a file one line at a time and perform some operations using that input.

Not to mention, that you're calling an external process every time you're doing subtraction or taking the modulus, which can get expensive. awk has both of those functionalities built in.

As the following test points out, awk is quite a bit faster:

#!/usr/bin/env bash

seq 1000000 | 
awk '
  BEGIN {
    command = "date +%s"
    prevTime = 0
  }
  $1 % 50000 == 0 {
    command | getline currentTime
    close(command)

    print currentTime - prevTime
    prevTime = currentTime
  }
'

Output:

1335629268
0   
0   
0   
0   
0   
0   
0   
0   
0   
0   
0   
0   
0   
0   
1   
0   
0   
0   
0

Note that the first number is equivalent to date +%s . Just like in your test case, I let the first match be.

Note

*Yes the author is talking about the Korn Shell, not bash as the OP tagged, but bash and ksh are rather similar in a lot of ways. ksh is actually a superset of bash. So I would assume that the read command is not drastically different from one shell to another.