优化shell脚本（bash）以提高性能

Question

I have a bash script which I use to process a text file:

#/bin/bash

dos2unix sourcefile.txt

cat sourcefile.txt | grep -v '\/' | grep -v '\-\-' | grep -v '#' | grep '[A-Za-z]\*' > modified_sourcefile.txt

mv modified_sourcefile.txt sourcefile.txt
#
# Read the sourcefile file one line by line and iterate...
#

while read line
do

 echo $line | grep -v '\/' | grep -v '\-\-' | grep -v '#'
 if [ $? -eq 0 ]
 then

   # echo "Current Line is " $line ";"
    char1=`echo ${line:0:1}`
   # echo "1st char is " $char1

  if [ -n "$char1" ]
   # if a blank-line, neglect the line.
    then
        # echo "test passed"
        var1=`echo $line | cut -d '*' -f 1`
    var2=`echo $line | cut -d '*' -f 1`
    var3=`echo $line | cut -d - -f 1`
        var4=`echo $line | cut -d '*' -f 1`
        var5=`echo $line | cut -d '*' -f 2`
        var6=`echo $line | cut -d - -f 1`
        var7=`echo $line | cut -d '*' -f 3 `


        table1sql="INSERT IGNORE INTO table1 (id,name,active_yesno,category,description,
           last_modified_by,last_modified_date_time) SELECT ifnull(MAX(id),0)+1,'$var1',1,
           '$var2','$var3','admin',NOW() FROM table1;"

    echo $table1sql >> result.txt


    privsql="INSERT IGNORE INTO table2 (id,name,description,active_yesno,group_code,
             last_modified_by,last_modified_date_time) SELECT ifnull(MAX(id),0)+1,'$var1',
         '$var3',1,'$var2','admin',NOW() FROM table2;"

    echo $privsql >> result.txt     


    table1privmapsql="INSERT IGNORE INTO table1_table2_map (id,table1_id,table2_id,
                  last_modified_by,last_modified_date_time) SELECT ifnull(MAX(id),0)+1,
                  (select id from table1 where name='$var1'),(select id from table2 where name='$var1'),'admin',NOW() FROM table1_table2_map;"
    echo $table1privmapsql >> result.txt

        privgroupsql="INSERT IGNORE INTO table2_group (id,name,category,active_yesno,last_modified_by,
                      last_modified_date_time) SELECT ifnull(MAX(id),0)+1,'tablegrp','$pgpcode',1,'admin',NOW() FROM table2_group;"

        echo $privgroupsql >> result.txt


    privprivgrpsql="INSERT IGNORE INTO table2_table2group_map (id,table2_id,table2_group_id,
                        last_modified_by,last_modified_date_time) SELECT ifnull(MAX(id),0)+1,
                        (select id from table2 where name='$var1'),(select id from table2_group where name='tablegrp'),'admin',NOW() FROM table2_table2group_map;"
        echo $privprivgrpsql >> result.txt              

    rolesql="INSERT IGNORE INTO role (id,name,active_yesno,security_domain_id,last_modified_by,last_modified_date_time) 
                 SELECT (select ifnull(MAX(id),0)+1 from role),'$rolename',1, sd.id ,'admin',NOW() 
                 FROM security_domain sd WHERE sd.name = 'General';"

        echo $rolesql >> result.txt

    fi                  
 fi                        
done < "sourcefile.txt"

The thing is sourcefile.txt has over 11000 lines. So it takes about 25 min to complete :-( .

Is there a better way of doing it?

contents of sourcefile.txt:

AAA-something*LOCATION-some_where*ABC

Answer 1

To make this script faster you must minimize calls to external commands and use bash where is possible.

1. read this article to know what is useless use of commands.

2. read this article to know how to use bash to manipulate strings.

3. replace repeating values(var1, var2, var4) assignment to single value.

While optimizing cut you can replace

var1=`echo $line | cut -d '*' -f 1`

to

var1="${line%%\**}"

And

var5=`echo $line | cut -d '*' -f 2`

to

var5="${line%\**}"
var5="${var5##*\*}"

Maybe it not so human-readable, but works much faster than cut.

Also

 echo $line | grep -v '\/' | grep -v '\-\-' | grep -v '#'

can be replaced to something like that:

 if [[ "$line" =~ ([/#]|--) ]]; then :; else 
    # all code inside "if [ $? -eq 0 ]"
 fi

Answer 2

shell scripts are inherently slow, especially when they use a lot of external commands like yours. Biggest reason for this is because spawning external process is rather slow, and you do it a lot of times.

If you are really after high performance processing of your data, you should write Perl or Python script which would do what you need without ever spawning any external process: no dos2unix , no grep , no cut or anything like that.

Perl (and Python) are also perfectly capable of talking directly to database and inserting data, also without using external commands.

If you do it right, I predict that processing performance using Perl will be at least 100x faster than you have now.

If you are ok with Perl, you can start with something like this and adjust to your liking:

#!/usr/bin/perl -w

use strict;
use warnings;

open FILE, "sourcefile.txt" or die $!;
open RESULT, ">>result.txt" or die $!;
while (my $line = <FILE>) {
    # ignore lines with /, -- or #: 
    next if $line =~ m{/|--|#};
    my ($var1, $var2, $var3, $var4, $var5) =
        ($line =~ /^(\w+)-(\w+)\*(\w+)-(\w+)\*(\w+)/);
    # ignore line if regex did not match:
    next unless $var1 and $var2 and $var3 and $var4 and $var5;
    print RESULT "some sql stmt. using $var1, $var2, etc";
    print RESULT "some other sql using $var1, $var2, etc";
    # ...
}
close RESULT;
close FILE;

Answer 3

Before optimizing, profile! Learn how to use the time command. Find out which part of your script takes the most time, and put your effort there.

Having said that, I would think that having multiple passes of grep will slow things down a bit.

This:

cat sourcefile.txt | grep -v '\/' | grep -v '\-\-' | grep -v '#' | grep '[A-Za-z]\*'

can be replaced by this:

grep '[A-Za-z]\*' sourcefile.txt | grep -v -e '\/' -e '\-\-' -e '#'