Read file, parse by tabs and insert into mysql database with Bash Script

Question

I am doing a bash script to read a file line by line, parse and insert it in mysql database.

The script is the following:

#!/bin/bash

echo "Start!"

while IFS='     ' read -ra ADDR;
do
   for line in $(cat filename)
   do
      regex='(\d\d)-(\d\d)-(\d\d)\s(\d\d:\d\d:\d\d)'
      if [[$line=~$regex]]
      then
         $line='20$3-$2-$1 $4';
      fi
      echo "insert into table (time, total, caracas, anzoategui) values('$line', '$line', '$line', $
   done | mysql -uuser -ppassword database;
done < filename

The file 'filename' with data is something like this:

15/08/13 09:34:38       17528                                   5240    399     89      460     159     1107    33240
15/08/13 09:42:57       17528                                   5240    399     89      460     159     1107    33240
15/08/13 10:20:03       17492                                   5217    394     89      459     159     1101    33245
15/08/13 11:20:02       17521                                   5210    402     90      462     158     1112    33249
15/08/13 12:20:04       17540                                   5209    396     90      459     160     1105    33258

And its dropping this:

Answer 1

I would recommend creating your sql file using awk and then sourcing it from mysql . Re-direct the output once it looks ok to you into a new file (say insert.sql ) and source it from mysql command line. Something like this:

$ cat file
15/08/13 09:34:38       17528                                   5240    399     89      460     159     1107    33240
15/08/13 09:42:57       17528                                   5240    399     89      460     159     1107    33240
15/08/13 10:20:03       17492                                   5217    394     89      459     159     1101    33245
15/08/13 11:20:02       17521                                   5210    402     90      462     158     1112    33249
15/08/13 12:20:04       17540                                   5209    396     90      459     160     1105    33258

$ awk -F'[/ ]+' -v q="'" '{print "insert into table (time, total, caracas, anzoategui) values ("q"20"$3"-"$2"-"$1" "$4q","q$5q","q$6q","q$7q");"}' file
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 09:34:38','17528','5240','399');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 09:42:57','17528','5240','399');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 10:20:03','17492','5217','394');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 11:20:02','17521','5210','402');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 12:20:04','17540','5209','396');

Answer 2

Use the LOAD DATA statement. Do any transformations you have to on your file first, then

LOAD DATA LOCAL INFILE 'filename' INTO table (time, total, caracas, anzoategui)

Tab separated fields is the default.

Answer 3

Bash uses POSIX character classes. So instead of \\d you could use [0-9] or [[:digit:]] to match a digit.

Also, there must be spaces in between the brackets and operator in your regex test. So [[$line=~$regex]] would be fixed by doing [[ $line =~ $regex ]] .

In order to access the regular expressions enclosed in parentheses, you must access the builtin array variable BASH_REMATCH . So, to access the first match, you'd reference ${BASH_REMATCH[1]} , the second ${BASH_REMATCH[2]} , and so on.