在分隔文件中有效地转换 yyyymmdd

Question

I have large dat file (zipped) in pipe delimited format. The 6th column has date format yyyymmdd, which needs to be converted to yyyy-mm-dd, so that I can use bcp utility to load to a sql server.

The content of the file looks like (5.26million rows, uncompressed about 0.46G)

AA|AA|AAA|AAAAA|AAAA|20191001|AA|...
BB|BB|BBB|BBBBB|BBBB|20191001|BB|...

My question is why the command line awk is slower than the python script. Is there a faster way to do it in the command line with awk or sed? There are a total 14 columns per line. The python code split the line and stops at the 6th column. This may contribute to the difference. But columns after 6th are not much longer in general.

]$ time unzip -p /aa/bb/somedata.zip | awk 'BEGIN{FS=OFS="|"} { $6=substr($6,1,4) "-" substr($6,5,2) "-" substr($6,7,2);} 1' > data1.DAT
real 0m19.141s
user 0m20.219s

]$ time ~/test.py;
real 0m12.393s
user 0m10.458s

]$ time unzip -p /aa/bb/somedata.zip > data2.DAT
real 0m1.952s
user 0m1.619s

The python code looks like:

with zipfile.ZipFile('/aa/bb/somedata.zip', 'r') as zf:
    with zf.open(somedata.DAT, 'r') as f, open('/aa/bb/data2bcp.DAT', 'w') as wf:
        for line in TextIOWrapper(f):
            toks = line.split('|', 6)
            toks[5] = toks[5][0:4] +'-' + toks[5][4:6] + '-' + toks[5][6:8]
            wf.write('|'.join(toks))

After I remove the limit on split the run time of python is not much different

toks = line.split('|')

] $ time ~/test.py;
real 0m13.006s
user 0m12.164s

Answer 1

You may try below if you have gawk , compare performance of substr() and gensub() with your large dataset

$ cat infile
AA|AA|AAA|AAAAA|AAAA|20191001|AA|...
BB|BB|BBB|BBBBB|BBBB|20191001|BB|...   

$ awk  'BEGIN{FS=OFS="|"}{$6=gensub(/(....)(..)(..)/,"\\1-\\2-\\3","",$6)}1' infile
AA|AA|AAA|AAAAA|AAAA|2019-10-01|AA|...
BB|BB|BBB|BBBBB|BBBB|2019-10-01|BB|...