使用awk或sed打印用双引号引起来的CSV文件列

Question

I'm working on a csv file like the one below, comma delimited, each cell is enclosed in double quotes, but some of them contain double quote and/or comma inside double quote enclosure. The actual file contain around 300 columns and 200,000 rows.

"Column1","Column2","Column3","Column4","Column5","Column6","Column7"
"abc","abc","this, but with "comma" and a quote","18"" inch TV","abc","abc","abc"
"cde","cde","cde","some other, "cde" here","cde","cde","cde"

I'll need to remove some unless columns, and merge last few columns, instead of having "," in between them, I need </br> . and move second column to the end. Anything within the cells should be the same, with double quotes and commas as the original file. Below is an example of the output that I need.

"Column1","Column4","Column5","Column2"
"abc","18"" inch TV","abc</br>abc</br>abc","abc"
"cde","some other, "cde" here","cde</br>cde</br>cde","cde"

In this example I want to remove column3 and merge column 5, 6, 7.

Below is the code that I tried to use, but it is reading either double quote and/or comma, which is end of the row to be different than what I expected.

awk -vFPAT='([^,]*)|("[^"]+")' -vOFS=, '{print $1,$4,$5"</br>"$6"</br>"$7",$2}' inputfile.csv

sed -i 's@"</br>"@</br>@g' inputfile.csv

sed is used to remove beginning and ending double quote of a cell.

The output file that I'm getting right now, if previous field contains a double quote, it will consider that is the beginning of a cell, so the following values are often pushed up a column.

Other code that I have used consider every comma as beginning of a cell, so that won't work as well.

awk -F',' 'BEGIN{OFS=",";} {print $1,$4,$5"</br>"$6"</br>"$7",$2}' inputfile.csv

sed -i 's@"</br>"@</br>@g' inputfile.csv

Any help is greatly appreciated. thanks!

Answer 1

CSV is a loose format. There may be subtle variations in formatting. Your particular format may or may not be expressible with a regular grammar/regular expression. (See this question for a discussion about this.) Even if your particular formatting can be expressed with regular expressions, it may be easier to just whip out a parser from an existing library.

It is not a bash/awk/sed solution as you may have wanted or needed, but Python has a csv module for parsing CSV files. There are a number of options to tweak the formatting. Try something like this:

#!/usr/bin/python

import csv

with open('infile.csv', 'r') as infile, open('outfile.csv', 'wb') as outfile:
    inreader = csv.reader(infile)
    outwriter = csv.writer(outfile, quoting=csv.QUOTE_ALL)
    for row in inreader:
        # Merge fields 5,6,7 (indexes 4,5,6) into one
        row[4] = "</br>".join(row[4:7])
        del row[5:7]

        # Copy second field to the end
        row.append(row[1])

        # Remove second and third fields
        del row[1:3]

        # Write manipulated row
        outwriter.writerow(row)

Note that in Python, indexes start with 0 (eg row[1] is the second field). The first index of a slice is inclusive, the last is exclusive ( row[1:3] is row[1] and row[2] only). Your formatting seems to require quotes around every field, hence the quoting=csv.QUOTE_ALL . There are more options at Dialects and Formatting Parameters .

The above code produces the following output:

"Column1","Column4","Column5</br>Column6</br>Column7","Column2"
"abc","18"" inch TV","abc</br>abc</br>abc","abc"
"cde","some other, cde"" here""","cde</br>cde</br>cde","cde"

There are two issues with this:

• It doesn't treat the first row any differently, so the headers of columns 5, 6, and 7 are merged like the other rows.

• Your input CSV contains "some other, "cde" here" (third row, fourth column) with unescaped quotes around the cde . There is another case of this on line two, but it was removed since it is in column 3. The result contains incorrect quotes.

If these quotes are properly escaped, your sample input CSV file becomes

infile.csv (escaped quotes) :

"Column1","Column2","Column3","Column4","Column5","Column6","Column7"
"abc","abc","this, but with ""comma"" and a quote","18"" inch TV","abc","abc","abc"
"cde","cde","cde","some other, ""cde"" here","cde","cde","cde"

Now consider this modified Python script that doesn't merge columns on the first row:

#!/usr/bin/python

import csv

with open('infile.csv', 'r') as infile, open('outfile.csv', 'wb') as outfile:
    inreader = csv.reader(infile)
    outwriter = csv.writer(outfile, quoting=csv.QUOTE_ALL)
    first_row = True
    for row in inreader:
        if first_row:
            first_row = False
        else:
            # Merge fields 5,6,7 (indexes 4,5,6) into one
            row[4] = "</br>".join(row[4:7])
        del row[5:7]

        # Copy second field (index 1) to the end
        row.append(row[1])

        # Remove second and third fields
        del row[1:3]

        # Write manipulated row
        outwriter.writerow(row)

The output outfile.csv is

"Column1","Column4","Column5","Column2"
"abc","18"" inch TV","abc</br>abc</br>abc","abc"
"cde","some other, ""cde"" here","cde</br>cde</br>cde","cde"

This is your sample output, but with properly escaped "some other, ""cde"" here" .

This may not be precisely what you wanted, not being a sed or awk solution, but I hope it is still useful. Processing more complicated formats may justify more complicated tools. Using an existing library also removes a few opportunities to make mistakes.

Answer 2

This might be an oversimplification of the problem but this has worked for me with your test data:

cat /tmp/inputfile.csv | sed 's@\"\,\"@|@g' | sed 's@"</br>"@</br>@g' | awk 'BEGIN {FS="|"} {print $1 "," $4 "," $5 "</br>" $6 "</br>" $7 "," $2}'

Please not that I am on Mac probably that's why I had to wrap the commas in the AWK script in quotation marks.