Delete parentheses from a CSV file in Bash script

Question

I'm using a bash script to create a report for AdWords (the AdWords files are in python). I'm generating a "CAMPAIGN_PERFORMANCE_REPORT" (as CSV file), and one of the measures I take is "conversions". My problem is this - when I have more then thousand conversions, the number is with " on each side and a comma. Example:

2016-12-25,Campaign_A,Universal App Campaign,264.0
2016-12-25,Campaign_B,Universal App Campaign,"1,535.0"
2016-12-25,Campaign_C,Universal App Campaign,"1,472.0"
2016-12-25,Campaign_D,Universal App Campaign,"1,378.0"
2016-12-25,Campaign_E,Universal App Campaign,382.0
2016-12-25,Campaign_F,Universal App Campaign,431.0

When I insert this data into MySQL the cell is divided in 2 and I get "1" in the conversions instead of 1535 (for instance).

So I need your help in one of these two issues:

1. Does anyone know how can I take the "conversions" field as Long and not as Double from the AdWords API?
2. If Not, How can I replace parenthesis (") and commas (,) in several files in the same folder in linux? Since I have a csv file for each AdWords account...

Thank you!

Answer 1

This is too long for a comment.

If you are loading data into MySQL, then you should be using load data infile .

This command has an option: fields optionally enclosed by , where you can specify the double quote character. This will treat commas between the delimiter character as part of the value, not a value separator.

You can review the documentation here .

Answer 2

You can run the file through a sed filter like this:

sed -r ':l s/"([0-9]+),/"\1/g; t l; s/"([0-9.]+)"$/\1/g' yourfile > convertedfile

It uses a two step approach to get rid of commas and quotes:

1. as long as there is a quote followed by a number ( [0-9.]+ ) followed by , the comma is removed: :ls/"([0-9]+),/"\\1/g; tl; :ls/"([0-9]+),/"\\1/g; tl; (this is a "label; remove comma; if something was removed goto label" - construct)
2. remove quotes around numbers ( [0-9.]+ ) at the end of a line ( $ )