尝试使用python删除csv文件中的多余定界符时，文本修饰符放错了位置

Question

I am trying to remove extra delimiters in between the data using a python script. I usually work with large data sets. For example:

"abc","def","ghi","jkl","mno","pqr"
"","","fds","dfs","adfadf","AAAA111"
"","","fds","df,s","adfadf","AAAA111"

If I run the script, the script will remove the extra delimiter in line 2 "df,s":

"abc","def","ghi","jkl","mno","pqr"
"","","fds","dfs","adfadf","AAAA111"
"","","fds","dfs","adfadf","AAAA111"

I was able to run the script properly for one data type ,but i noticed for few text qualifier data the text qualifiers got misplaced and the result came out like this:

"abc","def","ghi","jkl","mno","pqr"
"""","""""""""","""""fds""""","""""dfs""""","""""adfadf""""","AAAA111""""
"""","""""""""","""""fds""""","""""dfs""""","""""adfadf""""","AAAA111""""

The script is:

#export the data
# with correct quoting, and that you are stuck with what you have.
import csv
from csv import DictWriter

with open("big-12.csv", newline='') as people_file:
    next(people_file)
    corrected_people = []
    for person_line in people_file:
        chomped_person_line = person_line.rstrip()
        person_tokens = chomped_person_line.split(",")

        # check that each field has the expected type
        try:
            corrected_person = {
"abc":person_tokens[0],
"def":person_tokens[1],
"ghi":person_tokens[2],
"jkl":"".join(person_tokens[3:-3]),
"mno":person_tokens[-2],
"pqr":person_tokens[-1]

            }

            if not corrected_person["DR_CR"].startswith(
                    "") and corrected_person["DR_CR"] !="n/a":
                raise ValueError

            corrected_people.append(corrected_person)
        except (IndexError, ValueError):
            # print the ignored lines, so manual correction can be performed later.
            print("Could not parse line: " + chomped_person_line)

    with open("corrected_people.txt", "w", newline='') as corrected_people_file:
        writer = DictWriter(
            corrected_people_file,
            fieldnames=[
                "abc", "def", "ghi", "jkl", "mno", "pqr"
          ],delimiter=',',quoting=csv.QUOTE_ALL)
        writer.writeheader()
        writer.writerows(corrected_people)

This script is removing the extra delimiters in between but im having trouble with the text qualifiers. If the text qualifiers issue is revolved then it wll be of great help. Python version Python 3.6.0 :: Anaconda 4.3.1 (64-bit)

Answer 1

writer = DictWriter(
    corrected_people_file,
    fieldnames=[
        "abc", "def", "ghi", "jkl", "mno", "pqr"
    ],delimiter=',',quoting=csv.QUOTE_ALL)

QUOTE_ALL will force all fields to be quoted, and existing double quotes will be escaped with another double quote.

So try to use QUOTE_NONE or QUOTE_MINIMAL , or strip the fields of quotes before writing.

I'm having trouble with the text qualifiers

Also, quoting fields does not mean those are text vs number, the quotes are only there to allow embedded separator characters and can be around numerical fields as well.

---

In general it is better and safer to use a csv reader instead of using split() . With a csv reader the field "df,s" will be read correctly since it's quoted. You can then remove the , from that single field.