如何更改奇怪的CSV分隔符？

Question

I have a CSV file that I can't open in Excel.

The CSV delimiter is |~| , and at the end of a row it is |~~| .

I have some sample data:

Education|~|Name_Dutch|~|Name_English|~|Faculty|~~|International Business|~|MB|~|MB|~|ED|~~|

Where the Header part is: Education|~|Name_Dutch|~|Name_English|~|Faculty|~~|

And the Data/Row part is: International Business|~|MB|~|MB|~|ED|~~|

I need to find out how to change this CSV file in just a normal , comma separated value using a Python Script.

Answer 1

You can assist the built-in csv module + string.split() function:

import csv

content = """Education|~|Name_Dutch|~|Name_English|~|Faculty|~~|International Business|~|MB|~|MB|~|ED|~~|"""

# Or read it's content from a file 

with open('output.csv', 'w+') as f:
    writer = csv.writer(f)
    lines = content.split('|~~|')
    for line in lines:
        csv_row = line.split('|~|')
        writer.writerow(csv_row)

it will output a file named output.csv

Education,Name_Dutch,Name_English,Faculty
International Business,MB,MB,ED
""

---

When dealing with a csv file, I would prefer using the csv module instead of doing .replace('|~|', ',') because the csv module has build-in support for special characters such as ,

Answer 2

The custom delimiters you mention seem to be unique enough so you can just do a string.replace on them. Then just write out the file. The read and write section has all the details you need. https://docs.python.org/2/tutorial/inputoutput.html

Answer 3

import csv

in_name = 'your_input_name.csv'
outname = 'your_outpt_name.csv'

with open(in_name, newline='') as csvfile:
    csvreader = csv.reader(csvfile, delimiter='~', quotechar='|')
    with open(outname, "w", newline='') as outfile:
        csvwriter = csv.writer(outfile, quotechar='"', quoting=csv.QUOTE_NONNUMERIC)
        for row in csvreader:
            line = []
            for item in row:
                if item != "":
                    line.append(item)
                else:
                    csvwriter.writerow(line)
                    line = []

As csv.reader doesn't recognize "~~" as end of line, it converts it to "" , so for csv.writer we repeatedly prepare the part of list (obtained from csv.reader ) until "" is reached.

Answer 4

If the file is small, you can simply read its entire contents into memory and replace all weird delimiters found and then write a new version of it back out.

However if the file is large or you just want to conserve memory usage, it's also possible to read the file incrementally, a single character at-a-time, and do accomplish what needs to be done.

The csvfile argument to the csv.reader constructor "can be any object which supports the iterator protocol and returns a string each time its next() method is called."

This means the "object" can be a generator function or a generator expression. In the code below I've implemented a simple FSM ( Finite State Machine ) to parse the oddly formatted file and yield each line of output it detects. It may seem like a lot of code, but operate very simply so should be relatively easy to understand how it works:

import csv

def weird_file_reader(filename):
    """Generator that opens and produces "lines" read from the file while
       translating the sequences of '|~|' to ',' and '|~~|' to '\n' (newlines).
    """
    state = 0
    line = []
    with open(filename, 'rb') as weird_file:
        while True:
            ch = weird_file.read(1)  # read one character
            if not ch:  # end-of-file?
                if line:  # partial line read?
                    yield ''.join(line)
                break
            if state == 0:
                if ch == '|':
                    state = 1
                else:
                    line.append(ch)
                    #state = 0  # unnecessary
            elif state == 1:
                if ch == '~':
                    state = 2
                else:
                    line.append('|'+ch)
                    state = 0
            elif state == 2:
                if ch == '|':
                    line.append(',')
                    state = 0
                elif ch == '~':
                    state = 3
                else:
                    line.append('|~'+ch)
                    state = 0
            elif state == 3:
                if ch == '|':
                    line.append('\n')
                    yield ''.join(line)
                    line = []
                    state = 0
                else:
                    line.append('|~~'+ch)
                    state = 0
            else:
                raise RuntimeError("Can't happen")

with open('fixed.csv', 'wb') as outfile:
    reader = csv.reader((line for line in weird_file_reader('weird.csv')))
    writer = csv.writer(outfile)
    writer.writerows(reader)

print('done')