Python-走思想目录来处理csv文件并保存它们

Question

I want to walk through a list of csv files inside folders, perform some calculation (always the same) on each file, and save a new file for each one.

files have data structured in this manner:

"[Couplet 10 : Jul]
C'est 1.3.5 sur la plaque
Fais ton biz coupe ta plaque
C'est JU, JU , JUL qui débarque
Pour mes blancs , beurres et blacks
Passe moi un stunt pour voir si sa cabre
Embrouilles sur le sable , cocotiers sur la sappe
Je dors pas je suis tout pâle, je dis pas que je suis 2Pac
Je dis pas lui je vais le tuer si j'ai même pas 2 balles
C'est pour ceux qui XXX fais gaffe les shmits l'impact
Son anti B.D.H anti tapette",1

(...)

So far I have:

match = "^[\(\[].*?[\)\]]"
for d in directories:
        dir = os.path.join(data_dir, d)
        files_ = [os.path.join(dir, f) 
                      for f in os.listdir(dir) 
                      if f.endswith(".csv")]
        for f in files_:
            with open(f, 'rb') as f1, open('out.csv', 'wb') as out_file:
                reader = csv.reader(f1, delimiter='\t')
                for item in list(reader):
                item = re.sub(match, ' ', item, flags=re.MULTILINE)      
                out_file.write(item)

but I get this traceback:

File "process_csv.py", line 75, in load_data
    item = re.sub(match, ' ', item, flags=re.MULTILINE)      
  File "/Users/username/anaconda/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

what is the best way of achieving this?

Answer 1

According to the re docs , re.sub expects third parameter as a string. But list(reader) returns list of lists with CSV fields, not strings. So you need to extract string from this lists and pass it to re.sub :

item = re.sub(match, ' ', item[0], flags=re.MULTILINE)

or whatever index you need to use in the calculations.

To understand it better, try:

test.csv: 
a 
b 
c

>>> f = open('test.csv')
>>> reader = csv.reader(f)
>>> list(reader)
[['a'], ['b'], ['c']]

UPDATE

To make it working on the real data example:

1. Set delimiter to " (by default) or change regex if quotes are important for processing.
2. Specify newline character as '' when opening files. In python 2 open doesn't accept newline argument, use io package instead. io file opening has the same signature in general. Explanation from CSV package documentation:

If newline='' is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use \\r\\n linendings on write an extra \\r will be added. It should always be safe to specify newline='', since the csv module does its own (universal) newline handling.

 with open(f, 'rb', newline='') as f1, open('out.csv', 'wb', newline='') as out_file:
    ...

3. It seems that substitution required for 1st column, so use item[0] for sub

Finally, corrected code:

import io

...

match = "^[\(\[].*?[\)\]]"
for d in directories:
    dir = os.path.join(data_dir, d)
    files_ = [os.path.join(dir, f) 
                  for f in os.listdir(dir) 
                  if f.endswith(".csv")]
    for f in files_:
        with io.open(f, 'rb', newline='') as f1, io.open('out.csv', 'wb') as out_file:
            reader = csv.reader(f1)
            writer = csv.writer(out_file) 
            for item in reader:
                writer.writerow([
                    re.sub(match, ' ', item[0], flags=re.MULTILINE),
                    item[1]
                ])