在python中将tsv转换为tsv

Question

I have a tsv-file (tab-seperated) and would like to filter out a lot of data using python before I import it into a postgresql database. My problem is that I can't find a way to keep the format of the original file which is mandatory because otherwise the import processes won't work. The web suggested that I should use the csv library, but no matter what delimter I use I always end up with files in a different format than the origin, eg files, that contain a comma after every character or files, that contain a tab after every character or files that have all data in one row. Here is my code:

import csv
import glob

# create a list of all tsv-files in one directory
liste = glob.glob("/some_directory/*.tsv")

# go thru all the files
for item in liste:
    #open the tsv-file for reading and a file for writing   
    with open(item, 'r') as tsvin, open('/some_directory/new.tsv', 'w') as csvout:
    tsvin = csv.reader(tsvin, delimiter='\t')
    # I am not sure if I have to enter a delimter here for the outfile. If I enter "delimter='\t'" like for the In-File, the outfile ends up with a tab after every character
    writer = csv.writer(csvout)

# go thru all lines of the input tsv
    for row in tsvin:
        # do some filtering
        if 'some_substring1' in row[4] or 'some_substring2' in row[4]:
            #do some more filtering
            if 'some_substring1' in str(row[9]) or 'some_substring1' in str(row[9]):
            # now I get lost...
            writer.writerow(row)    

Do you have any idea what I am doing wrong? The final file has to have a tab between every field and some kind of line break at the end.

Answer 1

Somehow you are passing a string to w.writerow() , not a list as expected.

Remember that strings are iterable; each iteration returns a single character from the string. writerow() simply iterates over its argument writing each item separated by the delimiter character (by default a comma). So if you pass a string to writerow() it will write each character from the string separated by the delimiter.

How is it that row is a string? It could be that the delimiter for the input file is incorrect - perhaps the file does not use tabs but has fixed field widths using runs of spaces as the delimiter.

You can check whether the reader is correctly parsing your file by printing out the value of row :

for row in tsvin:
    print(row)
    ...

If the file is being correctly parsed, expect to see that row is a list, and that each element of the list corresponds to a column/field from the file.

If it is not parsing correctly then you might see that row is a string, or that it's a list but the fields are empty and/or out of place.

It would be helpful if you added a sample of your input file to the question.