从 pandas csv 读取时，我可以使用制表符或逗号作为分隔符吗？

Question

I have csv files. Some are comma delimited, and some are tab delimited.

df = pd.read_csv(data_file, sep='\t')

Is there a way to specify either tab or comma as delimiter when using pd.read_csv()? Or, is there a way to automatically detect whether the file is tab or comma delimited? If I know that, I can use different sep='' paramters when reading the file.

Answer 1

Recently I had a similair problem, I ended up using a different method but I explored using this Class.

You can use the standard CSV module from the standard library. Specifically using the Sniffer Class

from the documentation

"Sniffs" the format of a CSV file (ie delimiter, quotechar) Returns a Dialect object.

you can return the dialect object then pass dialect.delimiter to the sep arg in pd.read_csv

'text_a.csv'

cola|colb|col
A|B|C
E|F|G
A|B|C
E|F|G

---

'text_b.csv'

cola\tcolb\tcol
A\tB\tC
E\tF\tG
A\tB\tC
E\tF\tG
A\tB\tC

---

from csv import Sniffer

sniffer = Sniffer()

def detect_delim(file,num_rows,sniffer):

    with open(file,'r') as f:
        for row in range(num_rows):
            line = next(f).strip()
            delim = sniffer.sniff(line)
    print(delim.delimiter) # ideally you should return the dialect object - just being lazy.

---

detect_delim(file='text_a.csv',num_rows=5,sniffer=sniffer)
'|'
detect_delim(file='text_b.csv',num_rows=5,sniffer=sniffer)
'\t'

Answer 2

I'd just read the first row and see which gives you more columns:

import pandas as pd
tab = pd.read_csv(data_file, nrows=1, sep='\t').shape[1]
com = pd.read_csv(data_file, nrows=1, sep=',').shape[1]
if tab > com:
    df = pd.read_csv(data_file, sep='\t')
else:
    df = pd.read_csv(data_file, sep=',')

Answer 3

Is this useful ? You can used python regex parser with read_csv and specify different delimiters.

Answer 4

Ask the user to specify how the file is formatted if you don't expect to be able to determine from the file contents itself.

Eg a flag of some sort as --tab-delimited-file=true and then you flip the separator based on their input.