我正在尝试使用 pd.read_csv() 将文件加载到 Python 中，但我无法理解文件的格式

Question

This is my very first question on stackoverflow, so I must beg your patience.

I believe there is something wrong with the format of a csv file I need to load into Python. I'm using a Jupyter Notebook. The link to the file is here . It is from the World Inequality Database data portal.

I'm pretty sure the delimiter is a semi-colon ( sep=";" ) because the bottom half of the data renders neatly when I specify this argument. However the first half of the text in the file seems to make no sense. I have no idea how to tell the pd.read_csv() function how to read it. I suspect the first half of the data simply has terrible formatting. I've also tried header=None and sep="|" to no avail.

Any ideas or suggestions would be very helpful. Thank you very much!

Answer 1

This is common with speadsheets. You have have some commentary, tables may be inserted all over the place. It looks great to the content creator, but the CSV is a mess. You need to preprocess the CSV to create clean content for your analysis. In this case, its easy. The content starts at canned header and you can split the file there. If that header changes, you'll get an error and now its just one more sleepless night figuring out what they've done.

import itertools

canned_header_line = "Variable Code;country;year;perc;agdpro999i;"\
    "npopul999i;mgdpro999i;inyixx999i;xlceux999i;xlcusx999i;xlcyux999i"

def scrub_WID_file(in_csv_filename, out_csv_filename):
    with open(in_csv_filename) as in_file,\
            open(out_csv_filename, 'w') as out_file:
        out_file.writelines(itertools.dropwhile(
            lambda line: line.strip() != canned_header_line,
            in_fp))
    if not os.stat.st_size:
        raise ValueError("No recognized header in " + in_csv_filename)