如果多个大文本文件太大而无法单独转换，如何将它们转换为一个 CSV 文件？

Question

I have several large .text files that I want to consolidate into one .csv file. However, each of the files is to large to import into Excel on its own, let alone all together.

I want to create a use pandas to analyze the data, but don't know how to get the files all in one place.

How would I go about reading the data directly into Python, or into Excel for a .csv file?

The data in question is the 2019-2020 Contributions by individuals file on the FEC's website.

Answer 1

You can convert each of the files to csv and the concatenate them to fom one final csv file

import pandas as pd
csv_path = 'pathtonewcsvfolder'                    # use your path
all_files=os.listdir("path/to/textfiles")
x=0
for filename in all_files:
    df = pd.read_fwf(filename)
    df.to_csv(os.path.join(csv_path,'log'+str(x)+'.csv'))
    x+=1
all_csv_files = glob.iglob(os.path.join(csv_path, "*.csv"))

converted_df=pd.concat((pd.read_csv(f) for f in all_csv_files), ignore_index=True)
converted_df.to_csv('converted.csv')