Pandas - 试图将多个 .txt 文件存储在 a.csv 中

Question

I have a folder with about 500.txt files. I would like to store the content in a csv file, with 2 columns, column 1 being the name of the file and column 2 being the file content in string. So I'd end up with a CSV file with 501 rows.

I've snooped around SO and tried to find similar questions, and came up with the following code:

import pandas as pd
from pandas.io.common import EmptyDataError
import os


def Aggregate_txt_csv(path):
    for files in os.listdir(path):
            with open(files, 'r') as file:
                try: 
                    df = pd.read_csv(file, header=None, delim_whitespace=True)
                except EmptyDataError:
                    df = pd.DataFrame()
                
            return df.to_csv('file.csv', index=False)

However it returns an empty.csv file. Am I doing something wrong?

Answer 1

There are several problems on your code. One of them is that pd.read_csv is not opening file because you're not passing the path to the given file. I think you should try to play from this code

import os
import pandas as pd
from pandas.io.common import EmptyDataError

def Aggregate_txt_csv(path):
    files = os.listdir(path)
    df = []
    for file in files:
        try: 
            d = pd.read_csv(os.path.join(path, file), header=None, delim_whitespace=True)
            d["file"] = file
        except EmptyDataError:
            d = pd.DataFrame({"file":[file]})
        df.append(d)
    df = pd.concat(df, ignore_index=True)
    df.to_csv('file.csv', index=False)

Answer 2

• Use pathlib
  • Path.glob() to find all the files
  • When using path objects, file.stem returns the file name from the path.
• Use pandas.concat to combine the dataframes in df_list

from pathlib import Path
import pandas as pd

p = Path('e:/PythonProjects/stack_overflow')  # path to files
files = p.glob('*.txt')  # get all txt files

df_list = list()  # create an empty list for the dataframes
for file in files:  # iterate through each file
    with file.open('r') as f:
        text = '\n'.join([line.strip() for line in f.readlines()])  # join all rows in list as a single string separated with \n
        
    df_list.append(pd.DataFrame({'filename': [file.stem], 'contents': [text]}))  # create and append a dataframe


df_all = pd.concat(df_list)  # concat all the dataframes

df_all.to_csv('files.txt', index=False)  # save to csv

Answer 3

I noticed there's already an answer, but I've gotten it to work with a relatively simple piece of code. I've only edited the file read-in a little bit, and the dataframe is outputting successfully.

Link here

import pandas as pd
from pandas.io.common import EmptyDataError
import os


def Aggregate_txt_csv(path):
    result = []
    print(os.listdir(path))
    for files in os.listdir(path):
        fullpath = os.path.join(path, files)
        if not os.path.isfile(fullpath):
            continue

        with open(fullpath, 'r', errors='replace') as file:
            try:
                content = '\n'.join(file.readlines())
                result.append({'title': files, 'body': content})
            except EmptyDataError:
                result.append({'title': files, 'body': None})
            
    df = pd.DataFrame(result)
    return df

df = Aggregate_txt_csv('files')
print(df)
df.to_csv('result.csv')

Most importantly here, I am appending to an array so as not to run pandas' concatenate function too much, as that would be pretty bad for performance. Additionally, reading in the file should not need read_csv, as there isn't a set format for the file. So using '\n'.join(file.readlines()) allows you to read in the file plainly and take out all lines into a string.

At the end, I convert the array of dictionaries into a final dataframe, and it returns the result.

EDIT : for paths that aren't the current directory, I updated it to append the path so that it could find the necessary files, apologies for the confusion