在python熊猫中读取和合并过去14天的csv文件

Question

I need to write a script that read all csv files with specific names from past 14 days (every day in the morning), but when I do concat this gives me a little cube (in jupyter-notebook), and that sign there is nothing.

def get_local_file(pdate, hour, path='/data/'):
        """Get date+hour processing file from local drive

       :param pdate: str Processing date
        :param hour: str Processing hour
        :param path: str Path to file location
        :return: Pandas DF Retrieved DataFrame
        """

        sdate = pdate + '-' + str(hour)
        for p_file in os.listdir(path):
            if fnmatch.fnmatch(p_file, 'ABC_*'+sdate+'*.csv'):
                return path+p_file

def get_files(pdate, path='/data/'):
    hours = [time(i).strftime('%H') for i in range(24)]
    fileList=[]
    for hour in hours:
        fileList.append(get_local_file(pdate, hour))
    return fileList

end_datetime = datetime.combine(date.today(), time(0, 0, 0))
proc_datetime = end_datetime - timedelta(days=14)
while proc_datetime <= end_datetime:
    proc_datetime += timedelta(days=1)
    a = get_files(str(proc_datetime.date()).replace('-', '_'))
    frame = pd.DataFrame()
    list_ = []
    for file_ in a:
        if file_ != None:
            df = pd.read_csv(file_,index_col=None, header=0, delimiter=';')
            list_.append(df)
            frame = pd.concat(list_)

I'm pretty sure that is possible to make code from while loop and below much simpler, but have no idea how to do it.

Answer 1

If you want to create a single dataframe from a bunch of .csv files you can do it this way:

• initialize an empty list before the loop
• loop over files, for every one read in a dataframe and append it to the list
• concatenate the list into a single dataframe after the loop

I did not check if your handling of dates and filenames is correct but here are the relevant changes to your code regarding the concatenation part:

end_datetime = datetime.combine(date.today(), time(0, 0, 0))
proc_datetime = end_datetime - timedelta(days=14)
list_ = []
while proc_datetime <= end_datetime:
    proc_datetime += timedelta(days=1)
    a = get_files(str(proc_datetime.date()).replace('-', '_'))
    for file_ in a:
        if file_ != None:
            df = pd.read_csv(file_, index_col=None, header=0, delimiter=';')
            list_.append(df)
frame = pd.concat(list_)

Answer 2

import pandas
import glob
csvFiles = glob.glob(path + "/data/*.csv")
list_ = []
for file in csvFiles:
    if ((datetime.combine(date.today(), time(0, 0, 0)) - datetime(*map(int, file.split("-")[1].split("_")))).days < 14)
        df = pandas.read_csv(file, index_col=None, header=0, delimiter=';')
        list_.append(df_f)
frame = pandas.concat(list_, ignore_index=True)
frame.to_csv("Appended File.csv")

Assuming the file path doesnot have any hyphen(-) characters in it.