[英]Create a dataframe of csv files based on timestamp intervals
我相信我的問題真的很簡單,必須有一個非常簡單的方法來解決這個問題,但是由於我對 Python 很陌生,特別是 pandas,我無法自己解決。
我有數百個 csv 文件,格式如下: text_2014-02-22_13-00-00
所以格式是str_YY-MM-DD_HH-MI-SS 。 綜上所述,每個文件代表一小時的間隔。
我想根據我將使用Start_Time
和End_Time
設置的間隔從該間隔創建一個 dataframe 。 因此,例如,如果我將Start_Time
設置為 2014-02-22 21:40:00 並將End_Time
設置為 2014-02-22 22:55:00 (我使用的時間格式只是為了說明示例),然后我會得到一個 dataframe ,它包含上述間隔之間的數據,這些數據來自兩個不同的文件。
所以,我認為這個問題可能分為兩部分:
1 - 從文件名中讀取日期
2 - 根據我設置的時間間隔創建一個 dataframe。
希望我能做到簡潔而准確。 我非常感謝您對此的幫助! 也歡迎提出要查找的內容的建議
該解決方案有幾個不同的部分。
import os
import pandas as pd
import datetime
# step 1: create the path to folder
path_cwd = os.getcwd()
# step 2: manually 3 sample CSV files
df_1 = pd.DataFrame({'Length': [10, 5, 6],
'Width': [5, 2, 3],
'Weight': [100, 120, 110]
}).to_csv('text_2014-02-22_13-00-00.csv', index=False)
df_2 = pd.DataFrame({'Length': [11, 7, 8],
'Width': [4, 1, 2],
'Weight': [101, 111, 131]
}).to_csv('text_2014-02-22_14-00-00.csv', index=False)
df_3 = pd.DataFrame({'Length': [15, 9, 7],
'Width': [1, 4, 2],
'Weight': [200, 151, 132]
}).to_csv('text_2014-02-22_15-00-00.csv', index=False)
# step 3: save the contents of the folder to a list
list_csv = os.listdir(path_cwd)
list_csv = [x for x in list_csv if '.csv' in x]
print('here are the 3 CSV files in the folder: ')
print(list_csv)
# step 4: extract the datetime from filenames
def get_datetime_filename(str_filename):
'''
Function to grab the datetime from the filename.
Example: 'text_2014-02-22_13-00-00.csv'
'''
# split the filename by the underscore
list_split_file = str_filename.split('_')
# the 2nd part is the date
str_date = list_split_file[1]
# the 3rd part is the time, remove the '.csv'
str_time = list_split_file[2]
str_time = str_time.split('.')[0]
# combine the 2nd and 3rd parts
str_datetime = str(str_date + ' ' + str_time)
# convert the string to a datetime object
# https://chrisalbon.com/python/basics/strings_to_datetime/
# https://stackoverflow.com/questions/10663720/converting-a-time-string-to-seconds-in-python
dt_datetime = datetime.datetime.strptime(str_datetime, '%Y-%m-%d %H-%M-%S')
return dt_datetime
# Step 5: bring it all together
# create empty dataframe
df_master = pd.DataFrame()
# loop through each csv files
for each_csv in list_csv:
# full path to csv file
temp_path_csv = os.path.join(path_cwd, each_csv)
# temporary dataframe
df_temp = pd.read_csv(temp_path_csv)
# add a column with the datetime from filename
df_temp['datetime_source'] = get_datetime_filename(each_csv)
# concatenate dataframes
df_master = pd.concat([df_master, df_temp])
# reset the dataframe index
df_master = df_master.reset_index(drop=True)
# examine the master dataframe
print(df_master.shape)
# print(df_master.head(10))
df_master.head(10)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.