[英]Split timestamp column into two new columns in CSV using python and pandas
I have a large CSV file with over 210000 rows. 我有一个超过210000行的大型CSV文件。 I am new to python and pandas. 我是python和pandas的新手。 I would like to efficently loop through the timestamp column, Split the timestamp column into 2 new columns (date and time) and then format the new date column to %Y%m%d and delete the new time column .ie only write back to CSV file the new formated date column. 我想有效地遍历timestamp列,将timestamp列拆分为2个新列(日期和时间),然后将新的日期列格式化为%Y%m%d并删除新的time列。即,只写回CSV文件新格式的日期列。 How do you do this ? 你怎么做到这一点 ?
input file sample: 输入文件样本:
minit,timestamp,open,high,low,close
0,2009-02-23 17:32:00,1.2708,1.2708,1.2706,1.2706
1,2009-02-23 17:33:00,1.2708,1.2708,1.2705,1.2706
2,2009-02-23 17:34:00,1.2706,1.2707,1.2702,1.2702
3,2009-02-23 17:35:00,1.2704,1.2706,1.27,1.27
4,2009-02-23 17:36:00,1.2701,1.2706,1.2698,1.2703
5,2009-02-23 17:37:00,1.2703,1.2703,1.27,1.2702
6,2009-02-23 17:38:00,1.2701,1.2701,1.2696,1.2697
output file sample: 输出文件样本:
minit,date,open,high,low,close
0,20090223,1.2708,1.2708,1.2706,1.2706
1,20090223,1.2708,1.2708,1.2705,1.2706
2,20090223,1.2706,1.2707,1.2702,1.2702
3,20090223,1.2704,1.2706,1.27,1.27
4,20090223,1.2701,1.2706,1.2698,1.2703
5,20090223,1.2703,1.2703,1.27,1.2702
6,20090223,1.2701,1.2701,1.2696,1.2697
I started writing a sample code to accomplish this after i googled : 我在Google搜索后开始编写示例代码来完成此操作:
import csv
import itertools
import operator
import time
import datetime
import pandas as pd
from pandas import DataFrame, Timestamp
from numpy import *
def datestring_to_timestamp(str):
return time.mktime(time.strptime(str, "%Y-%m-%d %H:%M:%S"))
def timestamp_to_datestring(timestamp):
return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(timestamp))
def timestamp_to_float(str):
return float(datetime.datetime.strptime(str, '%Y-%m-%d %H:%M:%S').strftime("%s"))
def timestamp_to_intstring(str):
return datetime.datetime.strptime(str, '%Y-%m-%d %H:%M:%S').strftime("%s")
def timestamp_to_int(str):
return int(datetime.datetime.strptime(str, '%Y-%m-%d %H:%M:%S').strftime("%s"))
with open("inputfile.csv", 'rb') as input, open('outputfile.csv', 'wb') as output:
reader = csv.reader(input, delimiter = ',')
writer = csv.writer(output, delimiter = ',')
# Need to process loop or process the timestamp column
You can specify a date format string in the params to to_csv
which will output your dates how you like, no need to extract/convert/add new columns etc. 您可以在to_csv
的参数中指定日期格式字符串,该字符串to_csv
您的喜好输出日期,而无需提取/转换/添加新列等。
So load data using read_csv
: 因此,使用read_csv
加载数据:
df = pd.read_csv('mydata.csv', parse_dates=['timestamp']
In [15]:
df
Out[15]:
minit timestamp open high low close
0 0 2009-02-23 17:32:00 1.2708 1.2708 1.2706 1.2706
1 1 2009-02-23 17:33:00 1.2708 1.2708 1.2705 1.2706
2 2 2009-02-23 17:34:00 1.2706 1.2707 1.2702 1.2702
3 3 2009-02-23 17:35:00 1.2704 1.2706 1.2700 1.2700
4 4 2009-02-23 17:36:00 1.2701 1.2706 1.2698 1.2703
5 5 2009-02-23 17:37:00 1.2703 1.2703 1.2700 1.2702
6 6 2009-02-23 17:38:00 1.2701 1.2701 1.2696 1.2697
You can rename the column if you want at this stage, we can then pass the param date_format='%Y%m%d' to
to_csv` and this will just output the date portion to the csv, we can reload it and display what it has saved: 您可以根据需要在此阶段重命名该列,然后可以将参数date_format='%Y%m%d' to
to_csv`,这只会将日期部分输出到csv,我们可以重新加载它并显示什么它保存了:
In [19]:
df.rename(columns={'timestamp':'date'},inplace=True)
df.to_csv(r'c:\data\date.csv', date_format='%Y%m%d')
df1 = pd.read_csv(r'C:\data\date.csv', index_col=[0])
df1
Out[19]:
minit date open high low close
0 0 20090223 1.2708 1.2708 1.2706 1.2706
1 1 20090223 1.2708 1.2708 1.2705 1.2706
2 2 20090223 1.2706 1.2707 1.2702 1.2702
3 3 20090223 1.2704 1.2706 1.2700 1.2700
4 4 20090223 1.2701 1.2706 1.2698 1.2703
5 5 20090223 1.2703 1.2703 1.2700 1.2702
6 6 20090223 1.2701 1.2701 1.2696 1.2697
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.