[英]How to split a csv file into multiple csv based on a given criterion?
我需要根据给定的时间拆分几个 csv 文件。 在这些文件中,时间值以秒为单位,并在“时间”列中给出。
例如,如果我想在 0.1 秒内拆分aaa.csv
文件,则需要将第一组时间为 0.0 到 0.1(附件中的第 1 到 8 个)的行写入aaa1.csv
,然后将行写入时间大于 0.1 到 0.2(附件中的第 9 到 21 号)到aaa2.csv
等等......(基本上是给定时间的倍数)。
输出文件需要与输入文件同名,并在末尾加上一个数字。 并且输出文件需要写入不同的位置/文件夹。 时间值需要是一个变量。 所以一次我可以在 0.1 秒内拆分,而另一次我可以在 0.7 秒内拆分文件,依此类推。
我该如何为此编写一个python脚本? 该文件如下所示(整个 119K 文件可以从https://fil.email/vnsZsp7b下载):
No.,Time,Length
1,0,146
2,0.006752,116
3,0.019767,156
4,0.039635,144
5,0.06009,147
6,0.069165,138
7,0.0797,133
8,0.099397,135
9,0.120142,135
10,0.139721,148
11,0.1401,126
12,0.1401,120
13,0.140101,123
14,0.140101,120
15,0.141294,118
16,0.141295,118
17,0.141295,114
18,0.144909,118
19,0.160639,119
20,0.161214,152
21,0.185625,143
... etc
在@Serafeim 的回答之后,我尝试了这个:
import pandas as pd
import numpy as np
import glob
import os
path = '/root/Desktop/TT1/'
mystep = 0.4
for filename in glob(os.path.join(path, '*.csv')):
df = pd.read_csv(filename)
def data_splitter(df):
max_time = df['Time'].max() # get max value of Time for the current csv file (df)
myrange= np.arange(0, max_time, mystep) # build the threshold range
for k in range(len(myrange)):
# build the upper values
temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
#temp.to_csv("/root/Desktop/T1/xx_{}.csv".format(k))
temp.to_csv("/root/Desktop/T1/{}_{}.csv".format(filename, k))
data_splitter(df)
您只需要使用pandas
对数据帧应用逻辑操作。 ✔️
在这个答案的最后,我有一个“脚本想法”可以自动执行此操作,但首先让我们一步一步地进行:
# Load the files using pandas
import pandas as pd
df = pd.read_csv("/Users/serafeim/Downloads/Testfile.csv")
# Get the desired elements based on 'Time' column
mask = df['Time'] < 0.1
# Write the new file
df_1 = df[mask] # or directly use: df_1 = df[df['Time'] < 0.1]
# save it
df_1.to_csv("Testfile1.csv")
print(df_1)
No. Time Length
0 1 0.000000 146
1 2 0.006752 116
2 3 0.019767 156
3 4 0.039635 144
4 5 0.060090 147
5 6 0.069165 138
6 7 0.079700 133
7 8 0.099397 135
#For 0.1 to 0.2 applying 2 logical conditions
df_2 = df[(df['Time'] > 0.1) & (df['Time'] < 0.2)]
剧本思路:
import pandas as pd
import numpy as np
mystep = 0.2 # the step e.g. 0.2, 0.4, 0.6
#define the function
def data_splitter(df):
max_time = df['Time'].max() # get max value of Time for the current csv file (df)
myrange= np.arange(0, max_time, mystep) # build the threshold range
for k in range(len(myrange)):
# build the upper values
temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
temp.to_csv("/Users/serafeim/Downloads/aaa_{}.csv".format(k))
现在,调用函数:
df = pd.read_csv("/Users/serafeim/Downloads/Testfile.csv")
data_splitter(df) # pass the df to the function and call the function
最后,您可以创建一个循环并在data_splitter()
函数中逐个传递每个df
。
为了更清楚地说明函数的作用,如下所示:
for k in range(len(myrange)):
print myrange[k], myrange[k]+step
这打印:
0.0 0.2
0.2 0.4
0.4 0.6000000000000001
0.6000000000000001 0.8
0.8 1.0
因此它会根据当前 .csv 文件的Time
列的最大值自动创建上下阈值。
编辑2:
import glob, os
path = '/Volumes/'
mystep = 0.2
for filename in glob.glob(os.path.join(path, '*.csv')):
df = pd.read_csv(filename)
data_splitter(df)
import pandas as pd
import numpy as np
import glob
import os
path = '/root/Desktop/TT1/'
mystep = 0.4
#define the function
def data_splitter(df, name):
max_time = df['Time'].max() # get max value of Time for the current csv file (df)
myrange= np.arange(0, max_time, mystep) # build the threshold range
for k in range(len(myrange)):
# build the upper values
temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
temp.to_csv("/root/Desktop/T1/{}_{}.csv".format(name, k))
for filename in glob.glob(os.path.join(path, '*.csv')):
df = pd.read_csv(filename)
name = os.path.split(filename)[1] # get the name of the file
data_splitter(df, name) # call the splitting function
假设您有 2 个目录:Source 和 Test。 Source 包含所有源 csv 文件,Test 目录将包含所有输出文件。
import os
import glob
os.chdir("/home/prasanth-8508/Downloads/Source")
for csv_file in glob.glob("*.csv"):
contents, output_list = list(), list()
with open(csv_file) as f:
contents.append(f.read().replace('"', ""))
contents = ''.join(contents).split('\n')
header = contents[0]
contents = contents[1:]
op_file_counter = 1
split_factor = float(input("Enter split factor:"))
split_num = split_factor
i = 0
contents = list(filter(None, contents))
while i < len(contents)-1:
try:
row = contents[i].split(",")
if not(str(float(row[1])).startswith(str(split_num)[0:str(split_num).index(".")+2], 0, str(split_num).index(".")+2)):
output_list.append(contents[i])
i += 1
else:
if len(output_list) > 0:
with open("/home/prasanth-8508/Downloads/Test/file" + str(op_file_counter) + ".csv", "a+") as f:
f.write(header+'\n')
for j in output_list:
f.write(j+'\n')
op_file_counter += 1
output_list = list()
split_num += split_factor
split_num = round(split_num,1)
print(split_num)
except IndexError:
break
with open("/home/prasanth-8508/Downloads/Test/file" + str(op_file_counter) + ".csv", "a+") as f:
f.write(header+'\n')
for j in output_list:
f.write(j+'\n')
print(csv_file+" processed successfully")
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.