![](/img/trans.png)
[英]What is the pythonic way to read CSV file data as rows of namedtuples?
[英]What is a faster and more Pythonic way to read the CSV and make a data frame from it?
輸入 :50,000行的CSV; 每行包含910列值0/1。
輸出 :運行我的CNN的數據幀。
我寫了一段代碼,逐行讀取CSV。 對於每一行,我將數據分為兩部分,分別稱為神經元 (900列)和標簽 (10列)。 由於這些是列表,因此我將它們轉換為Numpy數組。 當我轉到下一行時,我做同樣的事情並堆疊數組以最終獲得4個常規數據集:
x_train,x_test,y_train,y_test
我的代碼有效,因為我在只有6行的小型CSV上進行了測試。 但是,在數組初始化之后,當我在50,000行的實際數據集上運行它時,要花很多時間才能將行轉換為數據幀。
所以我想知道是否有更快的方法來進行這種轉換,還是可以在這里等一下!
這是我的代碼:
import numpy as np
import pandas as pd
import time
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.utils import np_utils
from sklearn.model_selection import train_test_split
# Read the dataset from the CSV file into a dataframe
df = pd.read_csv("bci_dataset_labelled.csv")
start_init = time.time()
xvalues = np.zeros((900,), dtype=np.int)
yvalues = np.zeros((10,), dtype=np.int)
print("--- Arrays initialized in %s seconds ---" % (time.time() - start_init))
start_conversion = time.time()
for row in df.itertuples(index=False):
# separate the neurons from the labels
x = list(row[:900])
y = list(row[900:])
# convert the lists to numpy arrays
x = np.array(x)
y = np.array(y)
xvalues = np.vstack((xvalues, x))
yvalues = np.vstack((yvalues, y))
print("--- CSV rows converted to dataframe in %s seconds ---" % (time.time() - start_conversion))
start_split = time.time()
x_train, x_test, y_train, y_test = train_test_split(xvalues, yvalues, test_size=0.2)
print("--- Dataframe split into training and testing datasets in %s seconds ---" % (time.time() - start_split))
num_classes = y_test.shape[1]
num_neurons = x_train[0].shape[0]
# define baseline model
def baseline_model():
#create model
model = Sequential()
model.add(Dense(
num_neurons,
input_dim = num_neurons,
kernel_initializer = 'normal',
activation = 'relu'
))
model.add(Dense(
num_classes,
kernel_initializer = 'normal',
activation = 'softmax'
))
#compile model
model.compile(
loss = 'categorical_crossentropy',
optimizer = 'adam',
metrics = ['accuracy'])
return model
# build the model
model = baseline_model()
# fit the model
model.fit(x_train, y_train, validation_data = (x_test, y_test),
epochs = 10, batch_size = 200, verbose = 2)
# final evaluation of the model
scores = model.evaluate(x_test, y_test, verbose=0)
print("Baseline error: %0.2f%%" % (100-scores[1]*100))
它只是卡在這里:
Rachayitas-MacBook-Pro:bci_hp rachayitagiri$ python3 binarycnn.py
Using TensorFlow backend.
--- Arrays initialized in 2.4080276489257812e-05 seconds ---
任何建議將不勝感激! 謝謝!
編輯:將輸出作為文本從控制台而不是圖片中放置。 感謝您的建議。
您可能無法擊敗read_csv ,它是開箱即用的,並且可能比那里的任何其他解決方案都經過更好的測試。
從我看來,您的問題不在於read_csv
函數,而在於您從DataFrame中提取信息的方式。 您可以直接從DataFrame獲取xvalues
和yvalues
,而不是逐行讀取DataFrame,這非常昂貴。 DataFrames使您能夠以一種非常優化的方式進行操作。
據我了解,您的X值位於前900列中,Y值位於其后。 這是我的處理方式:
import pandas as pd
import numpy as np
import time
start_init = time.time()
df = pd.DataFrame(np.random.randint(0,100,size=(50000, 910)))
print("--- DataFrame initialized in %s seconds ---" % (time.time() - start_init))
start_conversion = time.time()
x = df.loc[:, :900] # Here's where you get your x values, 900 first values in each row
y = df.loc[:, 900:] # And here you retrieve the y values
# All that's left is to convert that to a numpy array by doing this
xvalues = x.values
yvalues = y.values
print("--- Took data out of DataFrame in %s seconds ---" % (time.time() -
start_conversion))
print(x.shape, y.shape)
我得到以下打印此代碼:
--- Arrays initialized in 0.6232161521911621 seconds ---
--- Took data out of DataFrame in 0.038640737533569336 seconds ---
(50000, 901) (50000, 10)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.