[英]Keras: Using a generator for multi-output model with model.fit_generator
[英]Use a generator for Keras model.fit_generator
在编写用于训练 Keras 模型的自定义生成器时,我最初尝试使用generator
语法。 所以我从__next__
yield
ed。 但是,当我尝试使用model.fit_generator
训练我的模式时,我会收到一个错误,指出我的生成器不是迭代器。 此修复程序是改变yield
以return
其也必要rejiggering的逻辑__next__
跟踪状态。 与让yield
为我完成工作相比,这相当麻烦。
有没有办法让这个工作与yield
工作? 如果我必须使用return
语句,我将需要编写更多的迭代器,这些迭代器必须具有非常笨拙的逻辑。
我无法帮助调试您的代码,因为您没有发布它,但我缩写了我为语义分割项目编写的自定义数据生成器,供您用作模板:
def generate_data(directory, batch_size):
"""Replaces Keras' native ImageDataGenerator."""
i = 0
file_list = os.listdir(directory)
while True:
image_batch = []
for b in range(batch_size):
if i == len(file_list):
i = 0
random.shuffle(file_list)
sample = file_list[i]
i += 1
image = cv2.resize(cv2.imread(sample[0]), INPUT_SHAPE)
image_batch.append((image.astype(float) - 128) / 128)
yield np.array(image_batch)
用法:
model.fit_generator(
generate_data('~/my_data', batch_size),
steps_per_epoch=len(os.listdir('~/my_data')) // batch_size)
我最近玩了 Keras 的生成器,我终于设法准备了一个例子。 它使用随机数据,因此尝试在其上教授 NN 毫无意义,但它很好地说明了将 Python 生成器用于 Keras。
生成一些数据
import numpy as np
import pandas as pd
data = np.random.rand(200,2)
expected = np.random.randint(2, size=200).reshape(-1,1)
dataFrame = pd.DataFrame(data, columns = ['a','b'])
expectedFrame = pd.DataFrame(expected, columns = ['expected'])
dataFrameTrain, dataFrameTest = dataFrame[:100],dataFrame[-100:]
expectedFrameTrain, expectedFrameTest = expectedFrame[:100],expectedFrame[-100:]
发电机
def generator(X_data, y_data, batch_size):
samples_per_epoch = X_data.shape[0]
number_of_batches = samples_per_epoch/batch_size
counter=0
while 1:
X_batch = np.array(X_data[batch_size*counter:batch_size*(counter+1)]).astype('float32')
y_batch = np.array(y_data[batch_size*counter:batch_size*(counter+1)]).astype('float32')
counter += 1
yield X_batch,y_batch
#restart counter to yeild data in the next epoch as well
if counter >= number_of_batches:
counter = 0
Keras模型
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten, Reshape
from keras.layers.convolutional import Convolution1D, Convolution2D, MaxPooling2D
from keras.utils import np_utils
model = Sequential()
model.add(Dense(12, activation='relu', input_dim=dataFrame.shape[1]))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])
#Train the model using generator vs using the full batch
batch_size = 8
model.fit_generator(
generator(dataFrameTrain,expectedFrameTrain,batch_size),
epochs=3,
steps_per_epoch = dataFrame.shape[0]/batch_size,
validation_data = generator(dataFrameTest,expectedFrameTest,batch_size*2),
validation_steps = dataFrame.shape[0]/batch_size*2
)
#without generator
#model.fit(
# x = np.array(dataFrame),
# y = np.array(expected),
# batch_size = batch_size,
# epochs = 3
#)
输出
Epoch 1/3
25/25 [==============================] - 3s - loss: 0.7297 - acc: 0.4750 -
val_loss: 0.7183 - val_acc: 0.5000
Epoch 2/3
25/25 [==============================] - 0s - loss: 0.7213 - acc: 0.3750 -
val_loss: 0.7117 - val_acc: 0.5000
Epoch 3/3
25/25 [==============================] - 0s - loss: 0.7132 - acc: 0.3750 -
val_loss: 0.7065 - val_acc: 0.5000
这是我实现它读取任何大小文件的方式。 它就像一个魅力。
import pandas as pd
hdr=[]
for i in range(num_labels+num_features):
hdr.append("Col-"+str(i)) # data file do not have header so I need to
# provide one for pd.read_csv by chunks to work
def tgen(filename):
csvfile = open(filename)
reader = pd.read_csv(csvfile, chunksize=batch_size,names=hdr,header=None)
while True:
for chunk in reader:
W=chunk.values # labels and features
Y =W[:,:num_labels] # labels
X =W[:,num_labels:] # features
X= X / 255 # any required transformation
yield X, Y
csvfile = open(filename)
reader = pd.read_csv(csvfile, chunksize=batchz,names=hdr,header=None)
主要的背面我有
nval=number_of_validation_samples//batchz
ntrain=number_of_training_samples//batchz
ftgen=tgen("training.csv")
fvgen=tgen("validation.csv")
history = model.fit_generator(ftgen,
steps_per_epoch=ntrain,
validation_data=fvgen,
validation_steps=nval,
epochs=number_of_epochs,
callbacks=[checkpointer, stopper],
verbose=2)
我想用 TensorFlow 2.x 升级Vaasha 的代码,以提高训练效率并简化数据处理。 这对于图像处理特别有用。
使用上面示例中 Vaasha 生成的 Generator 函数或使用tf.data.dataset API 处理数据。 后一种方法在处理具有元数据的任何数据集时非常有用。 例如,可以通过几条语句加载和处理 MNIST 数据。
import tensorflow as tf # Ensure that TensorFlow 2.x is used
tf.compat.v1.enable_eager_execution()
import tensorflow_datasets as tfds # Needed if you are using any of the tf datasets such as MNIST, CIFAR10
mnist_train = tfds.load(name="mnist", split="train")
使用 tfds.load 数据集。 加载和处理数据后(例如,转换分类变量、调整大小等)。
现在使用 TensorFlow 2.x 升级 keras 模型
model = tf.keras.Sequential() # Tensorflow 2.0 upgrade
model.add(tf.keras.layers.Dense(12, activation='relu', input_dim=dataFrame.shape[1]))
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adadelta',
metrics=['accuracy'])
#Train the model using generator vs using the full batch
batch_size = 8
model.fit_generator(generator(dataFrameTrain,expectedFrameTrain,batch_size),
epochs=3,
steps_per_epoch=dataFrame.shape[0]/batch_size,
validation_data=generator(dataFrameTest,expectedFrameTest,batch_size*2),
validation_steps=dataFrame.shape[0]/batch_size*2)
这将升级模型以在 TensorFlow 2.x 中运行
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.