在TensorFlow中讀取與mnist數據集相同格式的新數據集

Question

我有一個深度學習模型，必須輸入大小為100X100的圖像。 我的數據是

火車圖像x_train (530,100,100) ，火車標簽y_train (530,4) ，

測試圖片x_test(89,100,100) ，測試標簽y_test(89,4) 。

使用-讀取mnist數據集

mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

它會生成這樣的東西-

Datasets(train=tensorflow.contrib.learn.python.learn.datasets.mnist.DataSet object at 0x7fd64cae76a0, 
validation=tensorflow.contrib.learn.python.learn.datasets.mnist.DataSet object at 0x7fd64cae7be0, 
test=tensorflow.contrib.learn.python.learn.datasets.mnist.DataSet object at 0x7fd64cae7400)

我必須以相同的格式轉換數據，以便與我擁有的代碼一起使用。 請幫忙

    epochs = 20
    batch_size = 100
    image_vector = 28*28


for i in range(epochs):
    training_accuracy = []
    epoch_loss = []

    for ii in tqdm(range(mnist.train.num_examples // batch_size)):

        batch = mnist.train.next_batch(batch_size)

        images = batch[0].reshape((-1, 28, 28))
        targets = batch[1]

        c, _, a = session.run([model.cost, model.opt, model.accuracy], feed_dict={model.inputs: images, model.targets:targets})

        epoch_loss.append(c)
        training_accuracy.append(a)

    print("Epoch: {}/{}".format(i, epochs), " | Current loss: {}".format(np.mean(epoch_loss)),
          " | Training accuracy: {:.4f}%".format(np.mean(training_accuracy)))

編輯1：按照建議，我做了以下工作-

num_examples=271
batch_size=10
buffer_size=271
num_cpu_cores=4
dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.shuffle(buffer_size, reshuffle_each_iteration=True).repeat()
#dataset = dataset.apply(tf.data.batch( batch_size=batch_size, num_parallel_batches=num_cpu_cores))

#batch1=dataset.batch(10)

iterator = dataset.make_one_shot_iterator()
sess = tf.Session()
sess.run(tf.global_variables_initializer())
for ii in tqdm(range(num_examples // batch_size)):
    batch = iterator.get_next()
    images = batch[0]
    targets = batch[1]

    c, _, a = sess.run([model.cost, model.opt, model.accuracy])

    epoch_loss.append(c)
    training_accuracy.append(a)

    print("Epoch: {}/{}".format(i, epochs), " | Current loss: {}".format(np.mean(epoch_loss)),
          " | Training accuracy: {:.4f}%".format(np.mean(training_accuracy)))

圖像和目標不是批處理，而是單個圖像和標簽。

（，）

請建議如何批處理數據並將其輸入sess.run

編輯2：這是算法的整個代碼-

def LSTM_layer(lstm_cell_units, number_of_layers, batch_size, dropout_rate=0.8):
    '''
    This method is used to create LSTM layer/s for PixelRNN

    Input(s): lstm_cell_unitis - used to define the number of units in a LSTM layer
              number_of_layers - used to define how many of LSTM layers do we want in the network
              batch_size - in this method this information is used to build starting state for the network
              dropout_rate - used to define how many cells in a layer do we want to 'turn off'

    Output(s): cell - lstm layer
               init_state - zero vectors used as a starting state for the network
    '''


    #layer = tf.contrib.rnn.BasicLSTMCell(lstm_cell_units)
    layer = tf.nn.rnn_cell.LSTMCell(lstm_cell_units,name='basic_lstm_cell')

    if dropout_rate != 0:
        layer = tf.contrib.rnn.DropoutWrapper(layer, dropout_rate)

    cell = tf.contrib.rnn.MultiRNNCell([layer]*number_of_layers)

    init_size = cell.zero_state(batch_size, tf.float32)

    return cell, init_size

def rnn_output(lstm_outputs, input_size, output_size):
    '''
    Output layer for the lstm netowrk

    Input(s): lstm_outputs - outputs from the RNN part of the network
              input_size - in this case it is RNN size (number of neuros in RNN layer)
              output_size - number of neuros for the output layer == number of classes

    Output(s) - logits, 
    '''


    outputs = lstm_outputs[:, -1, :]

    weights = tf.Variable(tf.random_uniform([input_size, output_size]), name='rnn_out_weights')
    bias = tf.Variable(tf.zeros([output_size]), name='rnn_out_bias')

    output_layer = tf.matmul(outputs, weights) + bias
    return output_layer

def loss_optimizer(rnn_out, targets, learning_rate):
    '''
    Function used to calculate loss and minimize it

    Input(s): rnn_out - logits from the fully_connected layer
              targets - targets used to train network
              learning_rate/step_size


    Output(s): optimizer - optimizer of choice
               loss - calculated loss function
    '''
    loss = tf.nn.softmax_cross_entropy_with_logits(logits=rnn_out, labels=targets)
    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)
    return optimizer, loss
class PixelRNN(object):

    def __init__(self, learning_rate=0.001, batch_size=10, classes=4, img_size = (129, 251), lstm_size=64,
                number_of_layers=1, dropout_rate=0.6,clip_rate=None):

        '''
        PixelRNN - call this class to create whole model

        Input(s): learning_rate - how fast are we going to move towards global minima
                  batch_size - how many samples do we feed at ones
                  classes - number of classes that we are trying to recognize
                  img_size - width and height of a single image
                  lstm_size - number of neurons in a LSTM layer
                  number_of_layers - number of RNN layers in the PixelRNN 
                  dropout_rate - % of cells in a layer that we are stopping gradients to flow through
        '''

        #This placeholders are just for images
        self.inputs = tf.placeholder(tf.float32, [None, img_size[0], img_size[1]], name='inputs')
        self.targets = tf.placeholder(tf.int32, [None, classes], name='targets')

        cell, init_state = LSTM_layer(lstm_size, number_of_layers, batch_size, dropout_rate)

        outputs, states = tf.nn.dynamic_rnn(cell, self.inputs, initial_state=init_state)

        rnn_out = rnn_output(outputs, lstm_size, classes)

        self.opt, self.cost = loss_optimizer(rnn_out, self.targets, learning_rate)

        predictions = tf.nn.softmax(rnn_out)

        currect_pred = tf.equal(tf.cast(tf.round(tf.argmax(predictions, 1)), tf.int32), tf.cast(tf.argmax(self.targets, 1), tf.int32))
        self.accuracy = tf.reduce_mean(tf.cast(currect_pred, tf.float32))

        self.predictions = tf.argmax(tf.nn.softmax(rnn_out), 1)


tf.reset_default_graph()
model = PixelRNN()
num_examples=271
batch_size=10
buffer_size=271
num_cpu_cores=4
dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.shuffle(buffer_size, reshuffle_each_iteration=True).repeat()
#dataset = dataset.apply(tf.data.batch( batch_size=batch_size, num_parallel_batches=num_cpu_cores))

dataset = dataset.batch(10) # apply batch to dataset 

iterator = dataset.make_one_shot_iterator() # create iterator
sess = tf.Session()
sess.run(tf.global_variables_initializer())
for ii in tqdm(range(num_examples // batch_size)):
    batch = iterator.get_next() #run iterator
    images = batch[0]
    targets = batch[1]

    c, _, a = sess.run([model.cost, model.opt, model.accuracy],feed_dict={model.inputs: images, model.targets:targets})

    epoch_loss.append(c)
    training_accuracy.append(a)

    print("Epoch: {}/{}".format(i, epochs), " | Current loss: {}".format(np.mean(epoch_loss)),
          " | Training accuracy: {:.4f}%".format(np.mean(training_accuracy)))

當我按照建議進行操作時，出現以下錯誤-

TypeError：供稿的值不能是tf.Tensor對象。 可接受的feed值包括Python標量，字符串，列表，numpy ndarrays或TensorHandles。作為參考，張量對象是Tensor（“ IteratorGetNext：0”，shape =（?, 129，251），dtype = float32），該對象已傳遞給使用鍵Tensor（“ inputs：0”，shape =（?, 129，251），dtype = float32）的feed。

無法找出問題出在哪里

Answer 1

如果您的數據集為numpy數組或圖像文件列表，請使用from_tensor_slices 。 定義解析函數，如果使用文件名列表，則使用read_file和decode_image ，否則只需應用任何預處理

def parse_image(filename, label):
    file = tf.read_file(filename)
    image = tf.image.decode_image(file)
    #do any image/label preprocessing here
    return image, label

然后定義數據集對象。 通常，將數據集的長度用作混洗緩沖區，但這可能取決於大小。 重復功能將控制時期（無值傳遞=不定迭代）。 如果不需要任何預處理，則將dataset.apply行替換為dataset = dataset.batch(batch_size)

dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset_train.shuffle(buffer_size, reshuffle_each_iteration=True).repeat()
dataset = dataset.apply(tf.data.batch( batch_size=batch_size, num_parallel_batches=num_cpu_cores))

創建迭代器。 通常，如果僅輸入數組不會引起2GB graphdef限制，則無需使用feed dict。

iterator = dataset.make_one_shot_iterator()
batch = iterator.get_next()

編輯：您需要從數據集創建迭代器，這是整體結構：

dataset = dataset.batch(10) # apply batch to dataset 
iterator = dataset.make_one_shot_iterator() # create iterator
batch = iterator.get_next() #run iterator
images = batch[0]
targets = batch[1]

logits = Model_function(images)
loss = loss_function(logits, targets)
train_op = optimizer.minimize()

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(steps):
    sess.run(train_op)