想要从具有张量流的csv中分离出列车和测试数据

Question

I wanted to split train and test data of a csv with tensorflow but I didn't find an order like np.loadtxt in tensor and tried to do splits with numpy and convert it to tensor, but I get an error like below:

      TypeError: object of type 'Tensor' has no len()

and here is my code:

     import tensorflow as tf
     import numpy as np
     import matplotlib.pyplot as plt
     from sklearn.model_selection import train_test_split

     x = tf.convert_to_tensor( np.loadtxt('data.csv', delimiter=','))
     y = tf.convert_to_tensor(np.loadtxt('labels.csv', delimiter=','))

     x_train, x_test, y_train, y_test = train_test_split(x, y, 
     test_size=0.25, random_state='')

     model = tf.keras.models.Sequential([
     tf.keras.layers.Flatten(input_shape= (426,30,1)),
     tf.keras.layers.Dense(126, activation=tf.nn.tanh),
      #tf.keras.layers.Dropout(0.2),
      tf.keras.layers.Dense(10, activation=tf.nn.tanh)
       ])

       model.compile(optimizer='sgd',
          loss='mean_squared_error',
          metrics=['accuracy'])

       history = model.fit(x_train, y_train, epochs=5 )  #validation_data 
       = [x_test, y_test])
       model.evaluate(x_test, y_test)

      t_predicted = model.predict(x_test)
      out_predicted = np.argmax(t_predicted, axis=1)
      conf_matrix = tf.confusion_matrix(y_test, out_predicted)
      with tf.Session():
       print('Confusion Matrix: \n\n', tf.Tensor.eval(conf_matrix, 
     feed_dict=None, session=None))

      # summarize history for accuracy
      plt.plot(history.history['acc'])
      plt.plot(history.history['val_acc'])
      plt.title('model accuracy')
      plt.ylabel('accuracy')
      plt.xlabel('epoch')
      plt.legend(['train', 'test'], loc='upper left')
      plt.show()

     # summarize history for loss
     plt.plot(history.history['loss'])
     plt.plot(history.history['val_loss'])
     plt.title('model loss')
     plt.ylabel('loss')
     plt.xlabel('epoch')
     plt.legend(['train', 'test'], loc='upper left')
     plt.show()

Answer 1

Won't it be simpler to first load the csv file, do the split and then give Tf the result of the split?

sklearn.model_selection.train_test_split() is not meant to work with the Tensor objects you're getting from tf.convert_to_tensor() .

Reversing the order made your code work in small test script

x = np.loadtxt('data.csv', delimiter=',')
y = np.loadtxt('labels.csv', delimiter=',')

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.25)

x_train = tf.convert_to_tensor(x_train)
x_test = tf.convert_to_tensor(x_test)
y_train = tf.convert_to_tensor(y_train)
y_test = tf.convert_to_tensor(y_test)

Answer 2

The best practice it not to load full data into tensor. If your code is executed on GPU and if you data is huge the tensor might occupy a significant amount of GPU memory result in "Out of Memory" errors. The normally used way is

• Load the data into a RAM variable (usually numpy),
• Split the data into train and validation batches (using something like train_test_split )
• Iterate over splits with batch size and creating a tensor of batch size. Use it to train (validation split batch for validation)

When dealing with large/huge images, we cannot load all the images into memory. Here the normally followed approached is to load a batch of images into a tensor and use it for training/validataion. All the deeplearning frameworks provide mechanisms to load multiple batches in multithreads so that the next train step is not waiting for the next batch to be loaded.

If you still want to load the full data into a tensor and split it into train and test tensors then can use the tensorflow method tf.split . https://www.tensorflow.org/api_docs/python/tf/split