嘗試輸入大數據框時，Tensorflow Dataset API內存錯誤

Question

我有350K行和200列的pandas數據框。 嘗試使用數據集api構造輸入管道時，出現內存錯誤。 例如，當我僅輸入10K行時，一切正常，但對於所有行，則並非如此。 同樣，當使用tf.estimator.inputs.pandas_input_fn一切正常。

這是代碼

x_train, x_test, y_train, y_test = train_test_split(train, labels, test_size=0.25)

feature_columns = [tf.feature_column.numeric_column(c) for c in train.columns
                if train[c].dtype != 'object']

def train_input_fn():
    dataset = tf.data.Dataset.from_tensor_slices((dict(x_train), y_train))
    dataset = dataset.shuffle(1000)
    dataset = dataset.batch(100)
    iterator = dataset.make_one_shot_iterator()
    return iterator.get_next()

model = tf.estimator.DNNClassifier(feature_columns=feature_columns, hidden_units=[20, 2]
model.train(input_fn=train_input_fn, steps=1000)

和錯誤消息

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-09-28 14:42:03.736495: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-09-28 14:42:04.070692: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.797
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.63GiB
2018-09-28 14:42:04.072060: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-09-28 14:42:05.139979: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-28 14:42:05.140271: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]      0 
2018-09-28 14:42:05.140461: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0:   N 
2018-09-28 14:42:05.141143: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
Traceback (most recent call last):
  File "C:/Users/.../test.py", line 150, in <module>
    nn.train(input_fn=train_input_fn, steps=10)
  ...
  File "C:\Users\...\google\protobuf\text_format.py", line 118, in getvalue
    return self._writer.getvalue()
MemoryError

我嘗試設置不同的批處理大小，網絡體系結構，但錯誤仍然存​​在。

Answer 1

請在https://stackoverflow.com/a/56213870/31045上查看我的答案，以獲得在輸入​​數據量較大時創建生成器的代碼。