如何从生成器编写数据集以替换 tensorflow 中切片中的数据集，以获取具有 X_train 和 y_train 的表格数据集

Question

The following works but consumes all my GPU memory as the dataset gets larger.

tf_train = tf.data.Dataset.from_tensor_slices((X_train, y_train)).shuffle(1000).batch(512, drop_remainder=True).prefetch(1)

I tried various options but am stuck on how to write the generator.

tf_train = tf.data.Dataset.from_generator(generator=my_gen, output_signature=??)

I don't know how to write my_gen nor the output signature syntax.

X_train is a dataframe of numerical features and y_train is a df containing a numerical target variable.

Answer 1

You could change your generator function to:

def generate_sample():
    x = list("123456789")
    y = list("2345")
    while 1:
        yield np.array(x).astype(np.float32), (
            np.array(y).astype(np.float32),
            np.array(y).astype(np.float32),
        )    

output signature,

def generate_sample():
    x = list("123456789")
    y = list("2345")
    while 1:
        yield np.array(x).astype(np.float32), (
            np.array(y).astype(np.float32),
            np.array(y).astype(np.float32),
        )