TF2.0 自定义生成器的问题

Question

I am trying to learn how to use the tf.data.Dataset methods and working on a custom generator from a CSV file. The CSV holds three data points. 1 the path to the image, 2 the label, 3 the image name (Not used). What I am trying to do is read in the CSV in pandas and then iterate over each roll yeilding the read in an image (Keras load_image) and the lable which was encoded using Pandas. It gives me this error which makes me think it is the generator issue as it is not doing what it should.

ValueError: as_list() is not defined on an unknown TensorShape.

This is my code that I am working with.

    csv_path = '../Dataset/ShowData.csv'
    df = pd.read_csv(csv_path)
    base_path = "../Dataset/"
    # Make one hot encoding for lables
    le = LabelEncoder()
    df['Label'] = le.fit_transform(df['Label'])
    print(df.head(1)['Label'])
    print(df.tail(1)['Label'])
    '''
    # View the labels (if you want)
    list(encoder.classes_)
    # Convert some integers into their category names
    list(encoder.inverse_transform([2, 2, 1]))
    '''

    def process_dataframe(dataframe):
        for index, row in dataframe.iterrows():
            # print(row['Path'], row['Label'])
            # Load image and get lable
            img_path = os.path.join(base_path, row['Path'])
            img = load_img(img_path, target_size=(200, 200))
            img = img_to_array(img)
            img = img/255  # normalize the image
            label = row['Label']
            #label = to_categorical(label, num_classes, dtype=tf.float32)
            yield img, label


    def generate_dataset(dataframe):
        generator = lambda: process_dataframe(dataframe)
        return tf.data.Dataset.from_generator(generator=generator,
                                          output_types= (tf.float32, tf.int32))


    dataset = generate_dataset(df)
    data_batch = dataset.shuffle(10000).batch(32)
    print(data_batch)

    model = Sequential([
        Flatten(input_shape=(200, 200, 3)),
        Dense(128, activation='relu'),
        Dense(10, activation='softmax')
    ])

    model.compile(optimizer=tf.keras.optimizers.Adam(),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(),
              metrics=[tf.keras.metrics.Accuracy()])

    model.fit(data_batch, epochs=10, verbose=1)

Oh this is what the data_batch prints out

<DatasetV1Adapter shapes: (<unknown>, <unknown>), types: (tf.float32, tf.int32)>

And this is the label type

Name: Label, dtype: int32

Answer 1

You should define the output_shapes for the tf.data.Dataset.from_generator call. You can find the documentation at the Tensorflow page .