Tensorflow 图像生成器通过 dtype=string 的张量而不是 dtype=float32 的张量来丢失 function

Question

I am following the YOLO v1 paper to create an object detector from scratch with Tensorflow and python. My dataset is a set of images and a 7x7x12 Tensor that represents the label for the image. I import the image names and labels (as a string) into a dataframe from a CSV using pandas, and then do some operations on the labels to turn them into Tensors. I then create a generator using ImageGenerator.flow_from_dataframe(), and the feed that generator as the input for my model. I end up getting the following error when the model tries to call the custom loss function that I created:

File "/home/michael/Desktop/YOLO_Detector/src/main.py", line 61, in <module>
    epochs=10)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
    total_epochs=epochs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 128, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 98, in execution_function
    distributed_function(input_fn))
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 85, in distributed_function
    per_replica_function, args=args)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 763, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1819, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 2164, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
    return func(*args, **kwargs)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 312, in train_on_batch
    output_loss_metrics=output_loss_metrics))
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 253, in _process_single_batch
    training=training))
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 167, in _model_loss
    per_sample_losses = loss_fn.call(targets[i], outs[i])
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/losses.py", line 221, in call
    return self.fn(y_true, y_pred, **self._fn_kwargs)
  File "/home/michael/Desktop/YOLO_Detector/src/utils.py", line 25, in yolo_loss_function
    pos = kb.mean(y_true-y_pred)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/math_ops.py", line 902, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 10104, in sub
    "Sub", x=x, y=y, name=name)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 576, in _apply_op_helper
    param_name=input_name)
  File "/home/michael/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 61, in _SatisfiesTypeConstraint
    ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'x' has DataType string not in list of allowed values: bfloat16, float16, float32, float64, uint8, int8, uint16, int16, int32, int64, complex64, complex128

When I use the python debugger to check the y_true that is being passed to the loss function, I see the following tensor:

Tensor("IteratorGetNext:1", shape=(None, 1), dtype=string)

however when I manually check the label of the image by calling the following python code, I get a tensor with the correct shape and values:

img, label = next(train_gen)
print(type(label[0]))
print(label[0].shape)
print(label[0].dtype)

-------------Output--------------
<class 'tensorflow.python.framework.ops.EagerTensor'>
(7, 7, 12)
<dtype: 'float32'>

Below is the code that I am using to create the dataset and train the model:

import tensorflow as tf
from tensorflow import keras
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import numpy as np
import os
from src import utils, model


path = "/home/michael/Desktop/YOLO_Detector/dataset/labels.csv"

train_df = pd.read_csv(path, delim_whitespace=True, header=None)
train_df.columns = ['filename', 'output_tensor']
train_df["output_tensor"] = train_df["output_tensor"].apply(lambda x: utils.string_to_tensor(x))

# train_df["output_tensor"] = train_df["output_tensor"].apply(lambda x: tf.expand_dims(x, 3))

image_generator = keras.preprocessing.image.ImageDataGenerator(rescale=1. / 255.,
                                                               validation_split=0.2)

train_gen = image_generator.flow_from_dataframe(
    dataframe=train_df,
    directory="/home/michael/Desktop/YOLO_Detector/dataset",
    x_col='filename',
    y_col='output_tensor',
    class_mode='raw',
    batch_size=64,
    target_size=(448, 448),
    shuffle=False,
    subset="training"
)

validation_gen = image_generator.flow_from_dataframe(
    dataframe=train_df,
    directory="/home/michael/Desktop/YOLO_Detector/dataset",
    x_col='filename',
    y_col='output_tensor',
    class_mode="raw",
    batch_size=64,
    target_size=(448, 448),
    shuffle=False,
    subset="validation"
)

img, label = next(train_gen)
print(type(label[0]))
print(label[0].shape)
print(label[0].dtype)

model = model.create_model()
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_rate=0.0005,
    decay_steps=100000
)
sgd = keras.optimizers.SGD(learning_rate=lr_schedule, momentum=0.9)

model.compile(optimizer=sgd, loss=utils.yolo_loss_function, metrics=['accuracy'])
model.fit(x=train_gen,
          epochs=10)

I am using Tensorflow 2 with ROCm, and Eager Execution is on. How do I get the y_true tensor to be the correct output label (7x7x12 Tensor with dtype=float32) instead of it being a string?

Answer 1

I figured out the issue. The issue is that you cannot correctly store a Tensor or a Numpy array in a Pandas Dataframe. I ended up having to manually create the image/tensor pairs by doing the following:

img_list = []
labels_list = []
for i in range(64):
    labels_list.append(utils.string_to_numpy(train_df["output_tensor"][i]))

    image = tf.keras.preprocessing.image.load_img(f"/home/michael/Desktop/YOLO_Detector/dataset/{train_df['filename'][i]}", target_size=(448, 448))
    image_arr = keras.preprocessing.image.img_to_array(image) / 255.0
    img_list.append(image_arr)

img = np.asarray(img_list)
label = np.asarray(labels_list)

and then calling the img as x and label as y in model.fit()