自定义Keras生成器比Keras的bult生成器慢得多

Question

I have a multi label classification problem. I wrote this custom generator. It reads images and output labels from the disk, and returns them in batches of size 32.

def get_input(img_name):
    path = os.path.join("images", img_name)
    img = image.load_img(path, target_size=(224, 224))

    return img


def get_output(img_name, file_path):
    data = pd.read_csv(file_path, delim_whitespace=True, header=None)

    img_id = img_name.split(".")[0]
    img_id = img_id.lstrip("0")
    img_id = int(img_id)

    labels = data.loc[img_id - 1].values
    labels = labels[1:]

    labels = list(labels)
    label_arrays = []
    for i in range(20):
        val = np.zeros((1))
        val[0] = labels[i]
        label_arrays.append(val)

    return label_arrays


def preprocess_input(img_name):
    img = get_input(img_name)
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)    
    return x

def train_generator(batch_size):
    file_path = "train.txt"
    data = pd.read_csv(file_path, delim_whitespace=True, header=None)

    while True:
        for i in range(math.floor(8000/batch_size)):
            x_batch = np.zeros(shape=(32, 224, 224, 3))
            y_batch = np.zeros(shape=(32, 20))
            for j in range(batch_size):
                img_name = data.loc[i * batch_size + j].values
                img_name = img_name[0]
                x = preprocess_input(img_name)
                y = get_output(img_name, file_path)
                x_batch[j, :, :, :] = x
                y_batch[j] = y

            ys = []
            for i in range(20):
              ys.append(y_batch[:,i])

            yield(x_batch, ys)

Had a little problem with labels returned to the model, and got it solved in this question: training a multi-output keras model

I tested this generator on a single output problem. This custom generator is very slow. The ETA for a single epoch by using this custom generator is around 27 hours, while the builtin generator(using flow_from_directory) takes 25 minutes for a single epoch. What am I doing wrong?

The training process for both tests is identical, except for the generator used. Validation generator is similar to training generator. I know I will not reach the efficiency of Keras' built in generator, but this difference in speed is too much.

EDIT

Some guides I read for creating custom generators.

Writing Custom Keras Generators

custom generator for fit_generator() that yields multiple inputs with different shapes

Answer 1

Maybe the built in generator processes the data on your gpu while your custom generator runs on the cpu, making is significantly slower.

Another guess is because Keras is using Dataset in the background. Your implementation probably uses feed-dict which is the slowest possible way to pass information to TensorFlow. The best way to feed data into the models is to use an input pipeline to ensure that the GPU never has to wait for new stuff to come in.