使用fit_generator时，Keras模型的批量大小为1

Question

When I fit_generator with my batch generator it uses a batch size of 1, it does 1 step increment for each epoch. What am I doing wrong.

I have tried changing batch size arguments for my batch generator but still the same.

My batch generator function:

def batchGenerator(imgs, steerings, batchSize, isTraining):

    while True:
        batchImg = []
        batchSteering = []

        for i in range(batchSize):
            randIndex = random.randint(0, len(imgs) - 1)
            if isTraining:
                img, steering = randomAugment(imgs[randIndex], steerings[randIndex])
            else:
                img = imgs[randIndex]
                steering = steerings[randIndex]

            img = imgPreprocess(img)

            batchImg.append(img)
            batchSteering.append(steering)

        yield (np.asarray(batchImg), np.asarray(batchSteering))

This is my fit_generator arguments:

history = model.fit_generator(batchGenerator(X_train, y_train, 300, 1),
                              steps_per_epoch = 300,
                              epochs = 10,
                              validation_data = batchGenerator(X_valid, y_valid, 200, 0),
                              validation_steps = 200,
                              verbose = 1,
                              shuffle = 1)

When I run this the batch size seems to be 1, as for each epoch it is being incremented by 1. For each epoch it does 0/300, 1/300, 2/300, 3/300, etc.

What is going on?

Answer 1

Your steps_per_epoch should always be length of training data divided by batch_size , ie in this case X_train.shape[0]//batch_size .

Also, the way you are shuffling your data with the random index, it will mean some samples might be selected more than once and some, never. You can also think about random shuffling the entire training set first, and then pick sequential batches of data for training. I just wanted to point this out, if you missed.

Edit

def batchGenerator(imgs, steerings, batchsize, isTraining):
    while True:
        start = 0
        end = batchsize

        while start  < len(imgs): 
            x = imgs[start:end]
            y = steerings[start:end]
            if isTraining:            
                x , y = randomAugment(x, y)

            yield x, y

            start += batchsize
            end += batchsize

Maybe try something like this. You can handle the shuffling later if this works.

Answer 2

Your generator has no issue and your code is fine too. The way you interpret the output is wrong.

From the docs, you can see

steps_per_epoch: Integer or None. Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. When training with input tensors such as TensorFlow data tensors, the default None is equal to the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined.

Normally steps_per_epochs is set as X_train.shape[0]//batch_size

While training, the training is done for steps_per_epochs batches and one epoch is treated as completed. Since the data is taken in a random order there is no other way to tell the model that one epoch is over.

While training you can see 0/300, 1/300, 2/300, 3/300 and so on till 300/300 . It is quite normal. This means your model is trained for 300 steps where there batch size for each step is 300 (since you gave batch size as 300)

If you gave batch size as let's say 10, and steps_per_epoch as 100 you can see while training 1/100, 2/100 so on till 100/100 which means, your model is trained for 100 steps and each step is essentialy a batch of 10 samples

Answer 3

In contrast to the fit function, the output of fit_generator is the count of batches and not of training examples. Consequently,an increment of 1 means that one more batch has been processed. With steps_per_epoch you define how many batches will be processed per epoch.

By definition, one epoch is finished when each training example has been processed once. This is why people suggest to set steps_per_epoch to:

steps_per_epoch=number_of_examples//batch_size