Keras Sequential Model: fit_generator with batch input

Question

I have been trying to train a Sequential Keras model using a sparse matrix. Although I have specified the batch size in the code, it is being trained at batch_size = 1 (ie one row at a time).

Here's the code:

def batch_generator(X_toGen, y_toGen = None, batch_size = 32):
    counter = 0
    sample_index = np.arange(X_toGen.shape[0])
    np.random.shuffle(sample_index)
    while True:
        batch_index = sample_index[batch_size*counter:min(batch_size*(counter+1), X_toGen.shape[0])]
        counter += 1
        X_batch = X_toGen[batch_index,:]
        if y_toGen is not None:
            y_batch = y_toGen[batch_index]
            yield X_batch.toarray(), y_batch
        else:
            yield X_batch.toarray()

Can anyone please help me in generating a batch input for the Sequential model? Also, how different would the accuracy be when the model is being trained at batch_size = 32, rather than batch_size = 1?

Thanks

Answer 1

What you are getting is not 1 batch size output from this generator. Let me give you an example here:

If you have total 1000 samples and you are dividing this in batches 10. Then using fit_generator you will get only 10 batches which means you have 10 batches each of 10 samples.

So, the crux is fit_generator shows number of batches instead of total number of samples.