如何在itertools.cycle（）中随机播放？

Question

I'm using the itetools in the data generator in tensorflow based network training. The main framework is as follows:

def data_generator(filenames, batch_size):
    files= itertools.cycle(filenames)
    while True:
        X = []
        Y = []
        for _ in range(batch_size):
            filename = files.next()
            # read data into X and Y
            ....

        yield np.array(X), np.array(Y)

When using this data generater,

train_input = data_generator(train_filenames,batch_size=1)
for ep in range(num_epochs):
    for _ in range(num_train_samples):
        image_batch, label_batch = train_input.next()
        loss_val = sess.run([loss_op], feed_dict={})

My question is: in general, we need to shuffle the training data after each epoch, how to shuffle in this circumstance? Thanks.

Answer 1

You can not modify a cycle iterator once created, so you have to make a new one in each "epoch" loop:

def data_generator(filenames, batch_size):
    filenames = filenames[:] # make a copy
    random.shuffle(filenames)
    files = itertools.cycle(filenames)
    ...

for ep in range(num_epochs):
    train_input = data_generator(train_filenames, batch_size=1)
    ...