Feeding a queue from a list of arrays of different lengths

Question

I have a list of arrays (could be numpy arrays or just list of int ) of different lengths and I would like to feed it into some TensorFlow queue for testing a larger input pipeline -- instead of serializing them and re-read from file system. Is it possible?

Answer 1

I assume you're using an RNN here, for an RNN to take a variable length array you're going to need to pad them with zeros to the batch length and pass in a value that tells the RNN how long each sequence is.

I'd suggest taking the same approach here. Pad them before passing them into the TF queue and pass in another value that indicates the unpadded length.

If you're not using an RNN (what are you doing with variable length inputs then??) you must be processing them one-by-one (you didn't give us much detail there) so just break them up and send them into the queue one by one, read them off in TF as you need and recombine them with TF slicing and joining operations.