Tensorflow：可变序列长度和批次大小

Question

My dataset consists of sentences. Each sentence has a variable length and is initially encoded as a sequence of vocabulary indexes, ie. a tensor of shape [sentence_len]. The batch size is also variable.

I have grouped sentences of similar lengths into buckets and padded where necessary, to bring each sentence in a bucket to the same length.

How could I deal with having both an unknown sentence length AND batch size?

My data provider would tell me what the sentence length is at every batch, but I don't know how to feed that -> the graph is already built at that point. The input is represented with a placeholder x = tf.placeholder(tf.int32, shape=[batch_size, sentence_length], name='x') . I can turn batch_size or sentence_length to None , but not both.

UPDATE: in fact, interestingly, I can set both to None , but I get Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. Note: the next layer is an embedding_lookup.

I'm not sure what this means and how to avoid it. I assume it has something to do with using tf.gather later, which I need to use. Alternatively is there any other way to achieve what I need?

Thank you.

Answer 1

Unfortunately there is no workaround here unless you provide a tf.Variable() (which is not possible in your case) to the parameter of tf.nn.embedding_lookup() / tf.gather() . This is happening because, When you declare them with a placeholder of shape [None, None] , from tf.gather() function tf.IndexedSlices() become a sparse tensor .

I have already done projects facing this warning . What I can tell you that if there is a tf.nn.dynamic_rnn() next to the embedding_lookup then make the parameter named swap_memory of tf.nn.dynamic_rnn() to True . Also to avoid OOM or Resource Exhausted error make the batch size smaller (test for different batch size).

There are already some good explanation on this. Please refer to the following Question of the Stackoverflow.

Tensorflow dense gradient explanation?