TensorFlow：张量入队元组抛出TypeError

Question

I have a time based dataset which consists of categorical features, real values features, a mask that states whether the given feature was present at the time, and a "Deltas" array that contains the length of time since a value has been present.

I want to build a queue of tuples of these tensors so that the categorical features can be converted to one hot and so that the data, mask, and deltas can be used in different parts of the model. Below is some code that I wrote to do this:

import tensorflow as tf
import threading
import numpy as np

# Function to generate batches of data
def nextBatch(batch_size):
    n_steps = 14
    batch = []
    for _ in range(batch_size):
        # Create tuple of tensors
        ex = (np.random.randint(0,5, (n_steps, 2)),
              np.random.randn(n_steps, 10),
              np.random.randint(0,2, (n_steps, 12)),
              np.random.randint(0,2000, (n_steps, 12)))
        batch.append(ex)

    return batch


# Graph to enqueue data
tf.reset_default_graph()

q = tf.PaddingFIFOQueue(1000,
                        [np.uint16, tf.float32, tf.uint16, tf.uint16],
                        [(None,5), (None,48), (None,53), (None,53)])

def enqueue_op():
    # Stop enqueuing after 11 ops
    i = 0
    while True:
        q.enqueue_many(nextBatch(100))
        i += 1
        if i >11:
            return      

# Start enqueuing
t = threading.Thread(target=enqueue_op)
t.start()

When I run this I get a TypeError:

TypeError: Expected uint16, got array(...) of type 'ndarray' instead.

I am not sure what I am doing wrong, is it the dtype definition when I create my queue?

Answer 1

There are a few problems here:

1. Your thread is calling q.enqueue_many() repeatedly. Despite its (slightly confusing) name, the q.enqueue_many() method does not immediately enqueue data in the queue, but rather it returns a tf.Operation that must be passed to sess.run() to add the tensors in the queue. The code that runs in the separate thread is creating 10 enqueue-many operations and discarding them, which is probably not what you intended.

2. The return value of nextBatch(100) is a list of 100 tuples of 4 arrays. The q.enqueue_many() method expects a tuple of 4 arrays. If you want to enqueue a list of 100 tuples, you'll need to run a q.enqueue() op 100 times, or stack together the 100 arrays for each tuple component so that you have a single tuple of four arrays.

3. The arrays produce in nextBatch() don't match the shapes of the queue components. Assuming n_steps is the dimension that could be variable (for the purposes of padding), the function should produce arrays of (n_steps, 5) , (n_steps, 48) , (n_steps, 53) , and (n_steps, 53) to match the queue definition.

Here's a version of your code that works as I assume you intended:

import tensorflow as tf
import threading
import numpy as np

# Function to generate batches of data                                                                                                                         
def nextBatch(batch_size):
  n_steps = 14
  batch = []
  for _ in range(batch_size):
    # Create tuple of tensors                                                                                                                                  
    ex = (np.random.randint(0,5, (n_steps, 5)),
          np.random.randn(n_steps, 48),
          np.random.randint(0,2, (n_steps, 53)),
          np.random.randint(0,2000, (n_steps, 53)))
    batch.append(ex)
  return batch

q = tf.PaddingFIFOQueue(1000,
                        [tf.uint16, tf.float32, tf.uint16, tf.uint16],
                        [(None, 5), (None, 48), (None, 53), (None, 53)])

# Define a single op for enqueuing a tuple of placeholder tensors.
placeholders = [tf.placeholder(tf.uint16, shape=(None, 5)),
                tf.placeholder(tf.float32, shape=(None, 48)),
                tf.placeholder(tf.uint16, shape=(None, 53)),
                tf.placeholder(tf.uint16, shape=(None, 53))]
enqueue_op = q.enqueue(placeholders)

# Create a session in order to run the enqueue_op.
sess = tf.Session()

def enqueue_thread_fn():
  for i in range(10):
    batch = nextBatch(100)
    for batch_elem in batch:
      # Each call to `sess.run(enqueue_op, ...)` enqueues a single element in
      # the queue.
      sess.run(enqueue_op, feed_dict={placeholders[0]: batch_elem[0],
                                      placeholders[1]: batch_elem[1],
                                      placeholders[2]: batch_elem[2],
                                      placeholders[3]: batch_elem[3]})

# Start enqueuing                                                                                                                                              
t = threading.Thread(target=enqueue_thread_fn)
t.start()
t.join()
sess.close()