tf.contrib.learn.Estimator类是否使用所有数据？

Question

I have a training operation :

def train(self, batch_size, steps):
    x, y = self.generate(batch_size*steps)
    print('Training...')
    self.classifier.fit(x=x, y=y, batch_size=batch_size, steps=steps)

And the classifier is defined here :

self.classifier = learn.Estimator(model_fn=self.model, model_dir=self.dir)

My question is this - if x and y are bigger in size than batch_size , do they all get used when moving along the steps ? For example, if the batch_size is 128, but both x and y are 128,000 items, do all items get trained on by the time steps reaches 1000 steps?

I'm asking this because the generate function takes a really long time, and I want to know if most of that time is wasted if it is actually the case that only the first batch_size of examples from it are used.

Note: I know the x and y arguments are deprecated and I should use input_fn instead, so the question applies to both ways, for example if the training operation was this:

def train(self, batch_size, steps):
    self.classifier.fit(input_fn=lambda: self.generate(batch_size*steps), steps=steps)

In other words, the input_fn function, or the function that generates x,y tensors, should it be called with a demand for batch_size*steps data examples or just batch_size , because only that would be processed anyway?

Answer 1

If your batch_size is 128 and if you have 128000 items then all items get trained when steps reached 1000 steps. The estimator only pulls what you have described in batch_size for every training step .

I have written a piece of code, which reads inputs (each sample is just 1), and every training step sums up the ones it has seen till that time, which tells you how many data samples it has read till that time.

from tensorflow.contrib.learn.python.learn.estimators import model_fn as model_fn_lib

tf.logging.set_verbosity(tf.logging.INFO)

def model_fn(features, labels, mode):

   _sum = tf.Variable(0, dtype=tf.int32)   

   if mode == learn.ModeKeys.TRAIN:
       # Update global_step
       global_step=tf.contrib.framework.get_global_step()
       global_step_op = tf.assign(global_step, global_step+1)

       # Sum of all the elements in a batch
       update_sum_op = tf.assign_add(_sum, tf.reduce_sum(features)) 
       update_op = tf.group(global_step_op, update_sum_op)
       loss = _sum

   predictions = {'out': tf.identity(_sum, 'sum')}

   return model_fn_lib.ModelFnOps(mode=mode, predictions=predictions, loss=loss, train_op=update_op)


X = np.ones((1000, 1), dtype=np.int32)
y = np.ones((1000, 1), dtype=np.int32)

sess = tf.InteractiveSession()

feature_classifier = learn.SKCompat(learn.Estimator(model_fn=model_fn))
tensors_to_log = {'out':'sum'}
logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=1)
feature_classifier.fit(x=X, y=y, batch_size=123, steps=7, monitors=[logging_hook])

Here the total data samples is 1000 and the batch_size=123 and steps=7 .

The output at each step is:

INFO:tensorflow:out = 123
INFO:tensorflow:out = 246 (0.004 sec)
INFO:tensorflow:out = 369 (0.003 sec)
INFO:tensorflow:out = 492 (0.003 sec)
INFO:tensorflow:out = 615 (0.003 sec)
INFO:tensorflow:out = 738 (0.003 sec)
INFO:tensorflow:out = 861 (0.003 sec)