使用Tensorflow数据集在GPU上进行模型训练时的周期性开销

Question

As you can see it in the following code, I am trying to train a simple model on Tensorflow with a Tensorflow Dataset. The dataset is pretty huge and I suffle , repeat and batch it in order to do a stochastic gradien descent for training my model.

But I can observe a period overhead of the optimisation step (it is sess.run(train) in my code).

As you can see it here, every 5 steps, it needs 3s instead of 0.5 to do the optimisation.

Step 105 duration : 3.5233473777770996

Step 106 duration : 0.5653283596038818

Step 107 duration : 0.5391891002655029

Step 108 duration : 0.5480048656463623

Step 109 duration : 0.0415492057800293

Step 110 duration : 3.032115936279297

Step 111 duration : 0.5407207012176514

Step 112 duration : 0.5276811122894287

Step 113 duration : 0.5448746681213379

Step 114 duration : 0.04253268241882324

Step 115 duration : 3.1273345947265625

Moreover my GPU is almost all the time at 0% utilisation with around 90% of the memory used.

It seems that this overhead arrived when the Iterator finish to see all the dataset.

I am using Python 3.6 with Tensorflow 1.4 on Ubuntu 16.04.

Do you have any idea how I can speed up my training ?

Best,

import tensorflow as tf
import numpy as np
import os, time, multiprocessing
import matplotlib.pyplot as plt

def _floats_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(value=value.reshape(-1)))


def parser(record):
    num_features = 2000
    size_group = 300
    num_classes= 10
    class_indice = 0
    keys_to_features={
                'X': tf.FixedLenFeature([size_group*num_features],tf.float32),
                'label' : tf.FixedLenFeature([num_classes],tf.float32)}
    parsed = tf.parse_single_example(record, keys_to_features)

    label = parsed['label']
    label = tf.slice(label,[class_indice],[1])
    label = tf.squeeze(label) # To get a vector one dimension
    X = parsed['X']
    X= tf.reshape(X, [size_group,num_features])
    return X, label


def test_train_w_dataset():

    # Definition of the size 
    num_features = 2000
    num_ex = 2000
    size_group = 300
    num_classes = 10
    batch_size= 480
    max_iters = 300
    buffer_size = 10000

# Creation of the Dataset 
filename_tfrecords = 'tmp.tfrecords'
if not(os.path.isfile(filename_tfrecords)): # If the file doesn't exist we will create it
    print("Start creating the Dataset")
    writer = tf.python_io.TFRecordWriter(filename_tfrecords)

    for i in range(num_ex):
        if i % 1000 == 0: print("Step :",i)
        X = np.random.normal(size=(size_group,num_features))
        vectors =  2*np.random.randint(0,2,(num_classes,1))-1
        features=tf.train.Features(feature={
                    'X': _floats_feature(X),
                    'label' : _floats_feature(vectors)})
        example = tf.train.Example(features=features)       
        writer.write(example.SerializeToString())
    writer.close()
else:
    print("The dataset tfrecords already exist")

train_dataset = tf.data.TFRecordDataset(filename_tfrecords)
num_proc = multiprocessing.cpu_count()
train_dataset = train_dataset.map(parser,
                                    num_parallel_calls=num_proc)
dataset_shuffle = train_dataset.shuffle(buffer_size=buffer_size,
                                             reshuffle_each_iteration=True) 
dataset_shuffle = dataset_shuffle.batch(batch_size)
dataset_shuffle = dataset_shuffle.repeat() 
dataset_shuffle = dataset_shuffle.prefetch(batch_size) 
shuffle_iterator = dataset_shuffle.make_initializable_iterator()
X_, y_ = shuffle_iterator.get_next()

W=tf.Variable(tf.random_normal([num_features], stddev=1.),name="weights")
W=tf.reshape(W,(1,1,num_features))
Prod=tf.reduce_sum(tf.multiply(W,X_),axis=2)
Max=tf.reduce_max(Prod,axis=1)
Tan= tf.reduce_sum(tf.multiply(tf.tanh(Max),y_))
loss= tf.add(Tan,tf.reduce_sum(tf.multiply(W,W)))

LR = 0.01
restarts = 1
optimizer = tf.train.GradientDescentOptimizer(LR) 
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
train = optimizer.minimize(loss)  
print("The graph is defined")
sess = tf.Session(config=config)

durationTab = []

for essai in range(restarts+1):
    # To do need to reinitialiszed
    t0 = time.time()
    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())
    sess.run(shuffle_iterator.initializer)
    t1 = time.time()
    duration = t1 - t0
    print('Duration of initialization : ',duration)
    for step in range(max_iters):
        t0 = time.time()
        sess.run(train)
        t1 = time.time()
        duration = t1 - t0
        print("Step ",str(step),' duration : ',duration)
        durationTab += [duration]


plt.plot(durationTab)
plt.ylabel('Duration')
plt.xlabel('Iteration')
plt.show()

if __name__ == '__main__':

    test_train_w_dataset()

Answer 1

For GPU utilization, make sure you use the gpu optimized binary. Check operation placement (in tensorboard for example). Force placement of the operations on the gpu (see tf.device).

For the periodic spikes there could be a few reasons:

• Other processes block access to CPU/GPU/RAM/Disk and you need to wait for that to pass. You can try to kill other superfluous tasks that might be running on your system.
• You run out of ram. Check how much of the swap space is used. If it's growing when you run then the spikes might be just system thrashing, though it looks too well behaved for that.
• Disk access. You mentioned that this sees to correlate with looping over the data. It could be the system just needs to read the data again so you need to wait for the disk, though usually that's not visible. You could speed it up by making sure the data is contiguous on hard-drive, move it to an SSD or RAM.

Since a lot of the reasons have to do with RAM, you should probably try a smaller model (smaller batches, less layers, less nodes/layer) and see if it goes away. If it does then you need to go out and buy more RAM.

Answer 2

It seems that adding dataset_shuffle = dataset_shuffle.cache() between the batch and repeat function remove those periodic overhead. Nevertheless, I am not sure that the Dataset is fully read with the use of this command.