Loss nan for triplet loss Tensorflow

Question

I'm trying to train a triplet loss model in TensorFlow (tensorflow==1.13.1) but the loss goes to NAN after the first iteration. I already read a lot of tips from this forum but nothing has worked so far. Smaller learning rate (changed 0.01 to 0.0001). Also tried a bigger batch size because someone suggested that a batch might not contain a full triplet if the size is to small. However, I built the batches myself with this code: '''

def get_triplets(self):
    a = random.choice(self.anchors)
    p = random.choice(self.positives[a])
    n = random.choice(self.negatives[a])

    return self.images[a], self.images[p], self.images[n]

def get_triplets_batch(self, n):
    idxs_a, idxs_p, idxs_n = [], [], []
    for _ in range(n):
        a, p, n = self.get_triplets()
        idxs_a.append(a)
        idxs_p.append(p)
        idxs_n.append(n)
    return idxs_a, idxs_p, idxs_n

I have also seen that people add a small number to the prediction to avoid this NAN. I don't know if I have to do this for triplet loss or where. The code for training is this

'''

_, l, summary_str = sess.run([train_step, loss, merged], feed_dict={anchor_input: batch_anchor, positive_input: batch_positive, negative_input: batch_negative})

I am new to using StackOverflow so I wanted to include the files I used in a Github in case I provided too little information. https://github.com/AndreasVer/Triplet-loss-tensorflow.git

Answer 1

I have released a package for triplet generation to avoid the NaN loss. This usually happens when your batch does not have any positive pairs.

https://github.com/ma7555/kerasgen (Disclaimer: I am the owner)