How to theano tensor sum an array with nan values

Question

I am writing an autoencoder model in using Theano, (I am very new to Theano). The cost function has a sparsity constraint. The KL divergent function produces NaN values in the array, when I sum the array to add it to the overall cost it gives a NaN value. Is there any way to get around this problem.

KL = rho * (T.log(rho/rho_hat)) + (1 - rho) * (T.log((1 - rho)/(1 - rho_hat)))
# sparsity cost
SPcost = beta * KL.nansum()
# the loss function 
loss = T.nnet.categorical_crossentropy(y_hat, y).mean() + loss_reg 

I am trying to debug using a test function

test=theano.function([X], SPcost)
test(train_X)

SPcost should give me a single scalar value, instead it shows array(nan) I have tried to use numpy nansum() but that gives me an error. What is the correct way of summing the array with the NaN values? Any suggestion would be much appreciated.

due to numerical issues NaN may pop up anytime, so it is basically unavoidable. I looked for functions in theano for dealing with nan but did not find anything that helps me.

Answer 1

When you're stuck with nan 's of which you fully understand the implications, switch() and isnan() offer a way out:

KL = T.switch(T.isnan(KL), 0., KL)

Where KL would be a tensor containing nan 's that you wish to replace with 0 's (albeit, at a certain cost).

You can then T.sum(KL) like any normal day.