有关回归神经网络推广的建议

Question

I've trained a deep neural network of a few hundreds of features which analyzes geo data of a city, and calculate a score per sample based on the profile between the observer and the target location. That is, the longer the distance between the observer and target, the more features I will have for this sample. When I train my NN with samples from part of a city and test with other parts of the same city, the NN works very well, but when I apply my NN to other cities, the NN starts to give high standard deviation of errors, especially on cases which the samples of the city I'm applying the NN to generally has more features than samples of the city I used to train this NN. To deal with that, I've appended 10% of empty samples in training which was able to reduce the errors by half, but the remaining errors are still too large compare to the solutions calculated by hand. May I have some advise of generalize a regression neural network? Thanks!

Answer 1

I was going to ask for more examples of your data, and your network, but it wouldn't really matter.

How to improve the generalization of a regression neural network?

You can use exactly the same things you would use for a classification neural network. The only difference is what it does with the numbers that are output from the penultimate layer!

I've appended 10% of empty samples in training which was able to reduce the errors by half,

I didn't quite understand what that meant (so I'd still be interested if you expanded your question with some more concrete details), but it sounds a bit like using dropout. In Keras you append a Dropout() layer between your other layers:

...
model.append(Dense(...))
model.append(Dropout(0.2))
model.append(Dense(...))
...

0.2 means 20% dropout, which is a nice starting point: you could experiment with values up to about 0.5. You could read the original paper or this article seems to be a good introduction with keras examples.

The other generic technique is to add some L1 and/or L2 regularization, here is the manual entry .

I typically use a grid search to experiment with each of these, eg trying each of 0, 1e-6, 1e-5 for each of L1 and L2, and each of 0, 0.2, 0.4 (usually using the same value between all layers, for simplicity) for dropout. (If 1e-5 is best, I might also experiment with 5e-4 and 1e-4.)

But, remember that even better than the above are more training data. Also consider using domain knowledge to add more data, or more features.