Keras LSTM 后实例运行缓慢

Question

I have an EC2 instance in AWS running a python script creating models for a bunch of stocks using Keras LSTM. Every model takes about 6 minutes to be fitted and stored. The problem that I am facing is a slowdown in AWS instance, so the models are not finished to be fitted. It takes more and more time for each model to be calculated. Let's say the first 6-7 models are calculated right in about 6 minutes each, but then the eighth model takes more than 15 minutes and the ninth more than an hour and then it stops.

Curiously I ran the script saturday morning without a problem. Then I tried again and it stopped. And now every time I try it gets stuck after a couple of models. I stopped and restarted the instance. It seems to deblock a bit, so if I run the script it starts fine (about 6 minutes) but again, after a while, the time increases and stops.

It's a really simple script: I have the tickers in a list and then:

for ticker in tickers:

get_and_prepare_data(ticker)

create_and_save_model(ticker)

As I suppose, the memory gets full or something like that. So maybe there is a way to free memory within the script. I don't know... or meybe even if I restart the instance there is some memory allocated in the instance...

Thanks in advance

Answer 1

OK, after searching a while I found the solution using Keras Backend Clear_session().

You can find more info here https://www.tensorflow.org/api_docs/python/tf/keras/backend/clear_session