Pause Python program on the fly (and resume)

Question

I am building a machine learning algorithm(like neural network) where class variables (ie numpy matrices ) represent various parameters of the system

Training the system is done by iteratively update all class variables. The more iterations the better. I want to get up every morning and check the class variables. After that I want to resume the program

I am calling the program in an interactive terminal . Here is what I can think of:

1. Print to terminal -> matrices too large, won't be helpful
2. save to disk and load in another terminal
3. set_trace() , but requires knowing when to pause beforehand

Is it possible to pause the program on the fly and play with the class variables and then resume ?

If anyone needs more details, the program is here: github link

Answer 1

I'm not familiar with numpy, but here is a simple class that can stop and resume:

class Program():

    def run(self):
        while 1:
            try:
                self.do_something()
            except KeyboardInterrupt:
                break

    def do_something(self):
        print("Doing something")


# usage:

a = Program()
a.run()

# will print a lot of statements

# if you hit CTRL+C it will stop
# then you can run it again with a.run()

Answer 2

What if you modify the model.do_EM() method to save the current state at each step and check a configuration file?

 def do_EM(self, n_iteration = 10):
    self.visualizer.visualize(self.param_alpha, self.param_mu, self.param_sigma)
    for i in range(n_iteration):
      print "iteration:", i 
      self.step_E()
      print "done step_E. ",
      self.step_M()
      print "done step_M. "
      self.visualizer.visualize(self.param_alpha, self.param_mu, self.param_sigma)

      # Save current state
      self.log.write( ... )
      # Check for config changes
      self.config.update( ... )