Access python program data while running

Question

I have a python program that's been running for a while, and because of an unanticipated event, I'm now unsure that it will complete within a reasonable amount of time. The data it's collected so far, however, is valuable, and I would like to recover it if possible.

Here is the relevant code

from multiprocessing.dummy import Pool as ThreadPool 
def pull_details(url):
    #accesses a given URL
    #returns some data which gets appended to the results list

pool = ThreadPool(25) 

results = pool.map(pull_details, urls)
pool.close() 
pool.join() 

So I either need to access the data that is currently in results or somehow change the source of the code (or somehow manually change the program's control) to kill the loop so it continues to the later part of the program in which the data is exported (not sure if the second way is possible).

It seems as though the first option is also quite tricky, but luckily the IDE (Spyder) I'm using indicates the value of what I assume is the location of the list in the machine's memory (0xB73EDECCL).

Is it possible to create a C program (or another python program) to access this location in memory and read what's there?

Answer 1

Can't you use some sort of mechanism to exchange data between the two processes, like queues or pipes.

something like below: from multiprocessing import Queue from multiprocessing.dummy import Pool as ThreadPool

def pull_details(args=None):
  q.put([my useful data])

q = Queue()
pool = ThreadPool(25) 

results = pool.map(pull_details(args=q), urls)
while not done:
    results = q.get()
pool.close() 
pool.join()