In python, about storing data to files

Question

I am a novice in programming and learning python 3.x about 3 or 4 months.

Nowadays, I'm trying to make a program to find the solutions of some 'magic squares'.

It is known that the 6x6 magic square has more than 200,000,000 solutions.

So, the numbers are too big to store in normal PC memory that I want

to store the calculated and found solutions to files from time to time.

Let's say, I want to save the solutions to a file when they become 1,000,000.

Like this following in short:

if len(resultList) == 1000000:
    file = open('{0}x{0} PMS Solutions {1:03}.txt'.format(ansr, fileNum), 'w')
    file.write(resultList)
    file.close()
    resultList = []

Then, while a file is being made, the process to find new solutions doesn't work.

My question:

Is there any way to make both of the processes-calculating and storing-work simultaneously?

Answer 1

If you are using python3.3 a simple and elegant way of achieving what you want is using a ThreadPoolExecutor :

def save_to_file(data):
    fname = '{0}x{0} PMS Solutions {1:03}.txt'.format(ansr, fileNum)
    with open(fname, 'w') as fout:
        fout.write(data)   # save the list to file in some way

Use it like:

executor = ThreadPoolExecutor(max_workers=2)

# ...

if len(resultList) >= 1000000:
    future = executor.submit(save_to_file, resultList)
    resultList = []

The same can be done using the threading module in python versions prior to 3.3 Something like:

thread = None

if len(resultList) >= 1000000:
    if thread is not None:
        thread.join()  # finish saving the other solutions first
    thread = threading.Thread(target=save_to_file, args=(resultList,))
    thread.start()
    resultList = []