Python on Mac Terminal - Killed: 9

Question

I wrote the following program in Python (working with a text file of 1000 words and finding all permutations of 4 words) and ran it on the Mac terminal:

from itertools import permutations

with open('wordlist.txt') as f:
    content = f.readlines()
content = [x.rstrip() for x in content] 

g = open("finalwordlist.txt", "a")

g.write('%s' % "\n".join(map("".join, permutations(content,4))))

After a short while, I got the following output from the terminal:

Killed: 9

No output was written to the output text file, so I assume the program terminated before the write() step.

This program worked when I was finding permutations of sizes less than 4 (ex. 1,2,3). Was the Killed: 9 because of the size of all the permutations? Or something to do with the Mac terminal environment?

How would I be able to work around this error? Thanks!

Answer 1

Let's see... you've generated a single Python object of roughly 1000^4 permutations of 4 words each. Average word length in usage is about 4.5 letters, but the average length of a lexicon is more -- I'll take a stab that it's more than 25 characters per permutation. This gives you 2.5 * 10^13 bytes in your object.

How does your RAM allocation do with a single string of 250 Terabytes? If you blow your memory limits, what is the error message?

Yes, this is an issue with you Mac environment: SegFault is not an error message. It appears that your system crashed so hard, that it never got back to issue the Python memory fault.

BTW, the repair is "obvious" -- quit trying to write 250 Tb in one call , There is nothing in the output differentiation that requires the block write. I hope, Instead. write them one at a time.

for combo in permutations(content,4):
    g.write("".join(combo) + "\n")