Python 2.7.3 memory error

Question

I have a specific case with python code. Every time I run the code, the RAM memory is increasing until it reaches 1.8 gb and crashes.

import itertools
import csv
import pokersleuth

cards = ['2s',  '3s',   '4s',   '5s',   '6s',   '7s',   '8s',   '9s',   'Ts',   'Js',   'Qs',   'Ks',   'As',   '2h',   '3h',   '4h',   '5h',   '6h',   '7h',   '8h',   '9h',   'Th',   'Jh',   'Qh',   'Kh',   'Ah',   '2c',   '3c',   '4c',   '5c',   '6c',   '7c',   '8c',   '9c',   'Tc',   'Jc',   'Qc',   'Kc',   'Ac',   '2d',   '3d',   '4d',   '5d',   '6d',   '7d',   '8d',   '9d',   'Td',   'Jd',   'Qd',   'Kd',   'Ad']
flop = itertools.combinations(cards,3)

a1 = 'Ks' ; a2 = 'Qs'
b1 = 'Jc' ; b2 = 'Jd'

cards1 = a1+a2
cards2 = b1+b2

number = 0
n=0
m=0

for row1 in flop:
    if (row1[0] <> a1 and row1[0] <>a2 and row1[0] <>b1 and row1[0] <>b2) and (row1[1] <> a1 and row1[1] <>a2 and row1[1] <>b1 and row1[1] <>b2) and (row1[2] <> a1 and row1[2] <> a2 and row1[2] <> b1 and row1[2] <> b2):
        for row2 in cards:
            if (row2 <> a1 and row2 <> a2 and row2 <> b1 and row2 <> b2 and row2 <> row1[0] and row2 <> row1[1] and row2 <> row1[2]):
                s = pokersleuth.compute_equity(row1[0]+row1[1]+row1[2]+row2, (cards1, cards2))
                if s[0]>=0.5:
                    number +=1
                    del s[:]
                del s[:]

        print number/45.0
        number = 0
        n+=1

Answer 1

My tests where the memory leak occurred were inconclusive, but under the assumption that it didn't happen in montecarlo.dll, I thought I'd try multiprocessing.Pool() and split the work in smaller chunks that I load off to some processes which I could terminate before they start using excessive memory:

from itertools import combinations, product, islice
from multiprocessing import Pool
from pokersleuth import compute_equity

num_procs = 4
num_jobs = 256
chunk_size = num_procs * num_jobs

join = ''.join

drawn = a1, a2, b1, b2 = 'Ks', 'Qs', 'Jc', 'Jd'
pairs = (a1 + a2, b1 + b2)

deck = (join(reversed(c)) for c in product('shcd', '23456789TJQKA'))
deck = [card for card in deck if card not in drawn]

def compute_chances(cards):
    return sum(compute_equity(cards + card, pairs)[0] >= 0.5
               for card in deck if card not in cards) / 45.0

if __name__ == '__main__':
    combis = (join(each) for each in combinations(deck, 3))
    i = 0
    while True:
        pool = Pool(processes=num_procs)
        chunk = list(islice(combis, chunk_size))
        for i, chances in enumerate(pool.imap(compute_chances, chunk), i + 1):
            print i, chances
        pool.terminate()
        if len(chunk) < chunk_size:
            break

The results are the same as in your program.

Here's what Task Manager says about memory consumption for the last 7 loops of 17 (17296 combinations with a chunk_size of 1024):

Each loop used about 400 MB and it took 34 miuntes to process all combinations.

Instead of manually typing in a whole deck of cards, I have the computer create it for me. I refuse to do things that a computer can do.

To keep the amount of data passed to each process as small as possible, I only send each combination of three cards to compute_chances() and have everything else computed there.

I'm not shure if montecarlo.dll is reentrant, but the results seem to indicate that it is.

The values of num_procs and num_jobs in my code worked well on my machine. You should play around with them to find optimal settings for yours.

Answer 2

You're running on Linux (am I right?), and you're hitting the maximum process image size on your system, because processes on linux cannot reduce their memory size. windows.

Your options are to split this up so that it can resume after hitting the memory limit, compiling a kernel with a higher process size limit , or running on windows .