具有子流程的Python多处理停止工作

Question

Can you help me?
- Avoid the program to hang after ~16.000 permutations,
- All processes should stop if one finds the solution (see output),
- Give me general advice to make my code better, faster and learn more about python?

I try to brute force my True Crypt container because I forgot my password. I can still remember the words, but I miss the combination. Therefore I thought about this script.

What should it do? I hand it a word list and it should a) build all permutations and b) try if it finds the right one. Because I work on Windows 7 I use True Crypt command-line interface for trying, which I access via a python subprocess.

After writing a single threaded version I thought about making it faster. I first tried multiple threads then found the GIL and used multiprocessing. For me this is a learning experience, I never used Python before and know only Java and PHP. And I never did parallel programming before.

The code:

import subprocess, os, sys, time, multiprocessing, Queue, itertools

wordlist = [
"foo",
"bar",
"zoo",
"hello",
"World",
]
tcFile = r"C:\dev\tc-brute-force\test.pa"
tcProg = r"C:\Program Files\TrueCrypt\TrueCrypt.exe"
tcMountLetter = "z"
verbose = 5 # as higher as more output is shown fatal=0-5=trace
counter = 0
numberofworkers = multiprocessing.cpu_count()*2
curenttime = time.time()

def getDuration(starttime):
        return time.time() - starttime

def callTC(password, event):
    commandArgs = [
        tcProg,
        '/a',
        '/s',
        '/q',
        '/v', tcFile,
        '/l', tcMountLetter,
        '/p',  password,
    ]

    child = subprocess.Popen(commandArgs, \
        stderr=open(os.devnull, 'w'), \
        stdout=open(os.devnull, 'w'))
    result = child.communicate() # Really important to get error code!

    if verbose > 4:
        print subprocess.list2cmdline(commandArgs).rstrip() + \
            " Status out=" + str(result[0]) + \
            " err=" + str(result[1]) + \
            ", code=" + str(child.returncode)

    if child.returncode == 0:
        event.set()
        print "Successfully opened TrueCrypt file with '%s' at iteration %d, duration %.3fs" % (password, counter, getDuration(curenttime))

def callTCDaemon(queue, event):
    while True:
        if queue.empty():
            break
        else:
            password = queue.get()
            callTC(password, event)

if __name__ == '__main__':

    manager = multiprocessing.Manager()
    event = manager.Event()
    worker = manager.Queue(numberofworkers)

    # start processes
    pool = []
    for i in xrange(numberofworkers):
        process = multiprocessing.Process(target=callTCDaemon, args=(worker, event))
        process.start()
        pool.append(process)

    # generate permutations
    for x in xrange(1, (len(wordlist)+1) ):
        for permutation in itertools.permutations(wordlist, x):

            # shutdown if result is found
            if event.is_set():
                # wait till finished
                for p in pool:
                    p.join(2)

                print "Finished TrueCrypt brute-force, after %d attempts, duration %.3fs" % (counter, getDuration(curenttime))
                sys.exit(1)

            counter += 1    
            combination = ""

            # build string from permutation
            for i in range(0, len(permutation)):
                combination += permutation[i]

            # output progress
            if verbose == 4 and counter%100 == 0:
                print "%15d|%15.3fs: %s" % (counter, getDuration(curenttime), combination)

            # avoid queue overload
            while worker.qsize() > 100:
                if verbose > 3: print "Wait because queue is full, size=%d" % (worker.qsize)
                time.sleep(4)           

            worker.put(combination)

Example output (altered a bit):

C:\dev\tc-brute-force>python TrueCryptBruteForceProcesses.py
            100|         23.013s: fooWorld
            200|         48.208s: barHelloWorld
Successfully opened TrueCrypt file with 'Worldfoo' at iteration 0, duration 50.218s
Successfully opened TrueCrypt file with 'Worldbar' at iteration 0, duration 50.249s
Successfully opened TrueCrypt file with 'Worldzoo' at iteration 0, duration 50.260s
Successfully opened TrueCrypt file with 'Worldhello' at iteration 0, duration 50.304s
Successfully opened TrueCrypt file with 'foobarzoo' at iteration 0, duration 50.354s
Successfully opened TrueCrypt file with 'helloWorld' at iteration 0, duration 50.433s
Successfully opened TrueCrypt file with 'foobarhello' at iteration 0, duration 50.438s
Successfully opened TrueCrypt file with 'foobarWorld' at iteration 0, duration 50.440s
Successfully opened TrueCrypt file with 'foozoobar' at iteration 0, duration 50.473s
Finished TrueCrypt brute-force, after 209 attempts, duration 50.733s

Answer 1

There are three main parts in the brute-force code:

• generate passwords
• check a single password
• check many passwords concurrently

Generate passwords

Yield all possible permutations:

import itertools

def generate_passwords(wordlist):
    for password_length in range(1, len(wordlist) + 1): # no repeats
        for password in itertools.permutations(wordlist, password_length):
            yield " ".join(password)

Check password

Look at TRUECRYPT EXPLAINED . Maybe you don't need to spawn a subprocess for each password.

from subprocess import call

def valid_password(password):
    rc = call(true_crypt_command(password), close_fds=True)
    return rc == 0, password

Check many password concurrently

import sys
from multiprocessing.dummy import Pool # use threads

wordlist = "foo bar zoo hello World".split()
pool = Pool(20) # check 20 passwords in parallel
for i, (found, password) in enumerate(
    pool.imap_unordered(valid_password, generate_passwords(wordlist))):
    if i % 1000 == 0: # report progress
       sys.stderr.write("\rchecked %d" % i)
    if found:
       print("Found: '%s'" % password)
       break
else:
    sys.exit("failed to find")

pool.close() 
####pool.join() # uncomment if it is not the end

Except for true_crypt_command() function that generates TrueCrypt command line, it is the full source.

Another code example: Brute force http basic auth .

Will imap_unordered stop all the other threads/processes if one reaches rc==0?

If you exit the program immediately after the break then OS kills all remaining threads automatically (they are daemon thread so they won't survive).

If your program continues to run after the loop then upto 20 threads (size of the pool) might continue to run. You can wait for them to end if you uncomment pool.join() .