多处理/线程-高估了，或者我缺少什么？

Question

I have a program that I am in the process of making multithreaded and multiprocessed. But in my initial tests, I find that old-fashioned serial processing beats either multiprocessing (using more CPU cores) or threading (using more threads pr. CPU core, but only one core at a time) hands down!!!

To illustrate I have made this simple code-snippet. In the main section of the script I have marked the 3 types of processing, so you can easily pick which one type of processing you want, and rem-out the other 2 options (rem-out the full sections to disable the feature):

My script iterates through a list of 2 strategies. And for each strategy it iterates through a list of 193 tickers (stock_list).

In the main section you may select which type of processing you want to test/employ:

1. Section is normal serial processing with only one CPU core and one thread.
2. Section is Multiprocessing using all the available CPU cores in the system.
3. Section is Threading using a queue of 40 threads to process the list.

I do not do anything fancy with them in this simple test-script, but only sleep 0.01s pr. iteration for me to get a feel for which is fastest.

Bly me, but it seems that processing the list the old-fashioned serial way is slightly faster than any of the other types...! My test results show these run-times:

1. Serial : 3.86s
2. Multiprocessing (cores) 4.03s
3. Multithreading (threads) 3.90s

I must be missing the point, and must have made a mistake in the code below. Please could someone with multiprocessing experience shine some light on this conundrum.

How do I speed up processing the stock_list through the strategies and make this example code run the fastest it possibly can?

import time
import threading
from threading import Thread, Lock
from queue import Queue
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

start = time.time()     # Start script timer

stock_list = ['aan', 'anf', 'ancx', 'ace', 'atvi', 'aet', 'agco', 'atsg', 'awh', 'all', 'afam', 'alj', 'dox', 'acas', 'afg', 'arii', 'asi', 'crmt', 'amkr', 'nly', 'anh', 'acgl', 'arw', 'aiz', 'atw', 'avt', 'axll', 'axs', 'blx', 'bkyf', 'bmrc', 'bku', 'banr', 'b', 'bbt', 'bbcn', 'bhlb', 'bokf', 'cjes', 'caci', 'cap', 'cof', 'cmo', 'cfnl', 'cacb', 'csh', 'cbz', 'cnbc', 'cpf', 'cvx', 'cb', 'cnh', 'cmco', 'cnob', 'cop', 'cpss', 'glw', 'crox', 'do', 'dds', 'dcom', 'dyn', 'ewbc', 'eihi', 'ebix', 'exxi', 'efsc', 'ever', 're', 'ezpw', 'ffg', 'fisi', 'fdef', 'fibk', 'nbcb', 'banc', 'frc', 'frf', 'fcx', 'gm', 'gco', 'gsol', 'gs', 'glre', 'hbhc', 'hafc', 'hdng', 'hcc', 'htlf', 'hele', 'heop', 'hes', 'hmn', 'hum', 'im', 'irdm', 'joy', 'jpm', 'kalu', 'kcap', 'kmpr', 'kss', 'lbai', 'lf', 'linta', 'lmca', 'lcut', 'lnc', 'lmia', 'mtb', 'mgln', 'mant', 'mpc', 'mcgc', 'mdc', 'taxi', 'mcc', 'mw', 'mofg', 'mrh', 'mur', 'mvc', 'myrg', 'nov', 'nci', 'navg', 'nni', 'nmfc', 'nnbr', 'nwpx', 'oln', 'ovti', 'olp', 'pccc', 'pre', 'pmc', 'psx', 'phmd', 'pjc', 'ptp', 'pnc', 'bpop', 'pfbc', 'pri', 'pl', 'rf', 'rnr', 'regi', 'rcii', 'rjet', 'rbcaa', 'sybt', 'saft', 'sasr', 'sanm', 'sem', 'skh', 'skyw', 'sfg', 'stt', 'sti', 'spn', 'sya', 'tayc', 'tecd', 'tsys', 'ticc', 'twi', 'titn', 'tol', 'tmk', 'twgp', 'trv', 'tcbk', 'trn', 'trmk', 'tpc', 'ucbi', 'unm', 'urs', 'usb', 'vlo', 'vr', 'voxx', 'vsec', 'wd', 'wres', 'wbco', 'wlp', 'wfc', 'wibc', 'xrx', 'xl']

tickers = []
strategies = []

def do_multiproces_work(ticker):
    print(threading.current_thread().name,strategy,ticker)
    time.sleep(0.01)


#==============================================================================
# Threading

# lock to serialize console output
lock = Lock()

def do_work(item):

    try:

        with lock: # This is where the work is done
            print(threading.current_thread().name,strategy,item)
            time.sleep(.01) # pretend to do some lengthy work.

    except Exception as e:
            print(str(e))

# The worker thread pulls an item from the queue and processes it
def worker():
    try:
        while True:
            item = q.get()
            do_work(item)
            q.task_done()

    except Exception as e:
            print(str(e))

#==============================================================================

if __name__ == '__main__':

    strategies = ['strategy0', 'strategy1']
    #==============================================================================
    # Strategies iteration
    #==============================================================================
    try:
        for strategy in strategies:
            ##=========================================================================
            ## Tickers iteration
            ##=========================================================================
           # 1. Normal Serial processing
            for ticker in stock_list:
                do_multiproces_work(ticker)

            #==============================================================================
#           # 2. Pure Multiprocessing (without multiple threads)
#            '''
#            pool = ThreadPool()
#            # Sets the pool size, If you leave it blank,
#            it will default to the number of Cores in your machine.
#            '''
#
#            # Make the Pool of workers
#            pool = ThreadPool()
#
#            # Do work and return the results
#            # Multiproc. Without threading
#            pool.map(do_work, stock_list)
#
#            #results = pool.map(urllib2.urlopen, urls)
#
#            #close the pool and wait for the work to finish
#            pool.close()
#            pool.join()

            #==============================================================================

#            # 3. Threading (from only one processor core)
#            # Create the queue and thread pool.
#            global q
#            q = Queue()
#            for i in range(40):  # No of parallel threads/Queues - 40 et godt valg.
#                 t = threading.Thread(target=worker)  # Kalder arb fct.
#                 t.daemon = True  # thread dies when main thread (only non-daemon thread) exits.
#                 t.start()
#
#            # stuff work items on the queue (in this case, just a number).
#            for item in stock_list:  # INPUT LISTE TIL ANALYSE
#                q.put(item)
#
#            q.join()       # block until all tasks are done
            #==============================================================================
    except Exception as e:
            print(str(e))

    # Stopping and printing result from script-timer
    seconds = time.time()-start

    m, s = divmod(seconds, 60)
    h, m = divmod(m, 60)
    print('Script finished in %.2f seconds' %(time.time()-start))

Answer 1

By using lock in the work function, will you make the code more or less serial instead of multithreaded. Is it needed?

If analysing the tickers with one strategy does not affect the others, then there is no need to use lock.

Lock should only be used if they access a shared resource, example a file, or printer.

Answer 2

You didn't specify, so I'm going to assume you are using the C implementation of Python 2.7.

In this case, Python threads do not run in parallel. There is something called the Global Interpreter Lock which prevents this. Hence, no speedup for you, since you now do the same waiting as in the single-thread case, but also a lot of task switching and waiting for threads to finish.

I'm not sure what the deal with the multiprocessing variant is, but I suspect it is overhead. Your example is rather contrived, no only in that you wait instead of doing work, but also because you don't wait for very long. One typically wouldn't use multiprocessing for something that only takes four seconds total...

As a footnote, I don't understand what your claim that threading uses only one core is based on. AFAIK, it's not a python constraint.