如何优化图像比较脚本的性能？

Question

I wrote a script that compares a huge set of images (more than 4500 files) against each other using a root mean square comparison. At first it resizes each image to 800x600 and takes a histogram. After that it builds an array of combinations and distributes them evenly to four threads which calculate the root mean square of every combination. Images with a RMS below 500 will be moved into folders to be manually sorted out later.

#!/usr/bin/python3

import sys
import os
import math
import operator
import functools
import datetime
import threading
import queue
import itertools
from PIL import Image


def calc_rms(hist1, hist2):
    return math.sqrt(
        functools.reduce(operator.add, map(
            lambda a, b: (a - b) ** 2, hist1, hist2
        )) / len(hist1)
    )


def make_histogram(imgs, path, qout):
    for img in imgs:
        try:
            tmp = Image.open(os.path.join(path, img))
            tmp = tmp.resize((800, 600), Image.ANTIALIAS)
            qout.put([img, tmp.histogram()])
        except Exception:
            print('bad image: ' + img)
    return


def compare_hist(pairs, path):
    for pair in pairs:
        rms = calc_rms(pair[0][1], pair[1][1])
        if rms < 500:
            folder = 'maybe duplicates'
            if rms == 0:
                folder = 'exact duplicates'
            try:
                os.rename(os.path.join(path, pair[0][0]), os.path.join(path, folder, pair[0][0]))
            except Exception:
                pass
            try:
                os.rename(os.path.join(path, pair[1][0]), os.path.join(path, folder, pair[1][0]))
            except Exception:
                pass
    return


def get_time():
    return datetime.datetime.now().strftime("%H:%M:%S")


def chunkify(lst, n):
    return [lst[i::n] for i in range(n)]


def main(path):
    starttime = get_time()
    qout = queue.Queue()
    images = []
    for img in os.listdir(path):
        if os.path.isfile(os.path.join(path, img)):
            images.append(img)
    imglen = len(images)
    print('Resizing ' + str(imglen) + ' Images ' + starttime)
    images = chunkify(images, 4)
    threads = []
    for x in range(4):
        threads.append(threading.Thread(target=make_histogram, args=(images[x], path, qout)))

    [x.start() for x in threads]
    [x.join() for x in threads]

    resizetime = get_time()
    print('Done resizing ' + resizetime)

    histlist = []
    for i in qout.queue:
        histlist.append(i)

    if not os.path.exists(os.path.join(path, 'exact duplicates')):
        os.makedirs(os.path.join(path, 'exact duplicates'))
    if not os.path.exists(os.path.join(path, 'maybe duplicates')):
        os.makedirs(os.path.join(path, 'maybe duplicates'))

    combinations = []
    for img1, img2 in itertools.combinations(histlist, 2):
        combinations.append([img1, img2])

    combicount = len(combinations)
    print('Going through ' + str(combicount) + ' combinations of ' + str(imglen) + ' Images. Please stand by')
    combinations = chunkify(combinations, 4)

    threads = []

    for x in range(4):
        threads.append(threading.Thread(target=compare_hist, args=(combinations[x], path)))

    [x.start() for x in threads]
    [x.join() for x in threads]

    print('\nstarted at ' + starttime)
    print('resizing done at ' + resizetime)
    print('went through ' + str(combicount) + ' combinations of ' + str(imglen) + ' Images')
    print('all done at ' + get_time())

if __name__ == '__main__':
    main(sys.argv[1]) # sys.argv[1] has to be a folder of images to compare

This works but the comparison runs for hours after completing the resizes within 15 to 20 minutes. At first I assumed that it was a locking queue from which the workers got their combinations so I replaced it with pre-defined array chunks. This did not reduce the execution time. I also ran it without moving the files to exclude a possible hard drive issue.

Profiling this using cProfile provides the following output.

Resizing 4566 Images 23:51:05
Done resizing 00:05:07
Going through 10421895 combinations of 4566 Images. Please stand by

started at 23:51:05
resizing done at 00:05:07
went through 10421895 combinations of 4566 Images
all done at 03:09:41
         10584539 function calls (10584414 primitive calls) in 11918.945 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     16/1    0.001    0.000 11918.945 11918.945 {built-in method exec}
        1    2.962    2.962 11918.945 11918.945 imcomp.py:3(<module>)
        1   19.530   19.530 11915.876 11915.876 imcomp.py:60(main)
       51 11892.690  233.190 11892.690  233.190 {method 'acquire' of '_thread.lock' objects}
        8    0.000    0.000 11892.507 1486.563 threading.py:1028(join)
        8    0.000    0.000 11892.507 1486.563 threading.py:1066(_wait_for_tstate_lock)
        1    0.000    0.000 11051.467 11051.467 imcomp.py:105(<listcomp>)
        1    0.000    0.000  841.040  841.040 imcomp.py:76(<listcomp>)
 10431210    1.808    0.000    1.808    0.000 {method 'append' of 'list' objects}
     4667    1.382    0.000    1.382    0.000 {built-in method stat}

The full profiler output can be found here .

Considering the fourth line I'm guessing that the threads are somehow locking. But why and why exactly 51 times regardless of the amount of images?

I am running this on Windows 7 64 bit.

Thanks in advance.

Answer 1

One major issue is that you're using threads to do work that is at least partially CPU-bound. Because of the Global Interpreter Lock, only one CPython thread can ever run at a time, which means you can't take advantage of multiple CPU cores. This will make multi-threaded performance for CPU-bound tasks at best no different from single-core execution, and probably even worse, because of the extra overhead added by threading. This is noted in the threading documentation :

CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing . However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

To get around the limitations of the GIL, you should do as the docs say, and use the multiprocessing library instead of the threading library:

import multiprocessing
...

qout = multiprocessing.Queue()

for x in range(4):
    threads.append(multiprocessing.Process(target=make_histogram, args=(images[x], path, qout)))

...
for x in range(4):
    threads.append(multiprocessing.Process(target=compare_hist, args=(combinations[x], path)))

As you can see, multiprocessing for the most part is a drop-in replacement for threading , so the changes shouldn't be too difficult to make. The only complication would be if any of the arguments you're passing between processes aren't picklable, though I think all of them are in your case. There is also an increased cost of IPC to send Python data structures between processes, but I suspect the benefit of truly parallel computations will outweigh that additional overhead.

All that said, you may be still somewhat I/O bound here, because of reliance on reads/writes to disk. Parallelizing won't make your disk I/O faster, so there's not much that can be done there.

Answer 2

With 4500 images to compare, I would suggest multiprocessing on a file-level, not (necessarily) multithreading within the image. As @dano has pointed out, the GIL will get in the way for that. My strategy would be:

1. one worker process per core (or configured number);
2. one orchestration process, which forks off the above; does some IPC to coordinate jobs to workers.

Looking (briefly) at your code looks like it would benefit from a lazy language; I don't see that makes any attempt to short-circuit comparisons. For example, if you do the RMS comparison for each segment of an image, you can stop comparing once you end comparing chunks once you determine they are sufficiently different. You might then also care to change the way you iterate through the chunks, and the size/shape of the chunks.

Apart from that, I would consider looking at cheaper mechanisms that avoid doing some many square roots; possibly using something that creates an 'approximate' square-root, perhaps using a look-up table.

If I'm not mistaken, you could also create an intermediate form (the histogram) that you should keep temporarily. No need to save the 800x600 image.

Also, it would be useful to know what you mean be 'equal' with regard to this exercise.