Python中的多处理多层图像合并

Question

I've spent a few hours searching stack and google for an answer to this and seem to get close but never quite get the answer. I feel like it's something simple in create_files but I'm just not getting it.

I'm trying to increase the speed of a program that gets a random image from numbered folders, and stacks them on top of each other with transparency using Pillow. Now as soon as you hit 18 folders, with 20 or so images in each, this can be quite slow at around 2 seconds per image on my M1 MacBook Pro. I thought of using multiprocessing to speed things up, and I don't know if it's because my code is so reliant on for loop iterator variables, but I couldn't seem to get pool.map to work for me without differing errors. I'll post a snippet of the relevant functions. Any advice would be amazing, I'm sure it's a simple refactor, might still just be a bit too green with Python.

The variables that some of the functions call are either the path to a folder, or an array of all of the image paths (image1 will be [0] image2 will be [1] etc).

def save_file(img, counter):
    if not os.path.exists(f"{output_folder}"):
        os.makedirs(f"{output_folder}")
    img.save(f"{output_folder}/file_{counter}.png")


def create_composite(layer_count, img):
    image_size = Image.open(image_paths[0][0])
    composite_img = Image.new("RGBA", image_size.size)
    composite_img = merge_images(layer_count, img, composite_img)
    return composite_img


def merge_images(layer_count, img, composite_img):
    for i in range(layer_count):
        next_img = Image.open(img[i])
        composite_img = Image.alpha_composite(composite_img, next_img)
    return composite_img


def randomise_layers(layer_count):
    # Creates an array with randomised files for each layer and checks for exceptions.
    image_stack = []
    for i in range(layer_count):
        # Checks if there are any image conflicts and adds them to the stack.
        image_stack.append(check_exception(i))
    return image_stack


def check_exception(index):
    # Checks if any of the layers have a known exception with another layer
    temp_file_path = file_paths(index)
    if is_exception(temp_file_path):
        check_exception(index)
    else:
        return temp_file_path


def is_exception(file_path):
    # Gets layer exceptions from file
    return False


def create_files(iterations):
    for i in range(iterations):
        save_file(create_composite(layer_count, randomise_layers(layer_count)), i)
        if iterations < 10:
            print(f"File {i+1} created.")

if __name__ == "__main__":
    iterations = 5
    create_files(iterations)

Answer 1

OK! So thanks to Albert's suggestion, I looked more into ProcessPoolExecutor, and did some refactoring.

my create_files function is now as follows:

def create_files(iterations):
    with concurrent.futures.ProcessPoolExecutor(os.cpu_count()-2) as exe:
        exe.map(save_file, list(range(iterations)))

Due to this refactoring and using map, I've had to move a bunch of variables to be global variables as I couldn't pass them into the functions onces the multiprocessing kicked in (seems to not hold onto any that have been declared in the same scope before). I'll look more into passing some of those values into the threads, another alternative would be to make lists of those variables, and do something like

var1 = [missing_variable] * iterations

to make them available.