如何将多处理应用于滑动 window

Question

I am creating 1 million values and splitting them into train and test by using a sliding window that uses a window of length 1000 values and slides by one value every time.

For example, The first process would be the splitting of first 1000 values to the train, and 1001st value to the test. The second process would be valued from 2 to 1001 in the train and 1002nd in the test, and so on.

It takes 76.28 seconds to run the script. I used timeit to measure this.

Now, I want to reduce this time by running a sliding window using multiple processors. I used Pool from multiprocessing with 4 CPUs but, it didn't change the performance at all. I am wondering what could be a better way here?

code:

from multiprocessing import Process
from multiprocessing import Pool
import numpy as np
import pandas as pd
from timeit import default_timer as timer

start = timer()

data = list(range(1_000_000))
window_size = 1_000
splits = []

def sw(window_size, data):
    for i in range(window_size, len(data)):
        train = np.array(data[i - window_size:i])
        test = np.array(data[i:i + 1])
        splits.append(('TRAIN:', train, 'TEST:', test))

#  sw(window_size, data)
#  print(splits)

if __name__ == '__main__':
    p= Pool(4)
    p = Process(target=sw, args=(window_size, data))
    p.start()
    p.join()

end = timer()
print(end - start)

Answer 1

Indeed as the comments point out, all you do is create a Pool named p , then reassign that variable to a process result. I rewrote your sliding window function a little. A simple way to parallelize independent tasks is to specify what you want to do to one item and then just use the map functor. Benchmarks were performed on an Intel core i5-6300U@2.40GHz (dual core with hyperthreading).

from multiprocessing import Process
from multiprocessing import Pool
import numpy as np
from timeit import default_timer as timer

NUM_EL = 1_000_000
WINDOW_SIZE = 1000
DATA = list(range(NUM_EL))


def window(start_idx, window_size=WINDOW_SIZE, data=DATA):
    _train = np.array(data[start_idx:start_idx + window_size])
    _test = np.array(data[start_idx + window_size + 1])
    # return something useful here
    return start_idx


if __name__ == '__main__':
    STARTS = list(range(NUM_EL - WINDOW_SIZE - 1))

    start = timer()
    result_single = list(map(window, STARTS))
    end = timer()
    print("Single core: ", end - start)

    start = timer()
    with Pool(4) as p:
       result_multi = p.map(window, STARTS)

    end = timer()
    print(result_single == result_multi)
    print("Multiprocessing: ", end - start)

>>> Single core:  99.9821742
>>> Multiprocessing:  38.71327739999998

Note: This code most likely does NOT work in any environment using IPython .