流式多处理进程 output

Question

I have a function that generates output before returning the result. While that function is running, I want to stream the intermediate output to a frontend, so a user is aware what's happening while waiting for the result. I'm trying to accomplish this with Multiprocessing, but intermediate output and the result is printed at once at the end. I can't find a comparable problem anywhere, so advice is much appreciated. This is my undesirably behaving code so far:

from multiprocessing import Process, Pipe
from sklearn.manifold import TSNE
import random
import io
import sys

def TSNE_transform():
    X = [[random.randint(1,9) for i in range(9)] for j in range(10)]
    model = TSNE(n_components=3, verbose=3)
    result = model.fit_transform(X)
    return result

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=TSNE_transform)
    p.start()
    while True:
        print(parent_conn.recv())
        if parent_conn.recv() == None:
            break
    p.join()

Function TSNE_transform is generating intermediate output.

I also tried to redirect the stdout like this, but I'm not sure how that could be streamed to a frontend:

old_stdout = sys.stdout

# Assign buffer to new_stdout
new_stdout = io.StringIO()

# Assign stdout to buffer
sys.stdout = new_stdout

X = [[random.randint(1,30) for i in range(30)] for j in range(100)]
model = TSNE(n_components=3, verbose=3)
result = model.fit_transform(X)

# Get values
output = new_stdout.getvalue()

# Revert to original stdout
sys.stdout = old_stdout

print(output)

Any help is welcome:)

Answer 1

You should use a Queue in order to communicate with the Process.
The simple example below is taken from the docs .

import multiprocessing as mp

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    ctx = mp.get_context('spawn')
    q = ctx.Queue()
    p = ctx.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()