简体   繁体   中英

multiprocessing.Pool().apply_async() doesn't seem to run my function

I'm trying to run 2 functions over two differents process with the help of multiprocessing.Pool().apply_async() , but it doesn't seem to run my function. No error messages tho. But when I try my function without multiprocessing then it works perfectly. Here is how my code looks like(short version) :

import multiprocessing, twitch_integration
p = multiprocessing.Pool()
p.apply_async(twitch_integration.get_user_followers, args=(userid1, "", conn,))
p.apply_async(twitch_integration.get_user_followers, args=(userid2, "", conn,))
p.close()
p.join()

I don't know if it's important to note, but get_user_followers function is recursive and requests module. I've put a print at the very beginning of get_user_followers, but it doesn't print anything.

I've been searching for the last 4 hours, I'm not exaggerating. If anyone could help me I would be very thankful. Thank you.

After asking for the results as @Steve mentioned, it raised a new error and here is the traceback :

Traceback (most recent call last):
  File "main.py", line 89, in <module>
    main(conn, cursor);
  File "main.py", line 68, in main
    r = result.get(timeout=1)
  File "C:\Python38\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
  File "C:\Python38\lib\multiprocessing\pool.py", line 537, in _handle_tasks
    put(task)
  File "C:\Python38\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Python38\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "C:\Python38\lib\socket.py", line 272, in __getstate__
    raise TypeError(f"cannot pickle {self.__class__.__name__!r} object")
TypeError: cannot pickle 'SSLSocket' object

Here is the get_user_followers function for more details(I know it's ugly but I'm getting started with Python and programmation):

import requests, json, sys, time
from mysql.connector import Error
def get_user_followers(user_id, pagination, conn):
    global FOLLOWER_COUNT
    global ERROR_COUNT
    global OAUTH_TOKEN
    global TOTAL_FOLLOWER
    global ERROR_IN_A_ROW
    print("YES")
    after = "&after={0}".format(pagination)
    first = "&first=100"
    query = 'users/follows?to_id={0}{1}{2}'.format(user_id, first, pagination if pagination == "" else after)
    try:
        response = get_response(query)
    except requests.exceptions.RequestException as e:
        print("pause 1 seconde. . . Retrying same request")
        time.sleep(1)
        print("Request error: ", e)
        get_user_followers(user_id, pagination, conn)
        return
    finally:
        pass
    if response.status_code == 200:
        ERROR_IN_A_ROW = 0
    elif response.status_code == 401:
        print(response.json())
        print("HTTP Error 401")
        ERROR_IN_A_ROW += 1
        if ERROR_IN_A_ROW == 3:
            print("AFTER 3 HTTP ERROR IN A ROW - EXITING PROGRAM")
            print("pagination: ", pagination, " user_id", user_id)
            return
        OAUTH_TOKEN = requests.post(POST_URL, data=POST_PARAMS).json()['access_token']
        get_user_followers(user_id, pagination, conn)
        return
    elif response.status_code == 429:
        print(response.json())
        print("HTTP Error 429")
        time.sleep(int(response.json()['Ratelimti-Reset']))
        get_user_followers(user_id, pagination, conn)
        return
    else:
        print(response.json())
        print("HTTP Error {0}".format(response.status_code))
        ERROR_IN_A_ROW += 1
        if ERROR_IN_A_ROW == 3:
            print("AFTER 3 HTTP ERROR IN A ROW - EXITING PROGRAM")
            print("pagination: ", pagination, " user_id", user_id)
            return
        get_user_followers(user_id, pagination, conn)
        return
    response = response.json()
    TOTAL_FOLLOWER = response['total']
    length = len(response['data'])
    if length == 0:
        try:
            print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER) + " ({0}%)".format(format((FOLLOWER_COUNT * 100 / TOTAL_FOLLOWER), ".2f")) + "\r", end="")
        except:
            print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER))
        FOLLOWER_COUNT = 0
        ERROR_COUNT = 0
        TOTAL_FOLLOWER = 0
        return 1
    pagination = response['pagination']['cursor']
    i = 0
    while i < length:
        try:
            conn.cursor().execute("""INSERT INTO user (id, username) VALUES ({0}, '{1}')""".format(response['data'][i]['from_id'], str(response['data'][i]['from_name'])))
        except Error as e:
            print("An error has occured : ", e)
            ERROR_COUNT += 1
        finally:
            pass
        i += 1
        FOLLOWER_COUNT += 1
        try:
            print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER) + " ({0}%)".format(format((FOLLOWER_COUNT * 100 / TOTAL_FOLLOWER), ".2f")) + " {0}".format(user_id))
        except:
            print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER))
    get_user_followers(user_id, pagination, conn)

To get your jobs/functions to run, you have to ask for their results. This should run your jobs:

import multiprocessing, twitch_integration
p = multiprocessing.Pool()
result = p.apply_async(twitch_integration.get_user_followers, args=(userid1, "", conn,))
r = result.get(timeout=1)
result = p.apply_async(twitch_integration.get_user_followers, args=(userid2, "", conn,))
r = result.get(timeout=1)
p.close()
p.join()

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM