I'm trying to run 2 functions over two differents process with the help of multiprocessing.Pool().apply_async() , but it doesn't seem to run my function. No error messages tho. But when I try my function without multiprocessing then it works perfectly. Here is how my code looks like(short version) :
import multiprocessing, twitch_integration
p = multiprocessing.Pool()
p.apply_async(twitch_integration.get_user_followers, args=(userid1, "", conn,))
p.apply_async(twitch_integration.get_user_followers, args=(userid2, "", conn,))
p.close()
p.join()
I don't know if it's important to note, but get_user_followers function is recursive and requests module. I've put a print at the very beginning of get_user_followers, but it doesn't print anything.
I've been searching for the last 4 hours, I'm not exaggerating. If anyone could help me I would be very thankful. Thank you.
After asking for the results as @Steve mentioned, it raised a new error and here is the traceback :
Traceback (most recent call last):
File "main.py", line 89, in <module>
main(conn, cursor);
File "main.py", line 68, in main
r = result.get(timeout=1)
File "C:\Python38\lib\multiprocessing\pool.py", line 771, in get
raise self._value
File "C:\Python38\lib\multiprocessing\pool.py", line 537, in _handle_tasks
put(task)
File "C:\Python38\lib\multiprocessing\connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "C:\Python38\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "C:\Python38\lib\socket.py", line 272, in __getstate__
raise TypeError(f"cannot pickle {self.__class__.__name__!r} object")
TypeError: cannot pickle 'SSLSocket' object
Here is the get_user_followers function for more details(I know it's ugly but I'm getting started with Python and programmation):
import requests, json, sys, time
from mysql.connector import Error
def get_user_followers(user_id, pagination, conn):
global FOLLOWER_COUNT
global ERROR_COUNT
global OAUTH_TOKEN
global TOTAL_FOLLOWER
global ERROR_IN_A_ROW
print("YES")
after = "&after={0}".format(pagination)
first = "&first=100"
query = 'users/follows?to_id={0}{1}{2}'.format(user_id, first, pagination if pagination == "" else after)
try:
response = get_response(query)
except requests.exceptions.RequestException as e:
print("pause 1 seconde. . . Retrying same request")
time.sleep(1)
print("Request error: ", e)
get_user_followers(user_id, pagination, conn)
return
finally:
pass
if response.status_code == 200:
ERROR_IN_A_ROW = 0
elif response.status_code == 401:
print(response.json())
print("HTTP Error 401")
ERROR_IN_A_ROW += 1
if ERROR_IN_A_ROW == 3:
print("AFTER 3 HTTP ERROR IN A ROW - EXITING PROGRAM")
print("pagination: ", pagination, " user_id", user_id)
return
OAUTH_TOKEN = requests.post(POST_URL, data=POST_PARAMS).json()['access_token']
get_user_followers(user_id, pagination, conn)
return
elif response.status_code == 429:
print(response.json())
print("HTTP Error 429")
time.sleep(int(response.json()['Ratelimti-Reset']))
get_user_followers(user_id, pagination, conn)
return
else:
print(response.json())
print("HTTP Error {0}".format(response.status_code))
ERROR_IN_A_ROW += 1
if ERROR_IN_A_ROW == 3:
print("AFTER 3 HTTP ERROR IN A ROW - EXITING PROGRAM")
print("pagination: ", pagination, " user_id", user_id)
return
get_user_followers(user_id, pagination, conn)
return
response = response.json()
TOTAL_FOLLOWER = response['total']
length = len(response['data'])
if length == 0:
try:
print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER) + " ({0}%)".format(format((FOLLOWER_COUNT * 100 / TOTAL_FOLLOWER), ".2f")) + "\r", end="")
except:
print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER))
FOLLOWER_COUNT = 0
ERROR_COUNT = 0
TOTAL_FOLLOWER = 0
return 1
pagination = response['pagination']['cursor']
i = 0
while i < length:
try:
conn.cursor().execute("""INSERT INTO user (id, username) VALUES ({0}, '{1}')""".format(response['data'][i]['from_id'], str(response['data'][i]['from_name'])))
except Error as e:
print("An error has occured : ", e)
ERROR_COUNT += 1
finally:
pass
i += 1
FOLLOWER_COUNT += 1
try:
print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER) + " ({0}%)".format(format((FOLLOWER_COUNT * 100 / TOTAL_FOLLOWER), ".2f")) + " {0}".format(user_id))
except:
print("{0}/{1}".format(FOLLOWER_COUNT, TOTAL_FOLLOWER))
get_user_followers(user_id, pagination, conn)
To get your jobs/functions to run, you have to ask for their results. This should run your jobs:
import multiprocessing, twitch_integration
p = multiprocessing.Pool()
result = p.apply_async(twitch_integration.get_user_followers, args=(userid1, "", conn,))
r = result.get(timeout=1)
result = p.apply_async(twitch_integration.get_user_followers, args=(userid2, "", conn,))
r = result.get(timeout=1)
p.close()
p.join()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.