python : multithreaded unix socket transfers are very slow
Question
I am trying to create a way for communication between multiple processes to transfer data using UNIX domain sockets. There are 2 programs (progserv.py and progclient.py) each with N (this example has 4) threads, each one communicating to the corresponding one on the other process. The progclient.py sends 'pickle'd form of datetime.datetime object. The progserv.py receives the data and determines the delay in communication. The idea here is to find the delay in communication over the inter-process data transfer.
The problem I am faced is that with increasing values of N, there is an increase in delay from less than 1msec to sometimes more than 250msec. The delay time is not consistent, it keeps fluctuating pretty huge.
Code of progserv.py:
import string
import random
import threading
import time
import os
import traceback
import socket
import pickle
from datetime import datetime
def worker(num):
addr = './tmp'+str(num)
logstr = "SERVER"+str(num)+":"
if os.path.exists(addr):
os.unlink(addr)
try:
msock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
msock.bind(addr)
msock.listen(1)
sock,_ = msock.accept()
sock.setblocking(0)
msock.close()
except:
print (logstr,"Error opening the socket for listening.Trace:", traceback.format_exc())
return
print (logstr,"Socket connection with client done")
buf=list()
count=1
while True:
try:
data = sock.recv(1500)
if len(data)<=0:
print (logstr,"Peer end probably closed down the connection")
break
else:
try:
d=(datetime.now()-pickle.loads(data)).total_seconds()*1000
buf.append((count,d))
count+=1
#print (logstr,"Received ",str(count-1),"packets")
except:
pass
except socket.error:
if len(buf)>0:
c,data=buf.pop(0)
print (logstr,"COUNTER=",c," DELTA=",data,"msec")
except:
print (logstr,"Exception in receiving data. Trace:", traceback.format_exc())
break
if isinstance(sock, socket.socket):
sock.close()
if isinstance(msock, socket.socket):
msock.close()
def main():
server_count=5
threads=list()
for i in range(server_count):
t=threading.Thread(target=worker, args=(i,))
threads.append(t)
t.start()
print ("MAIN: Started thread index",i)
time.sleep(0.1)
for t in threads:
t.join()
if __name__=='__main__':
main()
Code of progclient.py:
import traceback, os, time, random, socket
import threading, pickle
from datetime import datetime
def worker(num):
addr = './tmp'+str(num)
logstr="CLIENT"+str(num)+":"
try:
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
sock.connect(addr)
sock.setblocking(0)
except:
print (logstr,"Error opening the socket for connection. Trace:", traceback.format_exc())
return
while True:
try:
d=datetime.now()
bytes=sock.send(pickle.dumps(d))
if bytes<=0:
print (logstr,"Peer end probably closed down the connection")
break
else:
print (logstr,"Sent",bytes,"bytes")
except socket.error:
pass
except:
print (logstr,"Exception during sending. Trace:", traceback.format_exc())
break
time.sleep(1)
if isinstance(sock, socket.socket):
sock.close()
def main():
client_count=5
threads=list()
for i in range(client_count):
t=threading.Thread(target=worker, args=(i,))
threads.append(t)
t.start()
print ("CLIENT : Started thread index",i)
time.sleep(0.1)
for t in threads:
t.join()
if __name__=='__main__':
main()
A run of the delay times with 4 threads on each process:
MAIN: Started thread index 0
MAIN: Started thread index 1
MAIN: Started thread index 2
MAIN: Started thread index 3
MAIN: Started thread index 4
SERVER0: Socket connection with client done
SERVER0: COUNTER= 1 DELTA= 0.7969999999999999 msec
SERVER1: Socket connection with client done
SERVER1: COUNTER= 1 DELTA= 0.28200000000000003 msec
SERVER2: Socket connection with client done
SERVER3: Socket connection with client done
SERVER2: COUNTER= 1 DELTA= 258.839 msec
SERVER4: Socket connection with client done
SERVER3: COUNTER= 1 DELTA= 238.006 msec
SERVER4: COUNTER= 1 DELTA= 477.099 msec
SERVER0: COUNTER= 2 DELTA= 0.175 msec
SERVER2: COUNTER= 2 DELTA= 0.174 msec
SERVER1: COUNTER= 2 DELTA= 108.28 msec
SERVER4: COUNTER= 2 DELTA= 35.966 msec
SERVER3: COUNTER= 2 DELTA= 146.63899999999998 msec
SERVER1: COUNTER= 3 DELTA= 76.92 msec
SERVER2: COUNTER= 3 DELTA= 116.096 msec
SERVER0: COUNTER= 3 DELTA= 317.726 msec
SERVER3: COUNTER= 3 DELTA= 15.792 msec
SERVER4: COUNTER= 3 DELTA= 294.577 msec
SERVER0: COUNTER= 4 DELTA= 6.123 msec
SERVER1: COUNTER= 4 DELTA= 95.69999999999999 msec
SERVER2: COUNTER= 4 DELTA= 0.434 msec
SERVER4: COUNTER= 4 DELTA= 63.32599999999999 msec
SERVER3: COUNTER= 4 DELTA= 322.351 msec
SERVER0: COUNTER= 5 DELTA= 185.03300000000002 msec
SERVER1: COUNTER= 5 DELTA= 84.869 msec
SERVER2: COUNTER= 5 DELTA= 33.532 msec
SERVER3: COUNTER= 5 DELTA= 101.01400000000001 msec
SERVER4: COUNTER= 5 DELTA= 152.035 msec
SERVER0: COUNTER= 6 DELTA= 33.533 msec
SERVER2: COUNTER= 6 DELTA= 0.501 msec
SERVER1: COUNTER= 6 DELTA= 101.59400000000001 msec
SERVER3: COUNTER= 6 DELTA= 91.584 msec
SERVER4: COUNTER= 6 DELTA= 32.779999999999994 msec
SERVER1: COUNTER= 7 DELTA= 42.009 msec
SERVER0: COUNTER= 7 DELTA= 142.716 msec
SERVER2: COUNTER= 7 DELTA= 137.214 msec
SERVER3: COUNTER= 7 DELTA= 124.629 msec
SERVER4: COUNTER= 7 DELTA= 190.025 msec
SERVER0: COUNTER= 8 DELTA= 0.9859999999999999 msec
SERVER1: COUNTER= 8 DELTA= 160.696 msec
SERVER2: COUNTER= 8 DELTA= 60.282000000000004 msec
SERVER4: COUNTER= 8 DELTA= 0.617 msec
SERVER3: COUNTER= 8 DELTA= 169.09 msec
SERVER0: COUNTER= 9 DELTA= 129.679 msec
SERVER1: COUNTER= 9 DELTA= 107.504 msec
SERVER2: COUNTER= 9 DELTA= 8.319999999999999 msec
SERVER3: COUNTER= 9 DELTA= 97.829 msec
SERVER4: COUNTER= 9 DELTA= 262.472 msec
SERVER0: COUNTER= 10 DELTA= 188.364 msec
SERVER2: COUNTER= 10 DELTA= 101.07 msec
SERVER3: COUNTER= 10 DELTA= 116.54100000000001 msec
SERVER1: COUNTER= 10 DELTA= 318.44899999999996 msec
SERVER4: COUNTER= 10 DELTA= 16.676 msec
SERVER0: COUNTER= 11 DELTA= 7.071 msec
SERVER1: COUNTER= 11 DELTA= 60.224 msec
SERVER2: COUNTER= 11 DELTA= 101.07199999999999 msec
SERVER3: COUNTER= 11 DELTA= 45.32 msec
SERVER4: COUNTER= 11 DELTA= 11.719999999999999 msec
SERVER0: COUNTER= 12 DELTA= 125.88199999999999 msec
SERVER1: COUNTER= 12 DELTA= 75.607 msec
SERVER2: COUNTER= 12 DELTA= 84.648 msec
SERVER3: COUNTER= 12 DELTA= 44.051 msec
SERVER4: COUNTER= 12 DELTA= 290.6 msec
SERVER0: COUNTER= 13 DELTA= 0.306 msec
SERVER1: COUNTER= 13 DELTA= 58.865 msec
SERVER2: COUNTER= 13 DELTA= 101.007 msec
SERVER3: COUNTER= 13 DELTA= 12.927 msec
SERVER4: COUNTER= 13 DELTA= 212.348 msec
SERVER0: COUNTER= 14 DELTA= 53.222 msec
SERVER1: COUNTER= 14 DELTA= 63.087 msec
SERVER2: COUNTER= 14 DELTA= 112.285 msec
SERVER4: COUNTER= 14 DELTA= 0.192 msec
SERVER3: COUNTER= 14 DELTA= 111.608 msec
SERVER0: COUNTER= 15 DELTA= 41.897999999999996 msec
SERVER2: COUNTER= 15 DELTA= 0.514 msec
SERVER3: COUNTER= 15 DELTA= 30.408 msec
SERVER1: COUNTER= 15 DELTA= 261.84499999999997 msec
SERVER4: COUNTER= 15 DELTA= 69.785 msec
SERVER0: COUNTER= 16 DELTA= 88.022 msec
SERVER2: COUNTER= 16 DELTA= 19.599999999999998 msec
SERVER1: COUNTER= 16 DELTA= 201.69500000000002 msec
SERVER3: COUNTER= 16 DELTA= 52.805 msec
SERVER4: COUNTER= 16 DELTA= 8.53 msec
SERVER0: COUNTER= 17 DELTA= 0.324 msec
SERVER1: COUNTER= 17 DELTA= 159.975 msec
SERVER2: COUNTER= 17 DELTA= 58.62 msec
SERVER4: COUNTER= 17 DELTA= 0.462 msec
SERVER3: COUNTER= 17 DELTA= 100.717 msec
SERVER0: COUNTER= 18 DELTA= 68.68100000000001 msec
SERVER1: COUNTER= 18 DELTA= 8.701 msec
SERVER2: COUNTER= 18 DELTA= 16.889999999999997 msec
SERVER3: COUNTER= 18 DELTA= 156.637 msec
SERVER4: COUNTER= 18 DELTA= 56.296 msec
SERVER0: COUNTER= 19 DELTA= 29.073 msec
SERVER2: COUNTER= 19 DELTA= 30.326999999999998 msec
SERVER3: COUNTER= 19 DELTA= 0.155 msec
SERVER1: COUNTER= 19 DELTA= 251.63200000000003 msec
SERVER4: COUNTER= 19 DELTA= 5.305 msec
when attempting the same transfers using just 1 thread, then I see delays of approx 1msec. Its erratically increases with the number of threads used on each process.
The platform is a Jessie-based Raspberry Pi. Could someone please explain this behavior ?
1 answers
solution1
0 2017-09-19 22:48:54
this is due to Python's Global Interpreter Lock (GIL) and can only limit the concurrent processing speed. source : https://jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/
To have a faster chance or completing a task you will have to either make another process(not thread) or just move to a language like c++ which enables concurrent threading and processing without no "lock" on the thread limit.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
websockets transfers in raspberry is very slow
Python recursion is very slow
Python 3 Tkinter Treeview Slow Performance while Multithreaded
Python speech recognition very slow
python 3 https posting very slow
Python - Tkinter - Scrollbar Very Slow
random.shuffle very slow in Python 3 with list
Improve execution time of very slow python code
Python 3 in Yocto very slow on Raspberry Pi
Very slow http post requests in python
Related Tags