[英]Very Slow Binary File Download in Multithreaded Python Program on Raspberry Pi
I'm writing a multi-threaded Python application on a Raspberry Pi 4 that occasionally needs to download binary files on the order of ~200 kilobytes from a server, which during testing is my laptop on the local network.我正在 Raspberry Pi 4 上编写一个多线程 Python 应用程序,该应用程序偶尔需要从服务器下载大约 200 KB 的二进制文件,在测试期间,该服务器是我在本地网络上的笔记本电脑。 I've verified that these files are served by my laptop in about a second using Curl or a Python 3 CLI requests.get call on the RPi, but inside my application the requests download call hangs for at least 2 minutes before completing.
我已经使用 Curl 或 Python 验证了这些文件由我的笔记本电脑在大约一秒钟内提供服务 3 CLI requests.get 调用 RPi,但在我的应用程序中,请求下载调用在完成前至少挂起 2 分钟。 The affected code is here:
受影响的代码在这里:
# requests current composite from server
# args: timestamp: timestamp of last update
# returns: SUCCESS_RETURN if updated, NONE_RETURN otherwise, OFFLINE_RETURN on failure to connect
def getcomposite(self, timestamp=None):
try:
self.slplogger.info("Downloading composite for timestamp %s" % (dt.utcfromtimestamp(timestamp).strftime("%Y-%m-%d-%H:%M:%S") if timestamp else "None"))
# HANGS HERE
compositeresp = requests.post(SERVER_URL + "getcomposite", data={'mac' : self.mac, 'timestamp' : timestamp})
self.slplogger.info("Downloaded new composite: %s" % str(compositeresp.text[:min(10, len(compositeresp.text))]))
if compositeresp.text != NONE_RETURN and compositeresp.content:
with self.compositelock:
self.compositedata = np.load(BytesIO(compositeresp.content), allow_pickle=False)
# compute new input norm for adding subsequent input
self.compositenorm = np.mean(self.compositedata[:]['value'], dtype=int)
self.emptycomposite = False
self.slplogger.info("Set new composite: %s" % str(self.compositedata[:min(10, len(self.compositedata))]))
return SUCCESS_RETURN
return FAILURE_RETURN
except requests.exceptions.ConnectionError:
self.slplogger.info("Composite download failed. Unable to connect to server")
return OFFLINE_RETURN
The non-daemon thread that calls this method is defined here (COMPOSITE_POLL_INTERVAL is 2 seconds):这里定义了调用该方法的非守护线程(COMPOSITE_POLL_INTERVAL 为 2 秒):
# --------------------------------------------------------------------
# CompositePollingThread - Thread superclass that periodically
# polls strangeloop server for new additions to the composite loop
# --------------------------------------------------------------------
class CompositePollingThread(Thread):
# overloaded Thread constructor
# args: pedal: parent Pedal object that instantiated this thread
def __init__(self, pedal):
Thread.__init__(self)
self.stop = Event()
self.pedal = pedal
self.timestamp = None
self.pedal.slplogger.debug("Initialized composite polling thread")
# main thread execution loop
def run(self):
self.pedal.slplogger.debug("Started composite polling thread")
while self.pedal.running:
time.sleep(COMPOSITE_POLL_INTERVAL)
# timestamp to determine whether any new data needs to be downloaded
if not self.pedal.recording and self.pedal.getcomposite(timestamp=self.timestamp) == SUCCESS_RETURN:
self.timestamp = dt.utcnow().timestamp()
self.pedal.slplogger.debug("Downloaded new composite at %s" % dt.utcfromtimestamp(self.timestamp).strftime("%Y-%m-%d-%H:%M:%S"))
self.pedal.slplogger.debug("Ended composite polling thread")
I'm assuming this slow download is caused by a threading issue in the program.我假设这个缓慢的下载是由程序中的线程问题引起的。 It's also tasked with processing real-time input, which takes up the majority of the CPU.
它还负责处理占用大部分 CPU 的实时输入。 Is there anything I can do to improve this download speed?
我能做些什么来提高这个下载速度吗? Is there a way to give the thread more priority, or should I switch to the multiprocessing module to take advangage of the RPi 4's multiple cores?
有没有办法给线程更高的优先级,或者我应该切换到多处理模块以利用 RPi 4 的多核?
In the end, the issue was on the server.最后,问题出在服务器上。 I changed the endpoint response from a "flask.send_file" with a BytesIO wrapper around the data I needed to a simple flask.Response object with the data as the text (similar to Serve image stored in SQLAlchemy LargeBinary column ), and now it downloads in less than ten seconds.
I changed the endpoint response from a "flask.send_file" with a BytesIO wrapper around the data I needed to a simple flask.Response object with the data as the text (similar to Serve image stored in SQLAlchemy LargeBinary column ), and now it downloads不到十秒钟。 I don't know why the download took so long before only when accessed through the program (as opposed to CLI requests), but that change took care of it.
我不知道为什么只有在通过程序访问(而不是 CLI 请求)时才需要这么长时间才能下载,但是这种更改解决了它。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.