Very Slow Binary File Download in Multithreaded Python Program on Raspberry Pi

Question

I'm writing a multi-threaded Python application on a Raspberry Pi 4 that occasionally needs to download binary files on the order of ~200 kilobytes from a server, which during testing is my laptop on the local network. I've verified that these files are served by my laptop in about a second using Curl or a Python 3 CLI requests.get call on the RPi, but inside my application the requests download call hangs for at least 2 minutes before completing. The affected code is here:

# requests current composite from server 
# args:     timestamp: timestamp of last update
# returns:  SUCCESS_RETURN if updated, NONE_RETURN otherwise, OFFLINE_RETURN on failure to connect

def getcomposite(self, timestamp=None):
    try:
        self.slplogger.info("Downloading composite for timestamp %s" % (dt.utcfromtimestamp(timestamp).strftime("%Y-%m-%d-%H:%M:%S") if timestamp else "None"))

        # HANGS HERE
        compositeresp = requests.post(SERVER_URL + "getcomposite", data={'mac' : self.mac, 'timestamp' : timestamp})

        self.slplogger.info("Downloaded new composite: %s" % str(compositeresp.text[:min(10, len(compositeresp.text))]))

        if compositeresp.text != NONE_RETURN and compositeresp.content:
            with self.compositelock:
                self.compositedata = np.load(BytesIO(compositeresp.content), allow_pickle=False)
                # compute new input norm for adding subsequent input
                self.compositenorm = np.mean(self.compositedata[:]['value'], dtype=int)
                self.emptycomposite = False
                self.slplogger.info("Set new composite: %s" % str(self.compositedata[:min(10, len(self.compositedata))]))
            return SUCCESS_RETURN
        return FAILURE_RETURN
    except requests.exceptions.ConnectionError:

        self.slplogger.info("Composite download failed. Unable to connect to server")
        return OFFLINE_RETURN

The non-daemon thread that calls this method is defined here (COMPOSITE_POLL_INTERVAL is 2 seconds):

# --------------------------------------------------------------------
#   CompositePollingThread - Thread superclass that periodically 
#   polls strangeloop server for new additions to the composite loop
# --------------------------------------------------------------------

class CompositePollingThread(Thread):

    # overloaded Thread constructor
    # args:     pedal: parent Pedal object that instantiated this thread

    def __init__(self, pedal):
        Thread.__init__(self)
        self.stop = Event()
        self.pedal = pedal
        self.timestamp = None

        self.pedal.slplogger.debug("Initialized composite polling thread")

    # main thread execution loop

    def run(self):

        self.pedal.slplogger.debug("Started composite polling thread")

        while self.pedal.running:
            time.sleep(COMPOSITE_POLL_INTERVAL)

            # timestamp to determine whether any new data needs to be downloaded
            if not self.pedal.recording and self.pedal.getcomposite(timestamp=self.timestamp) == SUCCESS_RETURN:
                self.timestamp = dt.utcnow().timestamp()

                self.pedal.slplogger.debug("Downloaded new composite at %s" % dt.utcfromtimestamp(self.timestamp).strftime("%Y-%m-%d-%H:%M:%S"))

        self.pedal.slplogger.debug("Ended composite polling thread")

I'm assuming this slow download is caused by a threading issue in the program. It's also tasked with processing real-time input, which takes up the majority of the CPU. Is there anything I can do to improve this download speed? Is there a way to give the thread more priority, or should I switch to the multiprocessing module to take advangage of the RPi 4's multiple cores?

Answer 1

In the end, the issue was on the server. I changed the endpoint response from a "flask.send_file" with a BytesIO wrapper around the data I needed to a simple flask.Response object with the data as the text (similar to Serve image stored in SQLAlchemy LargeBinary column ), and now it downloads in less than ten seconds. I don't know why the download took so long before only when accessed through the program (as opposed to CLI requests), but that change took care of it.