Implementing a file-like object that writes to a queue

Question

I want to parallelize this code:

import numpy as np
import requests

# the 'in memory file' I use at the moment
bytesIO = io.BytesIO()

data = np.random.randint(0, 256, (30, 30))
np.savez(bytesIO, data=data)

# go to the beginning of the buffer again
bytesIO.seek(0)
# upload the file to a different server
requests.post("http://example.org/, files={'file': bytesIO},
              data={'filename': 'My_File'})

In here

1. the data is generated,
2. he data is sent in the request.

I want the data to be transmitted as it is being serialized to the buffer. Probably using 2 threads, connected by a queue .

For the transmission, requests supports streaming uploads .

But np.savez and also requests both expect a file-like object to read/write from/to. The queue is not file-like, and the BytesIO is not thread-safe.

What's the best way to solve this?

Answer 1

Wrap the queue in a custom file-like object. Follow the docs. This question has more details: Creating a custom file like object python suggestions?