使用python上传生成的大文件

Question

Is there a possibility to upload really big data using python?

For example to upload 100GB I can use requests module like:

with open('really_big_file') as f:
    requests.post('http://some.url/streamed', data=f)

Problem that I haven't got any file. I need to generate some random data and upload it to server. Something like this:

while uploaded_size < required_size:
    data = ''.join(random.choice(string.ascii_uppercase + string.digits) for x in range(1000))
    requests.post('http://some.url/chunk', data)
    uploaded_size += 1000

Problem with this code that it will upload many small files, while I need to upload one big file. Also I tried to generate data with function:

def gen():
    while uploaded_size < required_size:
        data = ''.join(random.choice(string.ascii_uppercase + string.digits) for x in range(1000))
        uploaded_size += 1000
        yield data
requests.post('http://some.url/chunked', data=gen())

This variant will throw error: Broken pipe

I would very appreciate if you will say where is my mistake or what module I should try.

UPD: Resolved with sockets

Answer 1

requests.post('http://some.url/chunked', data=''.join(gen()))

gen() returns an iterator. you should iterate over it to get the values.