使用 python http post 从内存上传二进制数据或文件

Question

I have seen the receipes for uploading files via multipartform-data and pycurl .
Both methods seem to require a file on disk.
Can these recipes be modified to supply binary data from memory instead of from disk ?
I guess I could just use a xmlrpc server instead.
I wanted to get around having to encode and decode the binary data and send it raw...
Do pycurl and mutlipartform-data work with raw data ?

Answer 1

This (small) library will take a file descriptor, and will do the HTTP POST operation: https://github.com/seisen/urllib2_file/

You can pass it a StringIO object (containing your in-memory data) as the file descriptor.

Answer 2

Find something that cas work with a file handle. Then simply pass a StringIO object instead of a real file descriptor.

Answer 3

I met similar issue today, after tried both and pycurl and multipart/form-data, I decide to read python httplib/urllib2 source code to find out, I did get one comparably good solution:

1. set Content-Length header(of the file) before doing post
2. pass a opened file when doing post

Here is the code:

import urllib2, os
image_path = "png\\01.png"
url = 'http://xx.oo.com/webserviceapi/postfile/'
length = os.path.getsize(image_path)
png_data = open(image_path, "rb")
request = urllib2.Request(url, data=png_data)
request.add_header('Cache-Control', 'no-cache')
request.add_header('Content-Length', '%d' % length)
request.add_header('Content-Type', 'image/png')
res = urllib2.urlopen(request).read().strip()
return res

see my blog post: http://www.2maomao.com/blog/python-http-post-a-binary-file-using-urllib2/

Answer 4

Following python code works reliable on 2.6.x. The input data is of type str. Note that the server that receives the data has to loop to read all the data as the large data sizes will be chunked. Also attached java code snippet to read the chunked data.

def post(self, url, data):
    self.curl = pycurl.Curl()
    self.response_headers = StringIO.StringIO()
    self.response_body = io.BytesIO()
    self.curl.setopt(pycurl.WRITEFUNCTION, self.response_body.write)
    self.curl.setopt(pycurl.HEADERFUNCTION, self.response_headers.write)
    self.curl.setopt(pycurl.FOLLOWLOCATION, 1)
    self.curl.setopt(pycurl.MAXREDIRS, 5)
    self.curl.setopt(pycurl.TIMEOUT, 60)
    self.curl.setopt(pycurl.ENCODING,"deflate, gzip")
    self.curl.setopt(pycurl.URL, url)
    self.curl.setopt(pycurl.VERBOSE, 1)
    self.curl.setopt(pycurl.POST,1)
    self.curl.setopt(pycurl.POSTFIELDS,data)
    self.curl.setopt(pycurl.HTTPHEADER, [ "Content-Type: octate-stream" ])
    self.curl.setopt(pycurl.POSTFIELDSIZE, len(data))
    self.curl.perform()
    return url, self.curl.getinfo(pycurl.RESPONSE_CODE),self.response_headers.getvalue(), self.response_body.getvalue()

Java code for the servlet engine:

int postSize = Integer.parseInt(req.getHeader("Content-Length"));
results = new byte[postSize];
int read = 0;
while(read < postSize) {
      int n = req.getInputStream().read(results);
      if (n < 0) break;
      read += n;
  }

Answer 5

找到了一个适用于cherrypy文件上传示例的解决方案： urllib2-binary-upload.py

Answer 6

import io         # Part of core Python
import requests   # Install via: 'pip install requests'

# Get the data in bytes. I got it via:
#   with open("smile.gif", "rb") as fp: data = fp.read()
data = b"GIF89a\x12\x00\x12\x00\x80\x01\x00\x00\x00\x00\xff\x00\x00!\xf9\x04\x01\n\x00\x01\x00,\x00\x00\x00\x00\x12\x00\x12\x00\x00\x024\x84\x8f\x10\xcba\x8b\xd8ko6\xa8\xa0\xb3Wo\xde9X\x18*\x15x\x99\xd9'\xad\x1b\xe5r(9\xd2\x9d\xe9\xdd\xde\xea\xe6<,\xa3\xd8r\xb5[\xaf\x05\x1b~\x8e\x81\x02\x00;"

# Hookbin is similar to requestbin - just a neat little service
# which helps you to understand which queries are sent
url = 'https://hookb.in/je2rEl733Yt9dlMMdodB'

in_memory_file = io.BytesIO(data)
response = requests.post(url, files=(("smile.gif", in_memory_file),))