How can I upload files through aiohttp using response from get request?
Question
To start off, I am writing an async wrapper for the WordPress REST API. I have a Wordpress site hosted on Bluehost. I am working with the endpoint for media (image) uploads. I have successfully managed to upload an image but there are 2 changes I would like to make. The second change is what I really want, but out of curiosity, I would like to know how to implement change 1 too. I'll provide the code first and then some details.
Working code
async def upload_local_pic2(self, local_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'title': title, 'status':'publish'}
mpwriter.append_json(json)
with open(local_url, 'rb') as f:
print(f)
payload = mpwriter.append(f)
async with self.session.post(url, data=payload) as response:
x = await response.read()
print(x)
Change 1
The first change is uploading using aiofiles.open() instead of just using open() as I expect to be processing lots of files. The following code does not work.
async def upload_local_pic(self, local_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'title': title, 'status':'publish'}
mpwriter.append_json(json)
async with aiofiles.open(local_url, 'rb') as f:
print(f)
payload = mpwriter.append(f)
async with self.session.post(url, data=payload) as response:
x = await response.read()
print(x)
Change 2
My other change is that I would like to have another function that can upload the files directly to the WordPress server without downloading them locally. So instead of getting a local picture, I want to pass in the url of an image online. The following code also does not work.
async def upload_pic(self, image_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'title':title, 'status':'publish'}
mpwriter.append_json(json)
async with self.session.get(image_url) as image_response:
image_content = image_response.content
print(image_content)
payload = mpwriter.append(image_content)
async with self.session.post(url, data = payload) as response:
x = await response.read()
print(x)
Details/Debugging
I'm trying to figure out why each one won't work. I think the key is the calls to print(image_content) and print(f) that show what exactly I am inputting to mpwriter.append
In the example that works where I just use the standard Python open() function, I am apparently passing in <_io.BufferedReader name='/redactedfilepath/index.jpeg'>
In the change 1 example with aiofile, I am passing in <aiofiles.threadpool.binary.AsyncBufferedReader object at 0x7fb803122250> Wordpress will return this html:
b'<head><title>Not Acceptable.</title></head><body><h1>Not Acceptable.</h1><p>An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.</p></body></html>'
And finally, in change 2 where I try to pass in what the get request to the url gives me I get <StreamReader 292 bytes> . The response returned by WordPress is the same as above with Mod Security.
Any idea how I can make these examples work? It seems like they are all some type of io reader but I guess the underlying aiohttp code treats them differently.
Also this shouldn't really matter, but this is the url I am passing into the change 2 example.
2 answers
solution1
1 2020-11-12 01:14:26
Ok, so I figured out both changes.
For the first change when trying to read a file with aiofiles , I need to just read the whole file instead of passing in the file handler. Also, I need to set the content disposition manually.
async def upload_local_pic(self, local_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'status':'publish'}
mpwriter.append_json(json)
async with aiofiles.open(local_url, mode='rb') as f:
contents = await f.read()
payload = mpwriter.append(contents)
payload.set_content_disposition('attachment', filename= title+'.jpg')
async with self.session.post(url, data=payload) as response:
x = await response.read()
print(x)
For the second change, it's a similar concept with just uploading a file directly from the URL. Instead of passing in the handler that will read the content, I need to read the entire content first. I also need to set the content-disposition manually.
async def upload_pic(self, image_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'status':'publish'}
mpwriter.append_json(json)
async with self.session.get(image_url) as image_response:
image_content = await image_response.read()
payload = mpwriter.append(image_content)
payload.set_content_disposition('attachment', filename=title+'.jpg')
async with self.session.post(url, data = payload) as response:
x = await response.read()
print(x)
solution2
0 2022-07-27 11:23:19
I will answer only to the title of the post (and not the questions that are in between).
The following code should give a short example of how to upload a file from URL#1 to URL#2 (without the need to download the file to the local machine and only then do the upload).
I will give two examples here:
- Read all the content of the file into the memory (not downloading). This is of-course not so good when working with huge files...
- Read and send the file in chunks (so we won't read all the file content at once).
Example #1: Reading all file content AT ONCE and uploading
import asyncio
import aiohttp
async def http_upload_from_url(src, dst):
async with aiohttp.ClientSession() as session:
src_resp = await session.get(src)
#print(src_resp)
dst_resp = await session.post(dst, data=src_resp.content)
#print(dst_resp)
try:
asyncio.run(http_upload_from_url(SRC_URL, DST_URL))
except Exception as e:
print(e)
Example #2: Reading file content IN CHUNKS and uploading
import asyncio
import aiohttp
async def url_sender(url=None, chunk_size=65536):
async with aiohttp.ClientSession() as session:
resp = await session.get(url)
#print(resp)
async for chunk in resp.content.iter_chunked(chunk_size):
#print(f"send chunk with size {len(chunk)}")
yield chunk
async def chunked_http_upload_from_url(src, dst):
async with aiohttp.ClientSession() as session:
resp = await session.post(dst, data=url_sender(src))
#print(resp)
#print(await resp.text())
try:
asyncio.run(chunked_http_upload_from_url(SRC_URL, DST_URL))
except Exception as e:
print(e)
Some notes:
- You need to define SRC_URL and DST_URL.
- I've only added the prints for debug (in-case you don't get a [200 OK] response).
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.