Use %20 instead of + for space in python query parameters

Question

I have written the following python script, using python requests ( http://requests.readthedocs.org/en/latest/ ):

import requests

payload = {'key1': 'value  1', 'key2': 'value 2'}
headers = {'Content-Type': 'application/json;charset=UTF-8'}
r = requests.get("http://example.com/service", params=payload, headers=headers, 
             auth=("admin", "password"))

If I look at the access log of the server, the incoming request is: /service?key1=value++1&key2=value+2

However, the server expects ... value%20%201& ...

I have read that using a + as a placeholder for a space is part of content type application/x-www-form-urlencoded, but clearly I have requested application/json.

Anybody know how to use %20 as a space in query parameters of pythons requests?

Answer 1

To follow up on @WeaselFox's answer, they introduced a patch that accepts a quote_via keyword argument to urllib.parse.urlencode . Now you could do this:

import requests
import urllib

payload = {'key1': 'value  1', 'key2': 'value 2'}
headers = {'Content-Type': 'application/json;charset=UTF-8'}
params = urllib.parse.urlencode(payload, quote_via=urllib.parse.quote)
r = requests.get("http://example.com/service", params=params, headers=headers,
    auth=("admin", "password"))

Answer 2

try it.

import urllib
urllib.urlencode(params)

http://docs.python.org/2/library/urllib.html#urllib.urlencode

Answer 3

I only find urllib.parse.quote , which can replace space to %20 .

But quote could not convert a dict.

so, We must use quote to transform dict in advance.

---

#for python3
from urllib.parse import quote

payload = {'key1': 'value  1', 'key2': 'value 2'}

newpayload = {}
for (k, v) in payload.items():
    newpayload[quote(k)] = quote(v)
print(newpayload)
#print result: {'key1': 'value%20%201', 'key2': 'value%202'}
# Now, you can use it in requests

Answer 4

this seems to be a known bug/issue in python :

http://bugs.python.org/issue13866

I think you will have to go around this issue using urllib and urllib2 and avoid requests. look at the bug reports for some tips on how to do that.

Answer 5

We can use urllib2.Request to call url

import urllib2

send_params = {'key1': 'value  1', 'key2': 'value 2'}
new_send_params = []
for (k, v) in send_params.items():
    new_send_params.append(k + "=" + urllib2.quote(v))

url = 'http://example.com/service?'+ '&'.join(new_send_params)
req = urllib2.Request(url)
response = urllib2.urlopen(req)
print "Request URL: " + url
#Request URL: http://example.com/service?key1=value%20&key2=value%202
print response.read()
#Python Master Request handler 2016-07-04 16:05:19.928132 . Your request path is  /service?key1=value%20&key2=value%202

Answer 6

PYTHON 2.7

• Override the urllib.quote_pluse with urllib.quote

• The urlencoder uses urllib.quote_pluse to encode the data.

---

code

import requests
import urllib
urllib.quote_plus=urllib.quote # A fix for urlencoder to give %20 
payload = {'key1': 'value  1', 'key2': 'value 2'}
headers = {'Content-Type': 'application/json;charset=UTF-8'}
param = urllib.urlencode(payload) #encodes the data
r = requests.get("http://example.com/service", params=param, headers=headers, 
             auth=("admin", "password"))

output

the output for param = urllib.urlencode(payload)
'key2=value%202&key1=value%20%201' 

Answer 7

from urllib.parse import urlencode

def to_query_string(params):
    return urlencode(params, doseq=True).replace('+', '%20')

---

You could pass a string to params instead of a dictionary, and manually handle the spaces.

Maybe something along the lines of

 
 
  
  def to_query_string(p, s=''): for k in p: v = p[k] if isinstance(v, str): s += f'{k}={v}&'.replace(' ', '%20') elif isinstance(v, int): s += f'{k}={v}&' elif isinstance(v, list): for i in v: s += f'{k}={i}&' return s[:-1] # remove last '&'
 
 

which can be used as

min = 10 max = 30 params = {'query': f'score between {min} and {max}', 'limit': 1, 'information': ['name', 'location']} response = get('/api/dogs', params=to_query_string(params))

Answer 8

requests uses internally quote() from urllib.parse

import requests
from requests.utils import quote

url='https://api-adresse.data.gouv.fr/search/'
requests.get(url, params='q='+quote('rue du Jura')).url

https://api-adresse.data.gouv.fr/search/?q=rue%20du%20Jura

If you give a str/byte (and not a dict), requests don't encode you params