run curl call in python using requests / http.client

Question

I am trying to execute a curl call in python.

curl -X POST --header "Content-Type: application/json" --header "Accept: application/json" --header "Authorization:xxxx" -d "{
\"ProductNames\": [
\"PReport\"
],
\"SearchType\": \"FullAddress\",
\"FullAddress\": \"123 Main Street, New York, 10001\",
\"ReferenceId\": \"\"
}" "https://api.aws.com:443/api/rep/grep"

I have tried:

 import http.client 
 import urllib.parse 

 addr = '123 Main Street, New York, 10001'
 prod = 'PReport'
 stype = 'FullAddress'
 

 conn = http.client.HTTPSConnection("https://api.aws.com:443/api/rep/grep") 

    headers = { 
        'Accept': "application/json", 
        'Authorization': "xxxx", 
        } 

    address = urllib.parse.quote(addr)

    url = "https://api.aws.com:443/api/rep/grep?ProductName={}&SearchType={}&FullAddress={}".format(prod, stype, address)

    conn.request("GET", url, headers=headers)

    res = conn.getresponse() 

In swagger, I only see: https://api.aws.com:443/api/rep/grep , not the full request url.

How do I neatly make a curl call in python with the info above?

Answer 1

When you say that you want to execute that curl call I assume that it works and you want to make the same request with Python.

There may be more issues with your Python code, but one issue I spot right away that I believe to be incorrect is you using URL query parameters where your curl command uses -d , which is a short form for --data . Here is the relevant documentation:

--data

(HTTP MQTT) Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button. This will cause curl to pass the data to the server using the content-type application/x-www-form-urlencoded. Compare to -F, --form.

[...]

https://curl.se/docs/manpage.html

That means that it goes into the request body. However, in your Python code you place those as query string parameters, which does not seem to be equivalent. But you also say that you do "not see the full request url", which confuses me a bit, because you should not have seen a URL other than "https://api.aws.com:443/api/rep/grep" with curl, either. One quick way to see what you get with curl is to use the tool "netcat", usually invoked by "nc" when you have it installed on your system.

For example, to listen on port 1234 you would call netcat like this:

nc -l 1234

For more information on netcat, see https://linux.die.net/man/1/nc . Of course, there are other tools you can use. This is just a suggestion.

When making the same request against localhost, port 1234, with plain HTTP, using your above curl command line, it will print something along the lines of:

POST /api/rep/grep HTTP/1.1
Host: localhost:1234
User-Agent: curl/7.79.1
Content-Type: application/json
Accept: application/json
Authorization:xxxx
Content-Length: 132

{
"ProductNames": [
"PReport"
],
"SearchType": "FullAddress",
"FullAddress": "123 Main Street, New York, 10001",
"ReferenceId": ""
}

So, the request parameters are not part of the URL in the way you believe and you might be having the wrong expectations of what you should be seeing in the implementation code of your API endpoint.

But, if the curl call is working nonetheless, then Python code that is more equivalent to your curl call might be this:

import requests

url = "https://api.aws.com:443/api/rep/grep"
headers = {
    "Content-Type": "application/json",
    "Accept": "application/json",
    "Authorization": "xxxx"
}
data = """{
    "ProductNames": [ "PReport" ],
    "SearchType": "FullAddress",
    "FullAddress": "123 Main Street, New York, 10001",
    "ReferenceId": ""
}"""

resp = requests.post(url, headers=headers, data=data)
print(resp.status_code)

This code, when run against a port listened on by netcat in the same manner, displays a request that appears to match that of the curl call above.

There is also a json parameter in most recent versions that might make this code cleaner by using a dictionary for the data to be sent:

data = {
    "ProductNames": [ "PReport" ],
    "SearchType": "FullAddress",
    "FullAddress": "123 Main Street, New York, 10001",
    "ReferenceId": ""
}
requests.post(url, headers=headers, json=data)

Documentation for the latest version of the requests Python module can be found under https://requests.readthedocs.io/en/latest/ .

Also, the task of converting various ways of performing HTTP requests of the kind in your question is so common that there are a lot of online tools that do so automatically. One example is https://curlconverter.com/ . That site also contains other relevant information, but I have not double-checked its results for your query. It does look like it produces the same code. It's probably best, though, to spend the time once to understand how the various parts of common HTTP requests are transmitted before relying too heavily on automated tools.