Python requests adding a token to JSON file

Question

I am doing a POST call with requests to an API with a JSON file as the payload.

From what I can tell from the API logs, a token is being added to the payload which is causing issues at the receiving API.

Here is the code of the post request:

import requests

def post_to_api(self, spider):
    url = 'http://106309.nitrousapp.com:3000/api/v1/imports/batch_create'
    headers = {'X-Api-Key': 'TOHhsxPyJnpZcUhLMMqj0XwipSY48PQxCXzIsW8FFpMt', 'Content-Type': 'application/json'}
    r = requests.post(url, files={'file': open('fancy_items.json', 'r+b')}, headers=headers)
    print r.json

The raw JSON is all correct, checked it on JSONLint.

However looking at the logs on the API, it shows a token added like this to the bottom of the file:

{"name": "Mystical Fire Colorant", "country": "USA", "price": "75", "category_old": "Sports & Outdoors", "image_url": "http://thingd-media-ec1.thefancy.com/default/701418813568058872_5171d5c216c7.jpg", "i
nterest": "113", "link": "https://fancy.com/things/701418813568058872/Mystical-Fire-Colorant", "company": "Fancy"}]}
--0c5171b3fe574bdb921838ef81c437c2--
':

I have tried a manual call to the API and it didn't add this token, so I am thinking its something that requests or Python is doing.

Here is the total JSON output from the API logs:

{"product":[{"name": "A.P.C. Locker Hoody (Dark Navy)", "country": "USA", "price": "215", "category_old": null, "image_url": "http://thingd-media-ec6.thefancy.com/default/1011706283071373547_87a8a65d3b2a.
jpg", "interest": "1", "link": "https://fancy.com/things/1011706283071373547/A.P.C.-Locker-Hoody-(Dark-Navy)", "company": "Fancy"},
{"name": "Funko POP! Hulk Ceramic Mug", "country": "USA", "price": "17", "category_old": null, "image_url": "http://thingd-media-ec1.thefancy.com/default/1011471115609971981_831322b6e212.jpg", "interest":
 "None", "link": "https://fancy.com/things/1011471115609971981/Funko-POP!-Hulk-Ceramic-Mug", "company": "Fancy"},
{"name": "Khaki Convertible Backpack by Hex", "country": "USA", "price": "100", "category_old": "Men", "image_url": "http://thingd-media-ec3.thefancy.com/default/986174222386597367_42a175cb4c86.jpg", "int
erest": "4", "link": "https://fancy.com/things/986174222386597367/Khaki-Convertible-Backpack-by-Hex", "company": "Fancy"},
{"name": "Darth Vader's Lightsaber MimoPowerTube2 Star Wars Series", "country": "USA", "price": "30", "category_old": null, "image_url": "http://thingd-media-ec1.thefancy.com/default/1011467404649176319_e
719110203ae.jpg", "interest": "3", "link": "https://fancy.com/things/1011467404649176319/Darth-Vader's-Lightsaber-MimoPowerTube2-Star-Wars-Series", "company": "Fancy"},
{"name": "Boba Fett MimoPowerTube2 Star Wars Series", "country": "USA", "price": "30", "category_old": null, "image_url": "http://thingd-media-ec3.thefancy.com/default/1011472861069581593_f2390073e692.jpg
", "interest": "2", "link": "https://fancy.com/things/1011472861069581593/Boba-Fett-MimoPowerTube2-Star-Wars-Series", "company": "Fancy"},
{"name": "Funko POP! Iron Man Ceramic Mug", "country": "USA", "price": "17", "category_old": null, "image_url": "http://thingd-media-ec4.thefancy.com/default/1011471256807021839_7b11976ede50.jpg", "intere
st": "1", "link": "https://fancy.com/things/1011471256807021839/Funko-POP!-Iron-Man-Ceramic-Mug", "company": "Fancy"},
{"name": "Funko POP! Spider-Man Ceramic Mug", "country": "USA", "price": "17", "category_old": null, "image_url": "http://thingd-media-ec4.thefancy.com/default/1011473609744456989_f62ee350ecbf.jpg", "inte
rest": "1", "link": "https://fancy.com/things/1011473609744456989/Funko-POP!-Spider-Man-Ceramic-Mug", "company": "Fancy"},
{"name": "Charcoal Boiled Wool Sweatpant by Apolis", "country": "USA", "price": "248", "category_old": "Men", "image_url": "http://thingd-media-ec1.thefancy.com/default/986743205628222511_1853b853a79b.jpg
", "interest": "2", "link": "https://fancy.com/things/986743205628222511/Charcoal-Boiled-Wool-Sweatpant-by-Apolis", "company": "Fancy"},
{"name": "Funko POP! Spider-Man Salt & Pepper Shaker Set", "country": "USA", "price": "22", "category_old": null, "image_url": "http://thingd-media-ec1.thefancy.com/default/1011477384920046901_2a177ed28ef
6.jpg", "interest": "None", "link": "https://fancy.com/things/1011477384920046901/Funko-POP!-Spider-Man-Salt-&-Pepper-Shaker-Set", "company": "Fancy"},
{"name": "Fur Real Greeting Card", "country": "USA", "price": "5", "category_old": null, "image_url": "http://thingd-media-ec5.thefancy.com/default/1011526354610429681_3718a08c1cd9.jpg", "interest": "2",
"link": "https://fancy.com/things/1011526354610429681/Fur-Real-Greeting-Card", "company": "Fancy"},
{"name": "Coolest Cooler", "country": "USA", "price": "485", "category_old": "Food", "image_url": "http://thingd-media-ec3.thefancy.com/default/671492680575882582_be703be1c98c.jpg", "interest": "139", "li
nk": "https://fancy.com/things/671492680575882582/Coolest-Cooler-po", "company": "Fancy"},
{"name": "QBracelet Smartphone Charger", "country": "USA", "price": "110", "category_old": "Gadgets", "image_url": "http://thingd-media-ec2.thefancy.com/default/698272167082598052_01939a626fe5.jpg", "inte
rest": "18", "link": "https://fancy.com/things/698272167082598052/QBracelet-Smartphone-Charger-po", "company": "Fancy"},
{"name": "Geodesic Playhouse", "country": "USA", "price": "300", "category_old": "Kids", "image_url": "http://thingd-media-ec4.thefancy.com/default/955512370883461577_260ed730ea0e.jpg", "interest": "13",
"link": "https://fancy.com/things/955512370883461577/Geodesic-Playhouse", "company": "Fancy"},
{"name": "Get Out Of My Life Tee", "country": "USA", "price": "37", "category_old": null, "image_url": "http://thingd-media-ec3.thefancy.com/default/1011529914114907917_071fb0dd5655.jpg", "interest": "1",
 "link": "https://fancy.com/things/1011529914114907917/Get-Out-Of-My-Life-Tee", "company": "Fancy"},
{"name": "Catnipped Tank Top", "country": "USA", "price": "24", "category_old": null, "image_url": "http://thingd-media-ec3.thefancy.com/default/1011535748509212461_e0ebf6f78f6a.jpg", "interest": "3", "li
nk": "https://fancy.com/things/1011535748509212461/Catnipped-Tank-Top", "company": "Fancy"},
{"name": "Harvard Naw Crewneck", "country": "USA", "price": "50", "category_old": null, "image_url": "http://thingd-media-ec6.thefancy.com/default/1011555628595814261_0e08275d489e.jpg", "interest": "2", "
link": "https://fancy.com/things/1011555628595814261/Harvard-Naw-Crewneck", "company": "Fancy"},
{"name": "Black Cat-O-Lantern Tee", "country": "USA", "price": "35", "category_old": null, "image_url": "http://thingd-media-ec2.thefancy.com/default/1011559329523834755_0c5541944a26.jpg", "interest": "1"
, "link": "https://fancy.com/things/1011559329523834755/Black-Cat-O-Lantern-Tee", "company": "Fancy"},
{"name": "Creep It Real Crewneck", "country": "USA", "price": "26", "category_old": null, "image_url": "http://thingd-media-ec3.thefancy.com/default/1011538479613484857_f4419054bbfa.jpg", "interest": "2",
 "link": "https://fancy.com/things/1011538479613484857/Creep-It-Real-Crewneck", "company": "Fancy"},
{"name": "F*ck Off Crop Tank", "country": "USA", "price": "30", "category_old": null, "image_url": "http://thingd-media-ec3.thefancy.com/default/1011572077062266779_84283d6694da.jpg", "interest": "1", "li
nk": "https://fancy.com/things/1011572077062266779/F*ck-Off-Crop-Tank", "company": "Fancy"},
{"name": "Pinemeowpple Crop Top", "country": "USA", "price": "26", "category_old": null, "image_url": "http://thingd-media-ec4.thefancy.com/default/1011563553640947595_83b04d570969.jpg", "interest": "5",
"link": "https://fancy.com/things/1011563553640947595/Pinemeowpple-Crop-Top", "company": "Fancy"},
{"name": "Monday Blues Ringer Tee", "country": "USA", "price": "42", "category_old": null, "image_url": "http://thingd-media-ec1.thefancy.com/default/1011544620326199127_9d35a8431b9b.jpg", "interest": "1"
, "link": "https://fancy.com/things/1011544620326199127/Monday-Blues-Ringer-Tee", "company": "Fancy"},
{"name": "Matit\u00e0 Coat Rack", "country": "USA", "price": "199", "category_old": "Home", "image_url": "http://thingd-media-ec2.thefancy.com/default/1011576991335325607_5aebe93b44ec.jpg", "interest": "4
", "link": "https://fancy.com/things/1011576991335325607/Matit%C3%A0-Coat-Rack", "company": "Fancy"},
{"name": "Gasometro Ring", "country": "USA", "price": "199", "category_old": "Women", "image_url": "http://thingd-media-ec5.thefancy.com/default/1011573460410834847_1420988eda40.jpg", "interest": "2", "li
nk": "https://fancy.com/things/1011573460410834847/Gasometro-Ring", "company": "Fancy"},
{"name": "Mystical Fire Colorant", "country": "USA", "price": "75", "category_old": "Sports & Outdoors", "image_url": "http://thingd-media-ec1.thefancy.com/default/701418813568058872_5171d5c216c7.jpg", "i
nterest": "113", "link": "https://fancy.com/things/701418813568058872/Mystical-Fire-Colorant", "company": "Fancy"}]}
--0c5171b3fe574bdb921838ef81c437c2--
':

Answer 1

It looks like requests is submitting a multipart/form-data encoded POST request because you're loading a file (ie you have the files argument in your requests.post() invocation). This behaviour is described in the POST a Multipart-Encoded File section of the requests documentation . Also, for more info on the different encodings in POST requests, see: What does enctype='multipart/form-data' mean?

I think what you're trying to do is POST some JSON, and not send a file that contains JSON. Here's some rough sample code:

import requests
import json

def post_to_api(self, spider):
    url = 'http://106309.nitrousapp.com:3000/api/v1/imports/batch_create'
    headers = {'X-Api-Key': 'TOHhsxPyJnpZcUhLMMqj0XwipSY48PQxCXzIsW8FFpMt', 'Content-Type': 'application/json'}
    with open('fancy_items.json') as f:
        json_data = json.load(f)
        r = requests.post(url, json=json_data, headers=headers)
        print r.status_code
        print r.content

See More complicated POST requests in the requests documentation for more info.

PS If that's a genuine API key, you might like to revoke it at some point.