Get Ajax Json Output with Python Requests Library

Question

I am requesting an Ajax Web site with a Python script and fetching cities and branch offices of http://www.yurticikargo.com/bilgi-servisleri/Sayfalar/en-yakin-sube.aspx

I completed the first step with posting {cityID: 34} to this url and fetc the JSON output.

http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/GetTownByCity

But I can not retrive the JSON output with Python although i get succesfully with Chrome Advanced Rest Client Extension, posting {cityID:54,townID:5416,unitOnDutyFlag:null,closestFlag:2}

http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-unitservices.aspx/GetUnit

All of the source code is here

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import requests
import json


class Yurtici(object):

    baseUrl = 'http://www.yurticikargo.com/'
    ajaxRoot = '_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/'
    getTown = 'GetTownByCity'
    getUnit = 'GetUnit'
    urlGetTown = baseUrl + ajaxRoot + getTown
    urlGetUnit = baseUrl + ajaxRoot + getUnit 
    headers = {'content-type': 'application/json','encoding':'utf-8'}

    def __init__(self):
        pass

    def ilceler(self, plaka=34): # Default testing value
        payload = {'cityId':plaka}
        url = self.urlGetTown
        r = requests.post(url, data=json.dumps(payload), headers=self.headers)
        return r.json() # OK

    def subeler(self, ilceNo=5902): # Default testing value
        # 5902 Çerkezköy 
        payload=  {'cityID':59,'townID':5902,'unitOnDutyFlag':'null','closestFlag':0}
        url = self.urlGetUnit
        headers = {'content-type': 'application/json','encoding':'utf-8'}
        r = requests.post(url, data=json.dumps(payload), headers=headers)
        print  r.status_code, r.raw.read()

if __name__ == '__main__':        
    a = Yurtici()
    print a.ilceler(37) # OK
    print a.subeler()   # NOT OK !!!

Answer 1

Your code isn't posting to the same url you're using in your text example.

Let's walk through this backwards. First, let's look at the failing POST.

url = self.urlGetUnit
headers = {'content-type': 'application/json','encoding':'utf-8'}
r = requests.post(url, data=json.dumps(payload), headers=headers)

So we're posting to a URL that is equal to self.urlGetUnit . Ok, let's look at how that's defined:

baseUrl = 'http://www.yurticikargo.com/'
ajaxRoot = '_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/'
getUnit = 'GetUnit'
urlGetUnit = baseUrl + ajaxRoot + getUnit

If you do the work in urlGetUnit, you get that the URL will be http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/GetUnit . Let's put this alongside the URL you used in Chrome to compare the differences:

http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-sswservices.aspx/GetUnit
http://www.yurticikargo.com/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy-unitservices.aspx/GetUnit

See the difference? ajaxRoot is not the same for both URLs. Sort that out and you'll get back a JSON response.