Getting data from xhr request using python

Question

I am trying to get the movies and series on the website https://www.jiocinema.com/search/avengers I have extracted the movies using selenium but I got to know about the xhr requests. I am new to the concept and don't know whether I can use the api or not ?

The API link is: https://prod.media.jio.com/apis/common/v3.1/search/search

The xhr response looks like

Is there any way I can get the data from the above xhr response ?

Related: Python, extract XHR response data from website

Answer 1

You can use the requests library to make post requests like so...

import requests

headers = {'User-Agent':'Some user agent'}
data = requests.post('https://prod.media.jio.com/apis/common/v3.1/search/search',headers=headers).text

You might need headers to make the request...

Answer 2

You actually don't need selenium for this. You are calling a REST -API here.

Simply do something like this:

import requests
import traceback

def searchApi(query):
    endpoint = "http://prod.media.jio.com/apis/common/v3.1/search/auto"
    data = {
        "q": query
    }
    try:
        response = requests.post(endpoint, data=data)
        if(response.status_code == 200):
            for msg in response:
                print(msg)
    except Exception:
        print(traceback.format_exc())

Usage:

searchApi("avengers")

Raw output:

{
    "code": 200,
    "message": "success",
    "data": {
        "items": [
            {
                "name": "avengers grimm",
                "type": "Movies"
            },
            {
                "name":"avengers  endgame   official trailer  hindi ",
                "type":"Videos"
            },
            {
                "name":"avengers  endgame   official trailer",
                "type":"Videos"
            },
            {
                "name":"avengers endgame   special look",
                "type":"Videos"
            }
            .... continues
        ]
    }
}

Alternatively, if you want to access the data-response directly.

import json

def searchApi(query):
    endpoint = "http://prod.media.jio.com/apis/common/v3.1/search/auto"
    data = {
        "q": query
    }
    try:
        response = requests.post(endpoint, data=data)
        if(response.status_code == 200):
            response = response.json()
            for msg in response["data"]["items"]:
                print("name: ", msg["name"], "type: ", msg["type"])
    except Exception:
        print(traceback.format_exc())

Formatted output msg["name"] and msg["type"] :

name:  avengers grimm type:  Movies
name:  avengers  endgame   official trailer type:  Videos
name:  avengers endgame   special look type:  Videos
name:  avengers  endgame   official trailer  hindi  type:  Videos
name:  the avengers  earth s mightiest heroes type:  TV Shows
name:  marvel's avengers  age of ultron type:  Movies
name:  marvel's avengers assemble type:  TV Shows
name:  marvel's avengers  age of ultron   official trailer  hindi  type:  Videos
name:  marvel's avengers  age of ultron   official trailer type:  Videos
name:  marvel's the avengers type:  Movies
name:  marvel's the avengers   official trailer type:  Videos
name:  marvel's the avengers official trailer   hindi type:  Videos
name:  making of south indian avengers type:  Videos