Website's search function with Python requests

Question

Cannot successfully implement website's search function using python requests. Performing search at the website results with https://bidbaits.ru/search/#used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0&orderBy=2 , but performing it with requests return nothing:

r = requests.get('https://bidbaits.ru/search/#used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0&orderBy=2')
print(r.text)

Also, how is it possible to send query as a parameter?

Answer 1

This page uses JavaScript to display elements on page but requests / BeautifulSoup can't run JavaScript .

But using DevTools in Chrome/Firefox you can see JavaScript ( AJAX / XHR ) reads data from other url

https://bidbaits.ru/searchresults/?page=1&seo=SEARCH&orderBy=2&used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0

which works with requests if you use header Referer .
(I checked it manually running code with different headers)

I use module json only to display it with indentations. You can work directly with dictionary data .

import requests

#s = requests.Session()

headers = {
    #'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0',
    #'Accept': 'application/json, text/javascript, */*; q=0.01',
    #'X-Requested-With': 'XMLHttpRequest',  # AJAX
    'Referer': 'https://bidbaits.ru/',
}

url = 'https://bidbaits.ru/searchresults/?page=1&seo=SEARCH&orderBy=2&used=1&new=1&original=1&replica=1&auctions=1&fixedPrice=1&discount=0&trustedSeller=0&lockLocation=0&q=cyarl&c=0'
r = requests.get(url, headers=headers)    

data = r.json() 

#print(data)

import json

print(json.dump(data, indent=2))

Result:

{
  "totalCount": "4",
  "totalCountMsg": "\u041d\u0430\u0439\u0434\u0435\u043d\u043e <span class=\"b-bsr_count\">4</span> \u0432\u0430\u0440\u0438\u0430\u043d\u0442\u0430",
  "items": [
    {
      "itemId": "544665",
      "name": "Jackson Cyarl Blade 48F 4g Sinking Ama Gaeru Rain Frog",
      "price": "700",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/544665/",
      "imageUrl": "//c.bidbaits.ru/items/544665/thumb/1580816722-0419.JPG",
      "userName": "Survivor-Demon",
      "userRating": "658",
      "userUrl": "/users/survivor-demon/",
      "typeId": "3",
      "bets": "0 \u0441\u0442\u0430\u0432\u043e\u043a",
      "isTrusted": true,
      "timeLeft": 235373
    },
    {
      "itemId": "735811",
      "name": "Jackson Cyarl 20mm 1,8g sinking",
      "price": "450",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/735811/",
      "imageUrl": "//c.bidbaits.ru/items/735811/thumb/1600933789-5099.jpg",
      "userName": "Dmitr79",
      "userRating": "22",
      "userUrl": "/users/dmitr79/",
      "typeId": "3",
      "bets": "0 \u0441\u0442\u0430\u0432\u043e\u043a",
      "isTrusted": false,
      "timeLeft": 572273
    },
    {
      "itemId": "840409",
      "name": "Jackson Cyarl 25S (1,8 \u0433.) OT",
      "price": "700",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/840409/",
      "imageUrl": "//c.bidbaits.ru/items/840409/thumb/1611249724-4935.JPG",
      "userName": "Ima2021",
      "userRating": "6",
      "userUrl": "/users/ima2021/",
      "typeId": "3",
      "bets": "0 \u0441\u0442\u0430\u0432\u043e\u043a",
      "isTrusted": false,
      "timeLeft": 2050658
    },
    {
      "itemId": "840416",
      "name": "Jackson Cyarl 25S (1,8 \u0433.) PA",
      "price": "700",
      "isSpecialOffer": false,
      "isRecommendedItem": false,
      "priceSpecial": 0,
      "url": "/bids/840416/",
      "imageUrl": "//c.bidbaits.ru/items/840416/thumb/1611250255-9944.JPG",
      "userName": "Ima2021",
      "userRating": "6",
      "userUrl": "/users/ima2021/",
      "typeId": "3",
      "bets": "0 \u0441\u0442\u0430\u0432\u043e\u043a",
      "isTrusted": false,
      "timeLeft": 2051635
    }
  ],
  "currentPage": 1,
  "hasNextPage": false,
  "isSEO": false,
  "pageTitle": "\u041f\u043e\u0438\u0441\u043a: cyarl",
  "

---

Working directly with data

data = r.json() 

#print(data)
#import json
#print(json.dumps(data, indent=2))

for item in data['items']:
    print('name:', item['name'])
    print('price:', item['price'])    
    print('---')

Result:

name: Jackson Cyarl Blade 48F 4g Sinking Ama Gaeru Rain Frog
price: 700
---
name: Jackson Cyarl 20mm 1,8g sinking
price: 450
---
name: Jackson Cyarl 25S (1,8 г.) OT
price: 700
---
name: Jackson Cyarl 25S (1,8 г.) PA
price: 700
---