当使用请求和beautifulsoup加载更多内容时，我无法抓取下一页上url不会更改的网站

Question

Python Beatifulsoup requests

import requests
import re
import os
import csv
from bs4 import BeautifulSoup





for d in searche:
    truelink = d.replace(" ","-")
    truelinkk=('https://www.fb.com

    r = requests.get(truelinkk,headers=headers).text
    soup=BeautifulSoup(r,'lxml')
    mobile=soup.find_all('li',class_='EIR5N')
 

I am beginner to python. I can't scrape a website where url doesn't change on its next page when load more using requests and beautifulsoup please can someone visit the site let me know the procedure for scraping above websites using beautifulsoup and requests. Any answer would be appreciated Thankyou Please look this link https://www.olx.in/hyderabad_g4058526/q-Note-9-max-pro?isSearchCall=true

Answer 1

You can use selenium in headless mode instead of requests . Eventho selenium is used for web automation it can help you in this case.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

begin = time.time()

options = Options()
options.headless = True
options.add_argument('--log-level=3')
driver = webdriver.Chrome(options=options)

Since the URL doesn't change you have to click on the button that you want by getting its xpath and:

driver.find_element_by_xpath('xpath code').click()

You can avoid using requests and you can get the source code of the page by using:

html_text = driver.page_source
soup = BeautifulSoup(html_text, 'lxml')