web 抓取隐藏的 DIV，仅通过单击网页中的按钮显示

Question

I need to scrape data from a website, there is a hidden div not showing until you click a button in the website. when I use code to get html content, I cannot get the hidden div content even if I can see the hidden div data in "Inspect"

Details of url, code and hidden DIV are as below:

import requests
import bs4

url = 'https://so.gushiwen.org/guwen/bookv_3694.aspx'
doc=requests.get(url)
print(bs4.BeautifulSoup(doc.text, "html.parser"))

Answer 1

You can use selenium to locate the desired div by id and use soup.send_keys('\n') :

from selenium import webdriver
d = webdriver.Chrome('/path/to/chromedriver')
d.get('https://so.gushiwen.org/guwen/bookv_3694.aspx')
d.find_element_by_id('right2321').send_keys('\n')

Now, you can use BeautifulSoup to scrape your desired content via:

from bs4 import BeautifulSoup as soup
content = soup(d.page_source, 'html.parser').find('div', {'id':'right2321'}).text