简体   繁体   中英

Selenium will not show javascript loaded table in webpage with Python

I am trying to scrape the following webpage: https://steamdb.info/app/730/graphs/ (I have gained permission from the website)

The problem is that the "Monthly Breakdown" table seems to be loaded by Javascript, and BeautifulSoup does not work. When using Selenium to open the webpage, it says that to see the table "You must have Javascript enabled.", which should be enabled when using Selenium. Here is my code:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--enable-javascript")
browser = webdriver.Chrome(options=options)
browser.maximize_window()

url = "https://steamdb.info/app/730/graphs/"
browser.get(url)

Any ways to solve this problem?

How the page should look: 在此处输入图像描述

How it looks on Selenium: 在此处输入图像描述

Try this and see if you no longer get that error message:

options.add_argument("javascript.enabled", True)

You may also need to look into "waits" here to make sure the async operation on the webpage has time to load.

Update:

To enable or disable javascript:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()

#value 1 enables it , if you set to 2 it disables it     
chrome_options.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 1})

driver =webdriver.Chrome(r".\chromedriver.exe",options=chrome_options)


driver.get("https://www.google.com")

THis wont solve your issue

you can check if the javascript is enabled by typing below in your address bar:

 chrome://settings/content/javascript?search=javascript

you can see that even if its enabled the website won't load properly.

it seems they have enabled security to avoid using selenium in thier website

Previous answer:

There is no command line argument called --enable-javascript chromium project try the above javascript-harmony flag instead.

below are the full list of supported chrome flags:

https://peter.sh/experiments/chromium-command-line-switches/#login-profile

please add screenshots and other information if more help is needed

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM