在网页中抓取一些Javascript代码

Question

The page I am trying to crawl has includes javascript code. (Possibly using AJAX?) When I crawl the page based on the html code, it can't get the javascript part. How can I do that?

I think I need some libraries in python which can crawl the javascript code including html codes.

Please give me some advice.

Below is the page link: view-source: http://www.bobaedream.co.kr/mycar/popup/mycarChart_4.php?zone=C&cno=652691&tbl=cyber

Answer 1

I recommend two ways.

First, request ajax url directly and parse HTML.

import requests
url = "http://www.bobaedream.co.kr/mycar/proc/mycar_regist_option.php"
data = {'param': 'ALL'}
response = requests.post(url, data=data)
# parse
...

Second, use web driver , like geckodriver, phantomjs and so on, using selenium library.

That library make virtual browser, run javascript and then render the DOM made by javascript.

This is public documents about selenium