Python：requests.get 获取错误的 html 文件

Question

I'm trying to scrape data from https://essentials.swissdox.ch , which only works with VPN. So what I did is, I generated a URL with my query parameters and tried to get the correspondent html file. The problem is, that although the link works, Python gives me the html file of the starting page of https://essentials.swissdox.ch . I really appreciate any help!

Example: I want the html file of the following url: https://essentials.swissdox.ch/View/log/index.jsp#&search=true&filter_de=la&sortorder=pubDateTime%20desc&formdata=%5B%7B%22name%22%3A%22SEARCH_mltid%22%2C%22value%22%3A%22%22%7D%2C%7B%22name%22%3A%22SEARCH_sc%22%2C%22value%22%3A%22swissdox%22%7D%2C%7B%22name%22%3A%22SEARCH_query%22%2C%22value%22%3A%22lissabon%22%7D%2C%7B%22name%22%3A%22filter_de%22%2C%22value%22%3A%22de%22%7D%2C%7B%22name%22%3A%22SEARCH_exact%22%2C%22value%22%3A%22true%22%7D%2C%7B%22name%22%3A%22dateDropdown%22%2C%22value%22%3A%22-1%22%7D%2C%7B%22name%22%3A%22SEARCH_pubDate_lower%22%2C%22value%22%3A%222020-02-04%22%7D%2C%7B%22name%22%3A%22SEARCH_pubDate_upper%22%2C%22value%22%3A%222020-02-04%22%7D%2C%7B%22name%22%3A%22SEARCH_tiall%22%2C%22value%22%3A%22%22%7D%2C%7B%22name%22%3A%22SEARCH_source%22%2C%22value%22%3A%22%22%7D%2C%7B%22name%22%3A%22SEARCH_author%22%2C%22value%22%3A%22% 22%7D%5D

Instead I get the html file of this page: https://essentials.swissdox.ch/View/log/index.jsp?reset=true

Here is what I have so far:

#Set keywords for URL
keyword_queries = ['lissabon']
startdate = "2007-01-01"
enddate = "2007-01-01"

#Encode  and hit URL
for keyword in keyword_queries:
    html_keyword= urllib.parse.quote_plus(keyword)
    URL = "https://essentials.swissdox.ch/View/log/index.jsp#&search=true&sortorder=pubDateTime%20desc&formdata=%5B%7B%22name%22%3A%22SEARCH_mltid%22%2C%22value%22%3A%22%22%7D%2C%7B%22name%22%3A%22SEARCH_sc%22%2C%22value%22%3A%22swissdox%22%7D%2C%7B%22name%22%3A%22SEARCH_query%22%2C%22value%22%3A%22" + html_keyword + "%22%7D%2C%7B%22name%22%3A%22SEARCH_exact%22%2C%22value%22%3A%22true%22%7D%2C%7B%22name%22%3A%22dateDropdown%22%2C%22value%22%3A%22-1%22%7D%2C%7B%22name%22%3A%22SEARCH_pubDate_lower%22%2C%22value%22%3A%22" + startdate + "%22%7D%2C%7B%22name%22%3A%22SEARCH_pubDate_upper%22%2C%22value%22%3A%22" + enddate + "%22%7D%2C%7B%22name%22%3A%22SEARCH_tiall%22%2C%22value%22%3A%22%22%7D%2C%7B%22name%22%3A%22SEARCH_source%22%2C%22value%22%3A%22%22%7D%2C%7B%22name%22%3A%22SEARCH_author%22%2C%22value%22%3A%22%22%7D%5D"
    weburl  = urllib.request.urlopen(URL)

    
    #Hit the url
    ua = UserAgent()
    page = requests.get(URL, {"User-Agent": ua.random})
    soup = BeautifulSoup(page.content, "html.parser")
    results = soup.find('div', class_='documentlist')
    print(page.content)

Answer 1

It looks like you used '#' instead of '?' in your url. Usually '?' will be used to start the query parameters, which are specified with '=' between key-value pairs.

Using '#' means to jump to a specific section in the page, in this case https://essentials.swissdox.ch/View/log/index.jsp which is what you are getting as response. Changing '#' to '?' seems to throw an error about invalid characters on the original URL. Make sure you use valid characters while percent encoding the query parameters.

Wiki - URL Syntax