urllib发出多个POST请求

Question

I have a website that I'm trying to pull data from, but to get to the data I need to go through 2 pages: a login screen and a second screen where I select the data to be read. My code looks like this:

    import urllib
    from bs4 import BeautifulSoup

    url = 'http://website.com'
    values = {'userName' : 'tom',
              'Login' : 'submit'}
    data = urllib.parse.urlencode(values).encode('ascii')
    req = urllib.request.Request(url, data)
    page = urllib.request.urlopen(req)
    soup = BeautifulSoup(page,'html.parser')
    print(soup.text)

My question is how I would submit a second POST request after the login request in order to get to the data that im looking for?

Answer 1

Generally, it always depends on how they authenticate the user and how they store this session, php, token-based, google authentication. Without knowing all of this information its hard to know. A common way of getting around this bloat is by using a headless web browser. A browser that can be controlled via code. Allowing you to click on the page as you would do normally!

I recommend seleniumhq for python! http://www.seleniumhq.org/