使用 python 请求库登录网页，状态码 200，解析失败且未经授权的页面仍然未经授权

Question

I'm attempting to build my first web scraper, which is aiming to scrape some data tables from a website and use them to populate pandas dataframes. The website requires a login. The webiste is called spotrac.com .

I'm running into a couple of issues. The first is that when I run the beautifulsoup post method I get the following error: "InvalidURL: Failed to parse: <Response [200]>". This error occurs when I run the following code:

import pandas as pd
import requests, lxml.html
from bs4 import BeautifulSoup

with requests.session() as s:    
    login = s.get('http://www.spotrac.com/signin/submit/')
    form = {'redirect' : '',
            'email' : 'my_email@gmail.com',
            'password' : 'my_password',
            }    

    p = s.post(login, data=form)

    r1 = s.get('http://www.spotrac.com/nfl/rankings/2019/base/quarterback/')
    soup1 = BeautifulSoup(r1.content,'lxml')
    table1 = soup1.find_all('table')[0]
    df1 = pd.read_html(str(table1))
    
    r2 = s.get('http://www.spotrac.com/nfl/rankings/2018/base/quarterback/')
    soup2 = BeautifulSoup(r2.content,'lxml')
    table2 = soup2.find_all('table')[0]
    df2 = pd.read_html(str(table2))
    
    print(r1.status_code) # 200
    print(r2.status_code) # 200

To allow my code to continue running, I put the post line in a try block, as follows:

with requests.session() as s:    
    try:
        p = s.post(login, data=form)
    except:
        pass

This does allow my code to continue running, and when I print out the status code, it shows 200. I believe this confirms that I am logged in?

The issue I run into after this is that when I go to a webpage that is unauthorized, the datatables aren't populating as if I am signed in. In my code, you'll see r1 and r2. When I go through the process of getting the datatable from r1, I have no issue. I believe that is because it is publicly available . When I try the same process for r2, I get the following error: "IndexError: list index out of range". This error occurs because the webpage does not load the data table, as the data table is only available to premium customers who have logged in. I know this is the case, because if you go to the r1 webpage ( http://www.spotrac.com/nfl/rankings/2019/base/quarterback/ ), you'll see a datatable without logging in. If you attempt to go to the r2 webpage ( http://www.spotrac.com/nfl/rankings/2018/base/quarterback/ ), you will not see a datatable. Instead, you get redirected to a login page.

Any help you can provide would be greatly appreciated. I really don't know how to proceed at this point.

Thank you, David

Answer 1

This was actually a very silly mistake. The issue is that I was passing the following "login" variable to the post method, rather than the login url:

login = s.get('http://www.spotrac.com/signin/submit/')
p = s.post(login, data=form)

I still wanted to run the get method, so I just assigned the url to a separate variable. Simple fix:

login = 'http://www.spotrac.com/signin/submit/'
r = s.get(login)
p = s.post(login, data=form)