Python请求提供与Internet Explorer不同的页面文本

Question

Looking at my stackoverflow user profile page: https://stackoverflow.com/users/2683104/roberto

The site indicates I have been a member for 316 days (screenshots at end of post). If I view source in my browser (IE11), I can see this data comes from a days-visited class.

But if I look for this same days-visited information using Python Requests, the data does not appear anywhere. Why?

from requests import Session
from BeautifulSoup import BeautifulSoup

s = Session()

url = 'https://stackoverflow.com/users/2683104/roberto'
page = s.get(url)
soup = BeautifulSoup(page.text)
print soup.prettify() #server response, prettified

# following returns error
# AttributeError: 'NoneType' object has no attribute 'getText'
#days_visited = soup.find('span', attrs={'id':'days-visited'}).getText()

s.close()

screenshot

view source

python Requests

Answer 1

That field is not visible to your script (or other users). If you want to scrap that piece of information, you will need to have your script login and store appropriate cookies.

This is what is seen by users that aren't you:

And the code block they see:

 <tbody>
            <tr>
                <th>visits</th>
                <td>member for</td>

                <td class="cool" title="2013-08-14 15:38:01Z">11 months</td>
            </tr>
            <tr>
                <th></th>
                <td>seen</td>

                <td class="supernova" title="2014-08-08 05:26:50Z">
                    <span title="2014-08-08 05:26:50Z" class="relativetime">6 mins ago</span>
                </td>
            </tr>
        </tbody>

Normally, I'd recommend against scraping Stack Overflow for data and use the API instead, but this particular piece of information isn't returned as part of the User object.

Answer 2

As the comments said, 'days-visited' only shows when you are logged-in. And it can be seen only by the member himself.

You may find you cookies in your browser and use cookies in you request.

http://docs.python-requests.org/en/latest/user/quickstart/#cookies