在Python中，如何从动态加载的网站请求特定数据？

Question

I want to load pages from PeoplePerHour.com into python to run some data analysis, but it keeps getting data from a page I didn't ask for, I think it must go to the main page and then refreshes somehow into the page I ask for.

For example: I want to pull the prices from all users at http://www.peopleperhour.com/freelance/data+analyst , and the data spans over multiple pages.

Say I want to request page 2, http://www.peopleperhour.com/freelance/data+analyst#page=2 . If I go here in a browser, it works fine and pulls up page 2, but I think it pulls up page one first and then "refreshes" into page 2 (I think). If I access this in python, it loads the HTML from the first page, and never sees page 2.

Here's my code:

import requests
from pattern import web
import re
import pandas as pd

def list_of_prices(url):
    html = requests.get(url).text
    dom = web.DOM(html)
    list = []
    for person in dom('.freelancer-list-item .medium.price-tag'):
        currency = person('sup')
        amount = person('span')
        list.append([currency[0].content if currency else 'na', amount[0].content if amount else 'na'])
    return list

list_of_prices('http://www.peopleperhour.com/freelance/data+analyst#page=2')

No matter what, this returns the prices from page 1.

What is going on that I'm just not seeing?

Answer 1

If I understand correctly, you want to iterate through the pages. If that's the case, I believe the problem is with your URL.

Here's the URL you gave: http://www.peopleperhour.com/freelance/data+analyst#page=2

The problem is, "page" is not a bookmark on that page. When you use the #page=2, it tells the browser to go down to the same page for a bookmark called "page=2".

Here's the URL for the Next button in that site: http://www.peopleperhour.com/freelance/data+analyst?sort=most-relevant&page=2

You'll see it says "&page=2" which means something else. In their code "page" is a variable being passed via the url, with a value of 2. You use the "&" if there are more than one of these variables. Also, you are missing a "?" symbol. If you're passing variables via the URL, you have to put a ? followed by the name=value pairs for your variables.

So, easy fix, change your url to this:

http://www.peopleperhour.com/freelance/data+analyst?page=2

That's in comparison to your old url:

http://www.peopleperhour.com/freelance/data+analyst#page=2

As a quick test, copy/paste the corrected url on your web browser. You will see it now is on page 2.

Answer 2

Getting dynamic content (those generated by client-side code) is always very tricky. There is no easy solution to this, but if you really want to dig into it, I recommend PyV8 , a JavaScript engine in Python .

Answer 3

Error in pattern when using pattern3 in python 3.6

Please click on the above Hyperlink to open the Image What is the alternative to executing the same code under python3.6 environment because due to this I have to install the pattern3, the pattern is not supported by the python 3.6 Thanks!