地理定位网站的 Python 网页抓取

Question

I'm trying to web scrape a website that requests users to share their location so it adapts the contents to the clients' country.

To do so, when I visit the site with Firefox a browser's pop up is displayed asking for that permission.

When scraping with Python I open the website with urllib. Is there any way I can post or send my coordinates so I get the correct geolocated website?

Thanks.

BR,

d.

Answer 1

Manually sumbit your data by browsing the website. Then check your post requests in your browser's developer tools. Store that information in a dictionary like shown below with data .

data = { "text": text }

Then use that to make a post request. req=urllib2.Request(url,urllib.urlencode(data), headers)

And then get your response.

response = urllib2.urlopen(req).read()