Python Scrape urllib2 HTTP错误

Question

I am trying to scrape a site but my code only works if I have the site open and then refresh it. I have tried multiple things and keep coming to the following two errors: The first: ValueError: "HTTPError: HTTP Error 416: Requested Range Not Satisfiable"

urlslist = open("list_urls.txt").read()
urlslist = urlslist.split("\n")
for urlslist in urlslist:

htmltext = urllib2.urlopen("www..."+ urlslist)
data = json.load(htmltext)

I have also tried using some headers and such but get the error 'ValueError: No JSON object could be decoded':

req = urllib2.Request('https://www....)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36')

htmltext = urllib2.urlopen(req)
data = json.load(htmltext)

I am stumped, any help?

Answer 1

When you request a url, you need to include the "http(s)://" part as well. Assuming that the text file you have just contains the "name.com" part of the url (eg instead of https://www.google.com , your text file has google.com), this is the code you need:

htmltext = urllib2.urlopen("https://www." + urlslist)

If the url is the stubhub.com (as you mentioned in your comment) one, you don't need the "s." It would be this instead:

htmltext = urllib2.urlopen("http://www." + urlslist)

The json error may simply be due to the fact that there are no json files to load. You'll need to take a look at the developer panel and make sure that json format files are being brought in.