如何修改此脚本以检查HTTP状态（404、200）

Question

I am currently using the following script to load a list of URLs then check the source of each for a list of error strings. If no error string is found in the source, the URL is considered valid and written to a text file.

How can I modify this script to check for HTTP status instead? If a URL returns a 404 it would be ignored, if it returns 200 the URL would be written to the text file. Any help would be much appreciated.

import urllib2
import sys

error_strings = ['invalid product number', 'specification not available. please contact   customer services.']

def check_link(url):
if not url:
    return False
f = urllib2.urlopen(url)    
html = f.read()
result = False
if html:
    result = True
    html = html.lower()
    for s in error_strings:
        if s in html:
            result = False
            break
return result


if __name__ == '__main__':
if len(sys.argv) == 1:
    print 'Usage: %s <file_containing_urls>' % sys.argv[0]
else:
    output = open('valid_links.txt', 'w+')
    for url in open(sys.argv[1]):
        if(check_link(url.strip())):
            output.write('%s\n' % url.strip());
    output.flush()
    output.close()

Answer 1

You can alter your call to urlopen slightly:

>>> try:
...     f = urllib2.urlopen(url)
... except urllib2.HTTPError, e:
...     print e.code
...
404

Utilizing the e.code , you can check if it 404s on you. If you don't hit the except block, you can utilize f as you currently do.

Answer 2

urlib2.urlopen gives back a file-like object with some other methods, one of which: getcode() is what you're looking for, just add a line:

if f.getcode() != 200:
    return False

In the relevant place

Answer 3

Try this. You can use this

 def check_link(url):
        if not url:
            return False
        code = None
        try:
            f = urllib2.urlopen(url)
            code = f.getCode()
        except urllib2.HTTPError, e:
            code = e.code
        result = True
        if code != 200:
            result = False
        return result

Alternatively, if you just need to maintain a list of invalid code strings and check against those, it will be something like below.

def check_link(url):
    if not url:
        return False
    code = None
    try:
        f = urllib2.urlopen(url)
        code = f.getCode()
    except urllib2.HTTPError, e:
        code = e.code

    result = True
    if code in invalid_code_strings:
         result = False

    return result