python urllib2超時

Question

好的，我已經在google中和這里的stackoverflow中搜索了此答案，幾個小時后卻沒有看到執行此操作的腳本的正確答案。

在這里，我粘貼了4個假定的python工作腳本示例，以為不存在的url設置默認超時，並使用套接字和/或超時參數設置超時。

沒有人可以使用，因此永遠不會觸發超時。

有任何想法嗎？

第一個例子：

import urllib2

try:                
    header_s = {"User-Agent":"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"}

    req = urllib2.Request("http://www.nonexistantdomainurl.com/notexist.php",headers = header_s)


    print urllib2.urlopen(req, None, 5.0).read()

except urllib2.URLError, e:
    print "Url Error: %r" % e

except Exception,e:
  print "Fallo de tipo ",e

else: 
    print "all ok!"

第二個例子：

import urllib2

try:
    response = urllib2.urlopen("http://www.nonexistantdomainurl.com/notexist.php", None, 2.5)
except urllib2.URLError, e:
    print "Oops, timed out?"

閾值示例：

from urllib2 import Request, urlopen, URLError, HTTPError
import base64


req = Request('http://www.nonexistantdomainurl.com/notexist.php')

try:
    response = urlopen(req,timeout=5.0)   

except HTTPError, e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
except URLError, e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason

第四個例子：

import urllib2
import socket


socket.setdefaulttimeout(5)

try:
    response = urllib2.urlopen("http://www.faluquito.com/equipo.php",timeout=5.0).read()   


except urllib2.URLError, e:
    print "Url Error: %r" % e

Answer 1

>>> import urllib2
>>> import time
>>> import contextlib
>>>
>>> def timeit():
...   s = time.time()
...   try:
...     yield
...   except urllib2.URLError:
...     pass
...   print 'took %.3f secs' % (time.time() - s)
...
>>> timeit = contextlib.contextmanager(timeit)
>>> with timeit():
...   r = urllib2.urlopen('http://loc:8080', None, 2)
...
took 2.002 secs
>>> with timeit():
...   r = urllib2.urlopen('http://loc:8080', None, 5)
...
took 5.003 secs

Answer 2

如果您的計算機具有unix程序dig，則可以識別出不存在的url，如下所示：

import logging
import subprocess
import shlex

logging.basicConfig(level = logging.DEBUG,
                    format = '%(asctime)s %(module)s %(levelname)s: %(message)s',
                    datefmt = '%M:%S')
logger = logging.getLogger(__name__)

urls = ['http://1.2.3.4',
       "http://www.nonexistantdomainurl.com/notexist.php",
       "http://www.faluquito.com/equipo.php",
        'google.com']

nonexistent = ['63.251.179.13', '8.15.7.117']
for url in urls:
    logger.info('Trying {u}'.format(u=url))

    proc = subprocess.Popen(shlex.split(
        'dig +short +time=1 +retry=0 {u}'.format(u = url)),
                            stdout = subprocess.PIPE, stderr = subprocess.PIPE)
    out, err = proc.communicate()
    out = out.splitlines()
    logger.info(out)
    if any(addr in nonexistent for addr in out):
        logger.info('nonexistent\n')
    else:
        logger.info('success\n')

在我的機器上，這產生了：

00:57 test INFO: Trying http://1.2.3.4
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying http://www.nonexistantdomainurl.com/notexist.php
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying http://www.faluquito.com/equipo.php
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying google.com
00:58 test INFO: ['72.14.204.113', '72.14.204.100', '72.14.204.138', '72.14.204.102', '72.14.204.101']
00:58 test INFO: success

請注意，對於不存在的網址，dig將返回['63.251.179.13', '8.15.7.117'] 。

我相信我的ISP將不存在的地址更改為63.251.179.13或8.15.7.117。 您的ISP可能會做一些不同的事情。 在這種情況下，您可能需要將nonexistent內容更改為其他內容。