使用Python cookie获取HTML源代码

Question

    import urllib

    #my url here stored as url

    htmlfile = urllib.urlopen(url)
    htmltext = htmlfile.read()
    print(htmltext)

I'm trying to get source code from a url

I get source code but it is from a different page saying two things; please enable cookies and this domain has banned your access based on your browser's signature

Is there any way that anyone knows of to get the source code when the browser knows your not actually on the page?

Answer 1

You may have to set an url opener

def createOpener(self):
        handlers = []                                                       
        cj = MyCookieJar();
        cj.set_policy(cookielib.DefaultCookiePolicy(rfc2965=True))
        cjhdr = urllib2.HTTPCookieProcessor(cj)
        handlers.append(cjhdr)
        opener = urllib2.build_opener(*handlers)
        opener.addheaders = [('User-Agent', self.getUserAgent()),
                                  ('Host', 'google.com')]
        return opener

where the cookie jar is

class MyCookieJar(cookielib.CookieJar):
    def _cookie_from_cookie_tuple(self, tup, request):
        name, value, standard, rest = tup
        version = standard.get('version', None)
        if version is not None:
            version = version.replace('"', '')
            standard["version"] = version
        return cookielib.CookieJar._cookie_from_cookie_tuple(self, tup, request)

At this point you create the opener and fetch the data reading the url handler like:

def fetchURL(self, url, data=None, headers={}):
        request = urllib2.Request(url, data, headers)
        self.opener = self.createOpener()
        urlHandle = self.opener.open(request)
        return urlHandle.read()

It's a good idea to have a User-Agent list and read from it:

with open(ffpath) as f:
    USER_AGENTS_LIST = f.read().splitlines()

and get a random one from it

index = random.randint(0,len(USER_AGENTS_LIST)-1)
uA=USER_AGENTS_LIST[index]

To have a list of user agent take a look at here .

This is just to have and idea to do this without any external framework.