如何保持登录网站并发送cookie或http标头或我的python程序之类的东西？

Question

Ok. This is what my program (in Python) does. First it downloads the html of a page on this website, locationary.com, and then it takes all of the business pages from that page and finds all of the yellowpages.com links for those businesses. After it finds them it inserts them into the website with the webbrowser module. (If you read through the code, this will probably make a LOT more sense).

First of all, I want a way to submit the yellowpages.com link without the webbrowser module because when I do I have to make it so Firefox is always logged in to locationary.com and then I had to figure out a way to close firefox so that it doesn't get overloaded with tabs. I tried to use urllib2 and urlopen but it didn't do anything when I ran it. Now I'm starting to think that I need to send some type of cookie or http header with my request. How do you do this?

If something doesn't make sense, please ask me to clarify!

Here is my code:

from urllib import urlopen
from gzip import GzipFile
from cStringIO import StringIO
import re
import urllib
import urllib2
import webbrowser
import mechanize
import time
from difflib import SequenceMatcher
import os

def download(url):
    s = urlopen(url).read()
    if s[:2] == '\x1f\x8b': # assume it's gzipped data
        with GzipFile(mode='rb', fileobj=StringIO(s)) as ifh:
            s = ifh.read()
    return s

for t in range(0, 1):
    s = download('http://www.locationary.com/place/en/US/Arizona/Phoenix-page7/?ACTION_TOKEN=NumericAction')
    findTitle = re.compile('<title>(.*)</title>')
    getTitle = re.findall(findTitle,s)    
    findLoc = re.compile('http://www\.locationary\.com/place/en/US/.{1,50}/.{1,50}/.{1,100}\.jsp')
    findLocL = re.findall(findLoc,s)

    W, X, XA, Y, YA, Z, ZA = [], [], [], [], [], [], []

    for i in range(2, 25):
        print i

        b = download(findLocL[i])
        findYP = re.compile('http://www\.yellowpages\.com/')
        findYPL = re.findall(findYP,b)
        findTitle = re.compile('<title>(.*) \(\d{1,10}.{1,100}\)</title>')
        getTitle = re.findall(findTitle,b)
        findAddress = re.compile('<title>.{1,100}\((.*), .{4,14}, United States\)</title>')
        getAddress = re.findall(findAddress,b)
        if not findYPL:
            if not getTitle:
                print ""
            else:
                W.append(findLocL[i])

            b = download(findLocL[i])

            if not getTitle:
                print ""
            else:
                X.append(getAddress)

            b = download(findLocL[i])

            if not getTitle:    
                print ""
            else:
                Y.append(getTitle)

    sizeWXY = len(W)

    def XReplace(text, d):
        for (k, v) in d.iteritems():
            text = text.replace(k, v)  
        XA.append(text)

    def XReplace(text, d):
        for (k, v) in d.iteritems():
            text = text.replace(k, v)  
        YA.append(text)

    for d in range(0, sizeWXY):
        old = str(X[d])
        reps = {' ':'-', ',':'', '\'':'', '[':'', ']':''}
        XReplace(old, reps)
        old2 = str(Y[d])
        YReplace(old2, reps)

    count = 0

    for e in range(0, sizeWXY):
        newYPL = "http://www.yellowpages.com/" + XA[e] + "/" + YA[e] + "?order=distance"
        v = download(newYPL)
        abc = str('<h3 class="business-name fn org">\n<a href="')
        dfe = str('" class="no-tracks url "')
        findFinal = re.compile(abc + '(.*)' + dfe)
        getFinal = re.findall(findFinal, v)

        if not getFinal:
            W.remove(W[(e-count)])
            X.remove(X[(e-count)])
            count = (count+1)
        else:
            for f in range(0,1):
                Z.append(getFinal[f])

    XA = []
    for c in range(0,(len(X))):
        aGd = re.compile('(.*), .{1,50}')
        bGd = re.findall(aGd, str(X[c]))
        XA.append(bGd)

    LenZ = len(Z)

    V = []
    for i in range(0, (len(W))):
        if i == 0:
            countTwo = 0

        gda = download(Z[i-(countTwo)])
        ab = str('"street-address">\n')
        cd = str('\n</span>')
        ZAddress = re.compile(ab + '(.*)' + cd)
        ZAddress2 = re.findall(ZAddress, gda)

        for b in range(0,(len(ZAddress2))):
            if not ZAddress2[b]:
                print ""
            else:
                V.append(str(ZAddress2[b]))
                a = str(W[i-(countTwo)])
                n = str(Z[i-(countTwo)])
                c = str(XA[i])
                d = str(V[i])
                m = SequenceMatcher(None, c, d)

                if m.ratio() < 0.50:
                    Z.remove(Z[i-(countTwo)])
                    W.remove(W[i-(countTwo)])
                    countTwo = (countTwo+1)

    def ZReplace(text3, dic3):
        for p, q in dic3.iteritems():
            text3 = text3.replace(p, q)  
        ZA.append(text3)

    for y in range(0,len(Z)):
        old3 = str(Z[y])
        reps2 = {':':'%3A', '/':'%2F', '?':'%3F', '=':'%3D'}
        ZReplace(old3, reps2)

    for z in range(0,len(ZA)):
        findPID = re.compile('\d{5,20}')
        getPID = re.findall(findPID,str(W[z]))
        newPID = re.sub("\D", "", str(getPID))
        finalURL = "http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=" + str(newPID) + "&xxx_c_1_f_987=" + str(ZA[z])
        webbrowser.open(finalURL)
        time.sleep(5)

    os.system("taskkill /F /IM firefox.exe")

Answer 1

You might try pycurl which will allow you to interact with web pages without using a browser.

pycurl allows cookie storage (and reuse), form submission (including login), as well as page navigation from Python. Your code above already seems to retrieve the resources you want via urllib. You can easily adapt your code by following the pycurl tutorials at the pycurl site.

Regarding your comment about the need to handle javascript, please elaborate about the context of that.

A different but related post here at Stackoverflow addresses a javascript question with pycurl as well as showing code to initiate a pycurl connection.