Python：AssertionError：代理必须是映射

Question

I get this error :

Traceback (most recent call last):
  File "script.py", line 7, in <module>
    proxy = urllib2.ProxyHandler(line)
  File "/usr/lib/python2.7/urllib2.py", line 713, in __init__
    assert hasattr(proxies, 'has_key'), "proxies must be a mapping"
AssertionError: proxies must be a mapping

when I run the following script:

import urllib2  
u=open('urls.txt')
p=open('proxies.txt')
for line in p:
    proxy = urllib2.ProxyHandler(line)
    opener = urllib2.build_opener(proxy)
    urllib2.install_opener(opener)
    for url in u:
        urllib.urlopen(url).read()

u.close()
p.close()

my urls.txt file has this:

'www.google.com'
'www.facebook.com'
'www.reddit.com'

and my proxies.txt has this:

{'https': 'https://94.142.27.4:3128'}
{'http': 'http://118.97.95.174:8080'}
{'http':'http://66.62.236.15:8080'}

I found them at hidemyass.com

from the googling I have done, most people that have had this problem have their proxies formatted wrong. Is this the case here?

Answer 1

As the documentation says:

If proxies is given, it must be a dictionary mapping protocol names to URLs of proxies.

But in your code, it's just a string. In particular, it's one line out of your proxies.txt file:

p=open('proxies.txt')
for line in p:
    proxy = urllib2.ProxyHandler(line)

Looking at the file, it looks like the lines are intended to be something like the repr of a Python dictionary. And, given that all of the keys and values are string literals, that means you could use ast.literal_eval on it to recover the original dicts:

p=open('proxies.txt')
for line in p:
    d = ast.literal_eval(line)
    proxy = urllib2.ProxyHandler(d)

Of course that won't work for your sample data, because one of the lines is missing a ' character. But if you fix that, it will…

However, it would probably be better to use a format that's actually intended for data interchange. For example, JSON is just as human-readable as what you've got, and not all that different:

{"https": "https://94.142.27.4:3128"}
{"http": "http://118.97.95.174:8080"}
{"http": "http://66.62.236.15:8080"}

The advantage of using JSON is that there are plenty of tools to validate, edit, etc. JSON, and none for your custom format; the rules for what is and isn't valid are obvious, rather than something you have to guess at; and the error messages for invalid data will likely be more helpful (like "Expecting property name at line 1 column 10 (char 10)" as opposed to "unexpected EOF while parsing").

---

Note that once you solve this problem, you're going to run into another one with the URLs. After all, 'www.google.com'\\n is not what you want, it's www.google.com . So you're going to have to strip off the newline and the quotes. Again, you could use ast.literal_eval here. Or you could use JSON as an interchange format.

But really, if you're just trying to store one string per line, why not just store the strings as-is, instead of trying to store a string representation of those strings (with the extra quotes on)?

---

There are still more problems beyond that.

Even after you get rid of the excess quotes, www.google.com isn't a URL, it's just a hostname. http://www.google.com is what you want here. Unless you want https://www.google.com , or some other scheme.

You're trying to loop through 'urls.txt' once for each proxy. That's going to process all of the URLs with just the first proxy installed, and then the remainder (which is nothing, since you already did all of them) with the first two installed, and then the remainder (which is still nothing) with all three installed. Move the url loop outside of the proxy loop.

Finally, these aren't really a problem, but while we're at it… Using a with statement makes it much easier to write more robust code than using manual close calls, and it makes your code shorter and more readable to boot. Also, it's usually better to wait until you need a file before you try to open it. And variable names like u and p are just going to cause more confusion in the long run than they'll save typing in the short run.

Oh, and just calling urllib.urlopen(url).read() and not doing anything with the result won't have any effect except to waste a few seconds and a bit of network bandwidth, but I assume you already knew that, and just left out the details for the sake of simplicity.

Putting it all together, and assuming you fix the two files as described above:

import json
import urllib2  

with open('proxies.txt') as proxies:
    for line in proxies:
        proxy = json.loads(line)
        proxy_handler = urllib2.ProxyHandler(proxy)
        opener = urllib2.build_opener(proxy_handler)
        urllib2.install_opener(opener)
with open('urls.txt') as urls:
    for line in urls:
        url = line.rstrip()
        data = urllib.urlopen(url).read()
        # do something with data

---

As it turns out, you want to try all of the URLs through each proxy, not try all of them through all the proxies, or through the first and then the first two and so on.

You could do this by indenting the second with and for under the first for . But it's probably simpler to just read them all at once (and probably more efficient, although I doubt that matters):

with open('urls.txt') as f:
    urls = [line.rstrip() for line in f]
with open('proxies.txt') as proxies:
    for line in proxies:
        proxy = json.loads(line)
        proxy_handler = urllib2.ProxyHandler(proxy)
        opener = urllib2.build_opener(proxy_handler)
        urllib2.install_opener(opener)
        for url in urls:
            data = urllib.urlopen(url).read()
            # do something with data

Of course this means reading the whole list of URLs before doing any work. I doubt that will matter, but if it does, you can use the tee trick to avoid it.