im trying to get proxies using regex python out of a web page

Question

import urllib.request
import re
page = urllib.request.urlopen("http://www.samair.ru/proxy/ip-address-01.htm").read()
re.findall('\d+\.\d+\.\d+\.\d+', page)

i dont understand why it says:

File "C:\\Python33\\lib\\re.py", line 201, in findall return _compile(pattern, flags).findall(string) TypeError: can't use a string pattern on a bytes-like object

Answer 1

import urllib
import re
page = urllib.urlopen("http://www.samair.ru/proxy/ip-address-01.htm").read()
print re.findall('\d+\.\d+\.\d+\.\d+', page)

Worked and gave me the result:

['056.249.66.50', '100.44.124.8', '103.31.250.115', ...

Edit

• This works for python2.7

Answer 2

The result of reading the file-like object returned by urllib.request.urlopen is a bytes object. You can either decode it into a unicode string and use a unicode regex:

>>> re.findall('\d+\.\d+\.\d+\.\d+', page.decode('utf-8'))
['056.249.66.50', '100.44.124.8', '103.31.250.115', '105.236.180.243', '105.236.21.213', '108.171.162.172', '109.207.61.143', '109.207.61.197', '109.207.61.202', '109.226.199.129', '109.232.112.109', '109.236.220.98', '110.196.42.33', '110.74.197.141', '110.77.183.64', '110.77.199.111', '110.77.200.248', '110.77.219.154', '110.77.219.2', '110.77.221.208']

... or use a bytes regex:

>>> re.findall(b'\d+\.\d+\.\d+\.\d+', page)
[b'056.249.66.50', b'100.44.124.8', b'103.31.250.115', b'105.236.180.243', b'105.236.21.213', b'108.171.162.172', b'109.207.61.143', b'109.207.61.197', b'109.207.61.202', b'109.226.199.129', b'109.232.112.109', b'109.236.220.98', b'110.196.42.33', b'110.74.197.141', b'110.77.183.64', b'110.77.199.111', b'110.77.200.248', b'110.77.219.154', b'110.77.219.2', b'110.77.221.208']

Depending on which datatype you prefer to work with.