Python re.findall挂在某些网站上

Question

I´ve got a python script to loop through a list of websites/domains to scrap phones and e-mails from my clients websites, 99% of websites scraps are OK and works. Some websites just hangs and cant even force break operation, like it is on an insane loop. Below an example. Anyone could help me improve or fix this?

import requests,re

try:   
    r = requests.Session()
    f = r.get('http://www.poffoconsultoria.com.br', verify=False, allow_redirects=False,timeout=(5,5) )
    s = f.text                  
    tels = set(re.findall(r"\s?\(?0?[1-9][1-9]\)?[-\.\s][2-5]\d{3}\.?-?\s?\d{4}",s))
    emails = set(re.findall(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}",s))
    print(tels)
    print(emails)
except Exception as e:
    print(e)

Answer 1

You should remove the \\s? from the first regex (you do not really need a whitespace at the start of the match), or replace with (?<!\\S) if you want to only match after a whitespace or start of string.

The real problem is with the second regex where . resides in a character class that is quantified with + . The \\. that follows it also matches a . and that makes it a problem when no matching text appears in the string. This is catastrophic backtracking .

Since the matches you expect are whole words, I suggest enhancing the pattern by 1) adding word boundaries, 2) making all adjoining subpatterns match different types of chars.

Use

r'\b[A-Za-z0-9._%+-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,4}\b'

to match emails.

See the (?:[A-Za-z0-9-]+\\.)+ part: it matches one or more repetitions of 1 or more alphanumeric/hyphen chars followed with a dot, and there is no \\. after this pattern, there is an alpha character class, so there should be no problem like the one present before.

Answer 2

So. I got the website data fine in Python27 using >>> string = requests.get('http://www.poffoconsultoria.com.br').text

I then took the length of the string and it was >>> len(strings) 474038 That's a really high value.

So for problems like these when one sees regex take such a long time (really, after getting the length of the page), you should visit the page in your browser and inspect the page source

When I inspected the page in my browser I found these:

The second regex [A-Za-z0-9._%+-]+ will definitely hang (really, take a long time) because it isn't quantifiable and has to search through those ginormous portions.

You either need to chunk the page or limit your regex. Or you could write a function that discards dictionary data if you suspect that what you need to return won't appear inside of them; basically though, those huge dictionaries above are causing the regex you posted to take a long time.

Answer 3

使用有效的电子邮件

(?i)(?:("[^"\\]*(?:\\.[^"\\]*)*"@)|((?:[0-9a-z](?:\.(?!\.)|[-!#$%&'*+/=?^`{}|~\w])*)?[0-9a-z]@))(?:(\[(?:\d{1,3}\.){3}\d{1,3}\])|((?:[0-9a-z][-\w]*[0-9a-z]*\.)+[a-z0-9][-a-z0-9]{0,22}[a-z0-9]))