通过python正则表达式提取具有非ASCII字符的单词

Question

I want to extract some text that contains non-ASCII characters. The problem is that the program considers non-ASCII as delimiters! I tried this:

regex_fmla = '(?:title=[\'"])([:/.A-z?<_&\s=>0-9;-]+)'
c1='<a href="/climate/cote-d-ivoire.html" title="Climate data: Côte d\'Ivoire">Côte d\'Ivoire</a>'
c2= '<a href="/climate/cameroon.html" title="Climate data: Cameroon">Cameroon</a>'
c_list =[c1, c2]
for c in c_list 
    print re.findall(regex_fmla , c)

The result is:

['Climate data: C']
['Climate data: Cameroon']

Notice that The first country is not correct, as the series broken at ô, it should be:

['Climate data: Côte d\'Ivoire']

I searched in StackOverflow, and I found an answer that suggests using the flag re.UNICODE, but it returns the same wrong answer!

How can I fix this?

Answer 1

I would suggest using BeautifulSoup for parsing html:

from bs4 import BeautifulSoup as bs

c1='<a href="/climate/cote-d-ivoire.html" title="Climate data: Côte d\'Ivoire">Côte d\'Ivoire</a>'
c2='<a href="/climate/cameroon.html" title="Climate data: Cameroon">Cameroon</a>'


for c in [c1, c2]:
    soup = bs(c, 'html.parser')
    print(soup.find('a')['title'])

for more links ( <a ...> ) use .findAll() method:

for c in [bightml]:
    soup = bs(c, 'html.parser')
    for a in soup.findAll('a'):
        print(a['title'])

if you need anything that has a title attribute:

for a in soup.findAll(title=True):
    print(a['title'])

Answer 2

I also would suggest BeautifulSoup , but it seems you want to know how to include those special chars, you can change your regular expression to this:

ex = 'title="(.+?)"'

and then:

c1='<a href="/climate/cote-d-ivoire.html" title="Climate data: Côte d\'Ivoire">Côte d\'Ivoire</a>'

for x in re.findall(ex, c1):
    print x

Output:

Climate data: Côte d'Ivoire

Answer 3

I suggest using beautiful soup, but if you would prefer sticking to re:

import re

regex_fmla = '(?:title=[\'"])([\w :\':/.]+)'

c1 = '<a href="/climate/cote-d-ivoire.html" title="Climate data: Côte d\'Ivoire">Côte d\'Ivoire</a>'
c2 = '<a href="/climate/cameroon.html" title="Climate data: Cameroon">Cameroon</a>'
c_list = [c1, c2]

for c in c_list:
    print(re.findall(regex_fmla, c, flags=re.UNICODE))

I believe the problem that caused the re.UNICODE not to work was explicitly defining the alphabet in your expression as [A-z0-9] . If we change that to simply [\\w] then the flag works correctly