使用BeautifulSoup从XML文档中提取Unicode文本

Question

I have this code:

for fileid in wordlist.fileids()[4:5]:
    url = open(fileid, 'r').read()
    soup = BeautifulSoup(url)
    find_all = soup.find_all("speech", soup)
    soup_sub = re.sub("<.+?>", "", str(find_all))
    print fileid
    print soup_sub

from local xml files it gets a certain ellement. Then it subs the xml code out of it and prints a list. A snipset of that list is down here. You can see that there is allot of unicode in it. How can i get this unicode out of that list?

<p>\nIk heet de minister van Sociale Zaken en Werkgelegenheid van harte welkom. Er hebben zich vijf sprekers voor dit VAO aangemeld.\u200a\n, \nVoorzitter. Ik wil drie moties indienen. Dit wordt topsport voor mij.\u200a\n\nMotie\nDe Kamer,\u200a\ngehoord de beraadslaging,\u200a\noverwegende dat bedrijfsongevallen wel bij de inspectie gemeld moeten worden en beroepsziekten niet;\u200a\noverwegende dat door registratie van beroepsziekten optimaal preventief beleid gevoerd kan worden;\u200a\</p>

Answer 1

First of all, if you are parsing XML with BeautifulSoup, do pick the right parser for the job (and have lxml installed). You can pass an open file object to BeautifulSoup, no need to read it all into memory before parsing:

with open(fileid, 'r') as xml_file:
    soup = BeautifulSoup(xml_file, 'xml')

Next, don't use str(find_all) ; that turns all your element objects into a single (byte) string, and you won't be able to access the original Unicode text contents anymore.

Use the Element.get_text() method to extract the text from each element:

speech_elements = soup.find_all("speech")
speech_text = [elem.get_text() for elem in speech_elements]

This'll ensure that you still get the full unicode contents, not some str() conversion; you now have a list with unicode objects per <speech> element found.