Python REGEX 内循环

Question

I am trying to parse a list of document with REGEX (could not user BeautifulSoup). I am now able to loop over each txt document inside my folder but I have now to parse them. I have been using Python for few days only and I am a bit confused now.

I want to generate a dictionary with <DOCNO> as an ID and <TEXT> as the value.

Example of a file:

<DOC>
<DOCNO> 443 </DOCNO>
<TEXT>Hello Word</TEXT>
</DOC>
<DOC>
<DOCNO> 3745 </DOCNO>
<TEXT> Hola amigo </TEXT>
</DOC>

My code so far:

    path = "data"

    for filename in os.listdir(path):
        print(filename)
        file = open(path + "/" + filename)
        page = file.read()
        page = page.replace('  ', ' ')

        //stuck here 
        doc_regex = re.compile("<DOC>.*?</DOC>", re.DOTALL)
        docno_regex = re.compile("<DOCNO>.*?</DOCNO>")
        text_regex = re.compile("<TEXT>.*?</TEXT>", re.DOTALL)

Answer 1

The best practice is to compile the RegEx once at module level (not in a for loop). For instance, you can write:

import re

doc_regex = re.compile("<DOC>(.*?)</DOC>", re.DOTALL)
docno_regex = re.compile("<DOCNO>(.*?)</DOCNO>")
text_regex = re.compile("<TEXT>(.*?)</TEXT>", re.DOTALL)

The first time the mode is loaded, the regex are compiled.

Note: in your RegEx, you need to use a group "(...)" to retrieve the content of each tag.

Several pitfalls:

• You ought to use os.path.join to calculate the fullpath of a file (on Windows, the path delimiter is \\ , not / ). os.path.join is doing that for you.

• You ought to use a with statement to open a file and specify the file encoding. In text mode, if encoding is not specified the encoding used is platform dependent.

So, your loop can be turned into:

path = "data"

for filename in os.listdir(path):
    fullpath = os.path.join(path, filename)
    print(filename)

    with open(fullpath, mode="r", encoding="utf-8") as fd:
        page = fd.read()

To parse your data, you can use re.findall . There are other ways, also…

You can use doc_regex to find each <DOC>...</DOC> and then docno_regex and text_regex to find the docno and the text .

In you loop, you can do that way:

    for doc_content in doc_regex.findall(page):
        docno = docno_regex.findall(doc_content)[0].strip()
        text = text_regex.findall(doc_content)[0].strip()
        print(docno, text)

To store each entry in a dictionary, you can define a dict , like that:

result = {}
for doc_content in doc_regex.findall(page):
    docno = docno_regex.findall(doc_content)[0].strip()
    text = text_regex.findall(doc_content)[0].strip()
    result[docno] = text

You get:

{'3745': 'Hola amigo', '443': 'Hello Word'}

Answer 2

Why can't BeautifulSoup be used? How about this?

from simplified_scrapy.simplified_doc import SimplifiedDoc
html ='''
<DOC>
<DOCNO> 443 </DOCNO>
<TEXT>Hello Word</TEXT>
</DOC>
<DOC>
<DOCNO> 3745 </DOCNO>
<TEXT> Hola amigo </TEXT>
</DOC>
'''
xml = SimplifiedDoc(html)
docs = xml.selects('DOC')
dic = {}
for doc in docs:
  dic[doc.DOCNO.text]=doc.TEXT.text
print (dic)

Result:

{'443': 'Hello Word', '3745': 'Hola amigo'}