如何在python中使用正则表达式从许多文档中删除相似的页脚

Question

I am trying to clean like 18000 documents to train a word2vec classifier. A sample document is like :

From: shou@logos.asd.sgi.com (Tom Shou)
Subject: Ford Explorer 4WD - do I need performance axle?

We're considering getting a Ford Explorer XLT with 4WD and we have the
following questions (All we would do is go skiing -- no off-roading):

1. With 4WD, do we need the "performance axle" - (limited slip axle).
Its purpose is to allow the tires to act independently when the tires
are on different terrain. 

2. Do we need the all-terrain tires (P235/75X15) or will the
all-season (P225/70X15) be good enough for us at Lake Tahoe?


Thanks,


Tom

-- *(there maybe --- also)*


===========================================================================*(only one of the two boundaries made with ===== may be present(like sometimes only the top boundary is present)  in some cases and may differ in length)* 

        Tom Shou            Silicon Graphics
    shou@asd.sgi.com        2011 N. Shoreline Blvd. 
    415-390-5362            MS 8U-815 
    415-962-0494 (fax)      Mountain View, CA 94043

===========================================================================

So i need to remove the footer part. I am able to remove the From and Subject lines (the first two lines from the document with regex) . But i am not able to remove this part:

-- 


    ===========================================================================

            Tom Shou            Silicon Graphics
        shou@asd.sgi.com        2011 N. Shoreline Blvd. 
        415-390-5362            MS 8U-815 
        415-962-0494 (fax)      Mountain View, CA 94043

    ===========================================================================

Now some footers are just having two dashes or three dashes like:

-- 


            Tom Shou            Silicon Graphics
        shou@asd.sgi.com        2011 N. Shoreline Blvd. 
        415-390-5362            MS 8U-815 
        415-962-0494 (fax)      Mountain View, CA 94043

or

  --- 


                Tom Shou            Silicon Graphics
            shou@asd.sgi.com        2011 N. Shoreline Blvd. 
            415-390-5362            MS 8U-815 
            415-962-0494 (fax)      Mountain View, CA 94043

or sometimes it can have _ or + instead of ==== like :

   --(this maybe --- or may not exist at all but then the below ______ lines will be there) 


 ________________________________________________________________________(this can be + also)

                Tom Shou            Silicon Graphics
            shou@asd.sgi.com        2011 N. Shoreline Blvd. 
            415-390-5362            MS 8U-815 
            415-962-0494 (fax)      Mountain View, CA 94043

 _________________________________________________________________________

I am not very good at regex but i tried to remove with a wrong regex like ((_|-|=|\\+){2,})(.|\\n)* but i didnt consider that there are overlaps of -- within the content so it removed the content also which i dont want. Like in the content there is a line(4th line) with 2 dashes All we would do is go skiing -- no off-roading): . So it removed everything including and after -- which i dont want. I only want to remove the footer.

So i wanted to know what should my regex be like or what method should i use to clean the footer even if -- or --- is not always present but (there is a box made with either ______ or ++++++ or =========) or vice versa.

Please help thanks in advance

Answer 1

You may use

re.sub(r'(?ms)^[ \t]*([-_=+])\1+.*', '', text)

See the regex demo

Details

• (?ms) - re.M ( ^ will match the start of a line) and re.DOTALL ( . will match any chars) are enabled
• ^ - start of a line
• [ \\t]* - zero or more horizontal whitespaces (you may also use [^\\S\\r\\n]* for that)
• ([-_=+]) - Group 1: a - , _ , = , or +
• \\1+ - the same char as captured into Group 1, one or more times
• .* - the rest of the string.