如何使用 python 从大文件中删除特定行？

Question

For work I have to process an XML-based file: place it in another directory, change extension (from.xaf to.xml), read some info from lines and remove two specific lines from the file (after it has been replaced.
I got everything working except for the last part. I need to remove two specific lines (or two specific parts from a line if the content of the xml is written in one line). That should not be a problem, there are already many posts about it on stackoverflow. The solution often given is to read it from the source file and copy it line by line (except for the line that needs to be deleted) for the whole file.
The problem is that the files i need to process are very big (anywhere from 100.000 to >5.000.000 lines), and there are a lot of files to process so doing this method takes a long time.
Is there a way to copy the file and edit the content directly, instead of copying the file line by line? The parts that need to be deleted are always somewhere in the top 20 lines.

What I tried was copying the files from the source to the destination, and then opening the source file again to read the first 20 lines and copying those to the destination file. However, this meant the whole destination file was overwritten (so anything after those 20 lines was gone).

Does anyone has an idea on how to handle this? Many thanks

example part:\

   \<companyIdent>XXXXX\</companyIdent>\
\<companyName>Company1\</companyName>\
\<taxRegistrationCountry>NL\</taxRegistrationCountry>\
\<taxRegIdent>123456789\</taxRegIdent>\
\<streetAddress>
    \<streetname>Address1\</streetname>\
    \<city>CITY\</city>\
    \<postalCode>1234AB\</postalCode>\
    \<country>NL\</country>\</streetAddress>\
\<customersSuppliers>
   \<customerSupplier>
      \<custSupID>C0001\</custSupID>\

I want to remove<streetAddress> and </streetAddress>. Only these two tags, so not the content in it (that's why i was thinking of removing lines instead of parsing it)

Answer 1

Using event-based SAX parser you can filter tags with low memory usage and good performance:

from xml.sax import make_parser
from xml.sax.saxutils import XMLFilterBase, XMLGenerator

# filter class which skips startElement and endElement events
# for tags configured
class MyFilter(XMLFilterBase):

    def __init__(self, tags_to_exclude, parent=None):
        super().__init__(parent)

        # tags to exclude
        self._tags_to_exclude = tags_to_exclude

    def startElement(self, name, attrs):
        if name not in self._tags_to_exclude:
            super().startElement(name, attrs)

    def endElement(self, name):
        if name not in self._tags_to_exclude:
            super().endElement(name)


# define tags to be ecluded    
tags_to_exclude = {'streetAddress'}

# create filter    
reader = MyFilter(tags_to_exclude, make_parser())

# parse source and write to target
with open('target.xml', 'w') as file:
    handler = XMLGenerator(file)
    reader.setContentHandler(handler)
    reader.parse('source.xml')