为什么此lxml.etree.HTMLPullParser泄漏内存？

Question

I'm trying to use lxml's HTMLPullParser on Linux Mint but I'm finding that the memory usage keeps increasing and I'm not sure why. Here's my test code:

# -*- coding: utf-8 -*-
from __future__ import division, absolute_import, print_function, unicode_literals
import lxml.etree
import resource
from io import DEFAULT_BUFFER_SIZE

for _ in xrange(1000):
with open('StackOverflow.html', 'r') as f:
    parser = lxml.etree.HTMLPullParser()
    while True:
        buf = f.read(DEFAULT_BUFFER_SIZE)
        if not buf: break
        parser.feed(buf)
    parser.close()

    # Print memory usage
    print((resource.getrusage(resource.RUSAGE_SELF)[2] * resource.getpagesize())/1000000.0)

StackOverflow.html is the homepage of stackoverflow that I've saved in the same folder as the python script. I've tried adding explicit deletes and clears but so far nothing has worked. What am I doing wrong?

Answer 1

Elements constructed by the parsers are leaking, and I can't see an API contract violation in your code that's causing it. Since the objects survive a manual garbage collection run with gc.collect() , your best bet is probably to try a different parsing strategy as a workaround.

To see the root cause, I used the memory exploration module objgraph and installed xdot to view the graphs it created.

Before running the code, I ran:

In [3]: import objgraph

In [4]: objgraph.show_growth()

After running the code, I ran:

In [6]: objgraph.show_growth()
tuple                  1616      +147
_Element                146      +146
list                   1100       +24
wrapper_descriptor     1423       +15
weakref                1155        +6
getset_descriptor       677        +4
dict                   2777        +4
member_descriptor       315        +3
method_descriptor       891        +2
_TempStore                2        +1

In [7]: import random

In [8]: objgraph.show_chain(
   ...: objgraph.find_backref_chain(
   ...: random.choice(objgraph.by_type('_Element')), objgraph.is_proper_module))
Graph written to /tmp/objgraph-bfuwa9.dot (8 nodes)
Spawning graph viewer (xdot)

Note: the numbers might be different than what you see depending on the webpage viewed.