使用selenium webdriver限制phantomjs的内存使用？

Question

I'm running phantomjs in Remote WebDriver mode with phantomjs --webdriver 8910 and then getting many pages using the Selenium python bindings with something like:

wd = webdriver.PhantomJS(port=8910)
for url in big_url_list:
    wd.get(url)
    # do something here, e.g. wd.save_screenshot or print wd.page_source
    wd.quit()

The wd.quit() appears to do nothing. Monitoring the process with top shows that the memory usage of the phantomjs process continues to increase as more pages are retrieved. If I restart the phantomjs process, then the memory usage drops low and begins it's steady climb again.

The close method mentioned in the phantomjs docs sounds promising, but I don't see any way to call that via the WebDriver protocol .

Is there (i) a way to keep the memory usage of phantomjs down when making many requests, or (ii) a reliable way to monitor and periodically restart the phantomjs process so that the memory never gets out of hand?

Answer 1

This SO question suggests the use of release which has been deprecated in favor of close . It seems that the python webdriver API exposes the close method ?

Answer 2

PhantomJS webpage close method calls by Ghostdriver only when you close WebDriver session. You can try to use runit to restart PhantomJS when memory limit was reached. Create bash script as follows:

#!/bin/sh
exec 2>&1
exec chpst -u your_user -m 104857600 /usr/bin/phantomjs --webdriver=8910

-m flag set memory limit to 100MB.

Place above script under some empty directory with name "run":

 |
 `-phantomjs-runit-+
                   `-run

Now you can start phantomjs

sv start ./phantomjs-runit/