Python selenium test gets stuck in urlopen

Question

My app relies on:

• Python 3
• Django 1.8
• Weasyprint
• Selenium

It runs flawlessly on dev and production environment, but not while testing with selenium.

Using weasyprint, I create a PDF from HTML, this library uses urllib to download CSS (eg http://localhost:8081/static/lib/bootstrap/css/bootstrap.min.css ), but it hangs (no errors, just stuck) while opening these.

If I enter this url directly in my browser while hanged, the CSS is displayed.

Command used:

./manage.py test tests.test_account.HomeNewVisitorTest

Relevant part of the test:

from selenium import webdriver

class HomeNewVisitorTest(StaticLiveServerTestCase):
    def setUp(self):
        if TEST_ENV_FIREFOX:
            self.driver = webdriver.Firefox()
        else:
            self.driver = webdriver.PhantomJS()
        self.driver.set_window_size(1440, 900)

    def tearDown(self):
        try:
            path = 'worksites/' + self.worksite_name.lower()
            os.rmdir(settings.MEDIA_ROOT + path)
        except FileNotFoundError:
            pass
        super().tearDown()

    def test(self):
        d = self.driver
        d.get(self.get_full_url('home'))
        d.find_element_by_css_selector('.btn-success[type=submit]').click()

In my view:

    # Generate PDF for contact directory
    template = get_template("pdf/annuaire.html")
    context = {"worksite": worksite}
    html = template.render(RequestContext(self.request, context))
    base_url = self.request.build_absolute_uri("/")
    pdf = weasyprint.HTML(string=html, base_url=base_url)
    pdf.write_pdf(directory + '/annuaire.pdf')

Here is the thread dump when stuck:

Fatal Python error: Aborted

Thread 0x0000000106f92000 (most recent call first):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374 in readinto
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313 in _read_status
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351 in begin
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171 in getresponse
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1185 in do_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1210 in http_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441 in _call_chain
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 481 in _open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 463 in open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 161 in urlopen
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/urls.py", line 276 in default_url_fetcher
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/urls.py", line 311 in fetch
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/contextlib.py", line 59 in __enter__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 297 in _select_source
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/contextlib.py", line 59 in __enter__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 223 in __init__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/css/__init__.py", line 198 in find_stylesheets
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/css/__init__.py", line 448 in get_all_computed_styles
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/document.py", line 312 in _render
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 132 in render
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 164 in write_pdf
  File "/Users/sebcorbin/Sites/planexo/worksite/views.py", line 111 in done
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 357 in render_done
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 730 in render_done
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 300 in post
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 686 in post
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/views/generic/base.py", line 89 in dispatch
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 237 in dispatch
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/views/generic/base.py", line 71 in view
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/contrib/auth/decorators.py", line 22 in _wrapped_view
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/handlers/base.py", line 132 in get_response
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 189 in __call__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/testcases.py", line 1099 in __call__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/contrib/staticfiles/handlers.py", line 63 in __call__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/wsgiref/handlers.py", line 137 in run
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/servers/basehttp.py", line 182 in handle
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 673 in __init__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/servers/basehttp.py", line 102 in __init__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 344 in finish_request
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 331 in process_request
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 305 in _handle_request_noblock
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 238 in serve_forever
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/testcases.py", line 1182 in run
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 920 in _bootstrap_inner
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 888 in _bootstrap

Current thread 0x00007fff7996a300 (most recent call first):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374 in readinto
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313 in _read_status
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351 in begin
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171 in getresponse
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1185 in do_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1210 in http_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441 in _call_chain
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 481 in _open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 463 in open
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 457 in _request
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 389 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 191 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/webelement.py", line 447 in _execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/webelement.py", line 68 in click
  File "/Users/sebcorbin/Sites/planexo/tests/test_account.py", line 203 in _test_worksite_form
  File "/Users/sebcorbin/Sites/planexo/tests/test_account.py", line 36 in test
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 577 in run
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 625 in __call__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/testcases.py", line 186 in __call__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/suite.py", line 122 in run
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/suite.py", line 84 in __call__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/runner.py", line 168 in run
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/runner.py", line 178 in run_suite
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/runner.py", line 211 in run_tests
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/commands/test.py", line 90 in handle
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/base.py", line 441 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/commands/test.py", line 74 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/base.py", line 390 in run_from_argv
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/commands/test.py", line 30 in run_from_argv
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/__init__.py", line 330 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/__init__.py", line 338 in execute_from_command_line
  File "./manage.py", line 10 in <module>

Answer 1

Thanks to you all for pinpointing the problem, I took for inspiration this article http://nedbatchelder.com/blog/201103/quick_and_dirty_multithreaded_django_dev_server.html and modified my manage.py file:

#!/usr/bin/env python
from http import server
import os
from socketserver import ThreadingMixIn
import sys


def monkey_patch_test_server():
    # This monkey-patches HTTPServer to create a base HTTPServer class that
    # supports multithreading
    originalhttpserver = server.HTTPServer

    class ThreadedHTTPServer(ThreadingMixIn, originalhttpserver):
        def __init__(self, server_address, RequestHandlerClass=None):
            originalhttpserver.__init__(self, server_address,
                                        RequestHandlerClass)

    server.HTTPServer = ThreadedHTTPServer


if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "planexo.settings")

    from django.core.management import execute_from_command_line

    if sys.argv[1] == 'test':
        monkey_patch_test_server()

    execute_from_command_line(sys.argv)

Now it works!

Answer 2

Use Python faulthandler module to print the traceback of all hung threads. This will pinpoint you the actual functions were the hanging is happening.

If needed edit your question and add the relevant information.

Otherwise my guess is that you are using Django test / development server in such a way that it can only handle one request at a time. Your initial HTTP request to dev server triggers a request to request data from itself, which Django dev server is unable to do. But this is just a guess.

Also relevant source code missing from the question.

Answer 3

I'll share my case:

I had this issue with my Django selenium tests (django==1.7.12 and selenium==2.53.1) with ChromeDriver 2.21.371459 and Google Chrome 48.0.2564.116.

I was able to isolate the issue. In my case it was happening only for pages referencing a static file (an image in a HTML <img> tag for instance http://cdn.local.myproject.net/static/myimage.png ) on my custom local cdn domain. The issue was not present if I used a relative path "/static/myimage.png" or localhost " http://127.0.0.1/static/myimage.png " so I figured it was a DNS problem.

I was able to bypass the problem by using the --dns-prefetch-disable option of chrome.

Example in Python:

from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--dns-prefetch-disable')
driver = Chrome(chrome_options=options)

I don't know if this is the general case but hopefully it can help some of you.

Answer 4

Use the faulthandler module. Maybe it can also be that Django, can only take 1 request.