如何在 Django 測試中從多個線程訪問 Postgresql 數據庫？

Question

我有一個 Django 網站和一個 PostgreSQL 數據庫，它運行一些多線程的后台任務。 但是，我在測試這個多線程部分時遇到了麻煩。

I followed https://docs.djangoproject.com/en/3.1/intro/tutorial01/ and https://docs.djangoproject.com/en/3.1/intro/tutorial02/ to start a new Django website. 然后我添加polls/tests.py如下：

import datetime

from django.test import TestCase, TransactionTestCase
from django.utils import timezone

from .models import Question

import threading, time

class QuestionModelTests(TransactionTestCase):  # line 10
# class QuestionModelTests(TestCase):           # line 11

    def test_was_published_recently_with_future_question(self):
        Question(question_text='asdf', pub_date=datetime.datetime.now()).save()
        print('a', Question.objects.filter())
        def thread1():
            print('b', Question.objects.filter())
        t = threading.Thread(target=thread1, daemon=True)
        t.start()
        
        time.sleep(1)
        assert not t.is_alive()

我還更改了settings.py以切換到新數據庫：

if 'POSTGRESQL':    # line 79
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'mysite',
            'USER': 'postgres',
            'PASSWORD': 'my_postgresql_password',
            'HOST': '127.0.0.1',
            'PORT': '5432',
        }
    }
else:               # line 90
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': BASE_DIR / 'db.sqlite3',
        }
    }

但是，如果我運行python3 manage.py test ，我會收到以下錯誤：

(tmp) [user@localhost mysite]$ python3 manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: database "test_mysite" already exists

Type 'yes' if you would like to try deleting the test database 'test_mysite', or 'no' to cancel: yes
Destroying old test database for alias 'default'...
System check identified no issues (0 silenced).
/home/user/.local/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1367: RuntimeWarning: DateTimeField Question.pub_date received a naive datetime (2021-03-24 23:16:39.936567) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
a <QuerySet [<Question: Question object (1)>]>
b <QuerySet [<Question: Question object (1)>]>
.
----------------------------------------------------------------------
Ran 1 test in 1.156s

OK
Destroying test database for alias 'default'...
/home/user/.local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py:304: RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the first PostgreSQL database instead.
  warnings.warn(
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.ObjectInUse: database "test_mysite" is being accessed by other users
DETAIL:  There is 1 other session using the database.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 302, in _nodb_cursor
    yield cursor
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/base/creation.py", line 293, in _destroy_test_db
    cursor.execute("DROP DATABASE %s"
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/user/.local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
django.db.utils.OperationalError: database "test_mysite" is being accessed by other users
DETAIL:  There is 1 other session using the database.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/mysite/manage.py", line 22, in <module>
    main()
  File "/home/user/mysite/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/user/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/user/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/user/.local/lib/python3.9/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/home/user/.local/lib/python3.9/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/user/.local/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/home/user/.local/lib/python3.9/site-packages/django/core/management/commands/test.py", line 53, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/user/.local/lib/python3.9/site-packages/django/test/runner.py", line 705, in run_tests
    self.teardown_databases(old_config)
  File "/home/user/.local/lib/python3.9/site-packages/django/test/runner.py", line 645, in teardown_databases
    _teardown_databases(
  File "/home/user/.local/lib/python3.9/site-packages/django/test/utils.py", line 298, in teardown_databases
    connection.creation.destroy_test_db(old_name, verbosity, keepdb)
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/base/creation.py", line 277, in destroy_test_db
    self._destroy_test_db(test_database_name, verbosity)
  File "/home/user/.local/lib/python3.9/site-packages/django/db/backends/base/creation.py", line 293, in _destroy_test_db
    cursor.execute("DROP DATABASE %s"
  File "/usr/lib64/python3.9/contextlib.py", line 166, in __exit__
    raise RuntimeError("generator didn't stop after throw()")
RuntimeError: generator didn't stop after throw()
(tmp) [user@localhost mysite]$ 

看來經過測試，線程還有數據庫連接，所以Django無法移除測試數據庫。

如果我使用默認的 SQLite3 數據庫（即禁用第 79 行並啟用第 90 行），那么它工作正常。 我也嘗試使用TestCase代替TransactionTestCase （見第 10 行和第 11 行），但效果不佳：

• 對於 PostgreSQL，線程無法再看到添加到數據庫中的新條目（ https://stackoverflow.com/a/31652691/7709727 ）
• 對於 SQLite3，我收到另一個錯誤（ sqlite3.OperationalError: database table is locked: polls_question ）

所以我的問題是，如何展示我的測試，以便測試可以以 PostgreSQL 作為服務器成功運行。 請注意，為了簡單起見，我將多線程部分放入了tests.py 。 實際上它來自另一個要測試的模塊。

Update: Complete code: https://gist.github.com/lxylxy123456/5a36719faf32736431f3ee444bf4c17c I am using Django 3.1.7

Answer 1

您需要在使用完數據庫連接后關閉它們。

from django import db
db.connections.close_all()

因此，在您的情況下，您需要終止線程並讓線程關閉 Django DB 連接。

Answer 2

關閉線程的目標 function 內的 DB 連接。 所以在您的示例代碼中，它看起來像這樣：

def thread1():
    # ...Your code here...
    from django.db import connection
    connection.close()

...

t = threading.Thread(target=thread1)
t.start()
t.join()