运行异步 Celery 任务时 Django 测试冻结

Question

我正在尝试在 Django 中创建单元测试，测试 Celery 任务以及该任务是否根据我提供的 CSV 数据创建对象。

出于某种奇怪的原因，我的测试在 AsyncResult 对象上调用 .get() 时冻结。

但是，当我监视 Celery 日志和数据库时，它们似乎正常工作并返回指示任务已完成的成功消息。

在测试环境之外一切正常，就在我的测试中运行 Celery 时。 我不知道发生了什么。

这是相关代码：

站/test.py

class StationsTests(TestCase):
    
    @classmethod
    def setUpTestData(cls):
        super(StationsTests, cls).setUpTestData()
        dirname = os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', 'csvimport'))

        cls.csv_data_type_station = 'station'
        cls.upload_type = 'safe_create'
        cls.file_station = os.path.join(dirname, 'CSVFiles/Station_test_csv.csv')


    @override_settings(CELERY_EAGER_PROPAGATES_EXCEPTIONS=True,
                    CELERY_ALWAYS_EAGER=True)
    def test_station_geoJSON_data(self):
        self.result_upload_csv = upload_csv.delay(
            self.file_station, self.csv_data_type_station, self.upload_type).get(timeout=10)

        self.assertEqual(self.result_upload_csv, '10 stations uploaded successfully')
        self.assertEqual(len(Station.objects.all()), 10)

设置.py：

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'db_hcbapp',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': 'localhost',
        'PORT': '5432'
    },
}
.....

CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
CELERY_IMPORTS = ('csvimport.tasks')
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'django-db'
CELERY_RESULT_EXTENDED = True

celery.py:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')
app = Celery('myapp')
app.conf.enable_utc = False
app.config_from_object(settings, namespace='CELERY')

app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

任务.py:

@shared_task(bind=True)
def upload_csv(self, file_path, csv_data_type, upload_type):

    progress_recorder = ProgressRecorder(self)
    csv_data = []   # Temporary list for storing the CSV-rows
    if not bool(upload_type == "safe_create") | bool(upload_type == "chunk_create"):
        return "The upload type needs to be either Safe Create or Chunk Create"

    with open(file_path, 'r') as file:
        reader = csv.DictReader(file)

        if csv_data_type == 'station':
            pattern = "FID,ID,Nimi,Namn,Name,Osoite,Adress,Kaupunki,Stad,Operaattor,Kapasiteet,x,y"
            pattern = pattern.replace("(", "\(").replace(")", "\)")
            if not re.search(pattern, ",".join(reader.fieldnames), re.IGNORECASE):
                raise Exception(
                    "CSV invalid as it does not contain the correct fieldnames '['FID','ID','Nimi','Namn','Name','Osoite','Adress','Kaupunki','Stad','Operaattor','Kapasiteet','x','y']' for a Station upload.")

            # Fields in the CSV (and the index number): [0]FID,[1]ID,[2]Nimi,[3]Namn,[4]Name,[5]Osoite,[6]Address,[7]Kaupunki,[8]Stad,[9]Operaattor,[10]Kapasiteet,[11]x,[12]y
            # Fieldnames are reassigned just in case so the object data can be gathered from correctly named row key-names.
            reader.fieldnames = ['FID', 'ID', 'Nimi', 'Namn', 'Name', 'Osoite',
                                 'Adress', 'Kaupunki', 'Stad', 'Operaattor', 'Kapasiteet', 'x', 'y']

        print("Reading the csv")
        for row in reader:
            csv_data.append(row)
        file.seek(0)

    # Creating objects ->>
    rowcount = len(csv_data)
    print(f"CSV read complete with {rowcount} rows. Creating objects...")

    try:
        object_list = []    # Temporary list for the objects about to be created in bulk
        if csv_data_type == 'station':

            print("Uploading a Station")
            for i, row in enumerate(csv_data):

                # Check empty rows
                if any(value for value in row.values()):
                    try:
                        csv_data = Station(
                            station_id=row['ID'],
                            fid=row['FID'],
                            name_fin=row['Nimi'],
                            name_swe=row['Namn'],
                            name_eng=row['Name'],
                            address_fin=row['Osoite'],
                            address_swe=row['Adress'],
                            city_fin=row['Kaupunki'],
                            city_swe=row['Stad'],
                            operator=row['Operaattor'],
                            capacity=int(row['Kapasiteet']),
                            geo_pos_x=Decimal(row['x']),
                            geo_pos_y=Decimal(row['y'])
                        )

                        # If station is found in the db, skip to the next iteration
                        if Station.objects.filter(station_id=csv_data.station_id).exists():
                            print("Station already exists")
                            continue

                        # Add the object to the list
                        object_list.append(csv_data)

                        # Chunk create checks if there are 100 Journey-objects, if so, creates them and clears the temporary list
                        if upload_type == 'chunk_create':
                            if len(object_list) >= 100:
                                Station.objects.bulk_create(object_list)
                                # Clear the list
                                object_list = []

                        print(
                            f"Row {i+1} : Station {row['ID']}.{row['Nimi']} created!")

                    except ValueError as e:
                        print(f"Row {i+1} : Value error in row {i+1}! ", e)

                    except Exception as e:
                        print(f"Row {i+1} : Exception: {e}")
                        continue
                else:
                    # In case a csv-row is empty
                    print(f"Row {i+1} : Field empty")

                # Updates the progress after every iteration and passes the current state (in percentages) to the front end
                progress_recorder.set_progress(
                    i + 1, rowcount, f'{round(((i+1) / rowcount) * 100),2}%')
            # If there are leftover objects in the list after bulk_create or safe_create has finished creating all objects, add them to the db.
            if object_list:
                Station.objects.bulk_create(object_list)
            return f"{rowcount} stations uploaded successfully"

        # Just in case neither Journey or Station upload_type is chosen
        else:
            print("Select a CSV containing proper values of either Stations or Journeys")

    except Exception as e:
        print(f"Exception, something went wrong: {e}")
        return f"Something went wrong: {e}"

就像我说的，外部测试任务和数据库完美运行。 Redis 和 Celery 均已启动并正在运行，任务结果已添加到 PostgreSQL 数据库中。

任何想法可能导致冻结？

• 我试过 mocking 使用 @patch('myapp.tasks.upload_csv.delay') 的测试方法，但这似乎并没有在我的真实数据库中创建任何对象。 因此self.assertEqual(len(Station.objects.all()), 10)我想验证任务在我的数据库中创建对象。

• 我已经尝试将我的数据库更改回 SQLite，但它仍然冻结。

• 我尝试在 tasks.py 中创建其他简单的临时任务。 但即使是那些冻结。

有任何想法吗？

Answer 1

我想通了，它也可能对其他人有帮助！

它显然在一个单独的数据库中运行测试，而 Celery 任务使用了我的真实数据库。 因此，即使 Celery / Redis 工作并返回值，它们也没有将值返回到我的测试环境。 在某种程度上有点明显。

我建议在测试方法中使用 @override_settings(CELERY_TASK_ALWAYS_EAGER=True)，这样它就包含在测试过程中了！