我将如何在python中创建（异步/线程/任务）backgroundqueue？

Question

For a C# application I have used a backgroundqueue where I could enqueue 'actions' in. I wish to do the same in Python.

The backgroundqueue should 'enqueue' an 'action' which contains a call to a function (with or without variables) and should keep going through the tasks while the main program keeps doing its own function.

I already tried with rq, but that seemed not to work. I would love to hear some suggestions!

Edit: The code this is about:

class DatabaseHandler:
def __init__(self):
    try:
        self.cnx = mysql.connector.connect(user='root', password='', host='127.0.0.1', database='mydb')
        self.cnx.autocommit = True
        self.loop = asyncio.get_event_loop()
    except mysql.connector.Error as err:
        if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
            print("Something is wrong with your user name or password")
        elif err.errno == errorcode.ER_BAD_DB_ERROR:
            print("Database does not exist")
        else:
            print(err)
    self.get_new_entries(30.0)

    async def get_new_entries(self, delay):
        start_time = t.time()
        while True:
            current_time = datetime.datetime.now() - datetime.timedelta(seconds=delay)
            current_time = current_time.strftime("%Y-%m-%d %H:%M:%S")
            data = current_time
            print(current_time)
            await self.select_latest_entries(data)
            print("###################")
            t.sleep(delay - ((t.time() - start_time) % delay))

    async def select_latest_entries(self, input_data):
        query = """SELECT FILE_NAME FROM `added_files` WHERE CREATION_TIME > %s"""
        cursor = self.cnx.cursor()
        await cursor.execute(query, (input_data,))
        async for file_name in cursor.fetchall():
            file_name_string = ''.join(file_name)
            self.loop.call_soon(None, self.handle_new_file_names, file_name_string)
        cursor.close()

    def handle_new_file_names(self, filename):
        # self.loop.run_in_executor(None, NF.create_new_npy_files, filename)
        # self.loop.run_in_executor(None, self.update_entry, filename)
        create_new_npy_files(filename)
        self.update_entry(filename)

    def update_entry(self, filename):
        print(filename)
        query = """UPDATE `added_files` SET NPY_CREATED_AT=NOW(), DELETED=1 WHERE FILE_NAME=%s"""
        update_cursor = self.cnx.cursor()
        self.cnx.commit()
        update_cursor.execute(query, (filename,))
        update_cursor.close()

The create_new_npy_files(filename) is a static method from a static class if that makes sense. It is a pretty time consuming function (1-2 sec)

Answer 1

If the action to execute is short and non-blocking, you can use call_soon :

loop = asyncio.get_event_loop()
loop.call_soon(action, args...)

If the action can take longer or can block, use run_in_executor to submit them to a thread pool:

loop = asyncio.get_event_loop()
future = loop.run_in_executor(None, action, args...)
# you can await the future, access its result once ready, etc.

Note that both above snippets assume that you already use asyncio in your program, based on the python-asyncio tag. This means that your select_statement would look like this:

async def select_statement():
    loop = asyncio.get_event_loop()
    while True:
        # requires an async-aware db module
        await cursor.execute(query, (input_data,))
        async for file_name in cursor.fetchall():
            loop.call_soon(self.handle_new_file_names, file_name_string))
            # or loop.run_in_executor(...)