[英]Running multithreaded code after Queue.task_done()
In a classical "Threading/Queue"-application. 在经典的“线程/队列”应用程序中。 I need to do further calculations in my "consumer"-function.
我需要在我的“消费者”功能中做进一步的计算。 After Queue is empty no further code is executed after urls.task_done().
在Queue为空之后,在urls.task_done()之后不再执行任何代码。
I am importing market data from an JSON api and import it into my MariaDB database. 我从JSON api导入市场数据并将其导入我的MariaDB数据库。 On the API every item that i want to fetch has an own url, so I am creating a queue for all available urls in a function.
在API上我想要获取的每个项目都有一个自己的url,所以我在一个函数中为所有可用的url创建一个队列。 A "consumer"-function processes the queue importing a new set of data or updating an existent entry depending on the already existing data in my database.
“消费者”功能处理队列,导入一组新数据或根据我数据库中已有的数据更新现有条目。 I already tried to wrap the actual while True loop into its own function but it didn't work for me.
我已经尝试将实际的True循环包装到它自己的函数中,但它对我不起作用。
def create_url():
try:
mariadb_connection = mariadb.connect(host='host
database='db',
user='user',
password='pw')
cursor = mariadb_connection.cursor()
cursor.execute('SELECT type_id from tbl_items')
item_list = cursor.fetchall()
print("Create URL - Record retrieved successfully")
for row in item_list:
url = 'https://someinternet.com/type_id=' + \
str(row[0])
urls.put(url)
return urls
except mariadb.Error as error:
mariadb_connection.rollback()
print("Failed retrieving itemtypes from tbl_items table
{}".format(error))
finally:
if mariadb_connection.is_connected():
cursor.close()
mariadb_connection.close()
def import(urls):
list_mo_esi = []
try:
mariadb_connection = mariadb.connect(host='host',
database='db',
user='user',
password='pw')
cursor = mariadb_connection.cursor()
while True:
s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[502, 503, 504])
s.mount('https://', HTTPAdapter(max_retries=retries))
jsonraw = s.get(urls.get())
jsondata = ujson.loads(jsonraw.text)
for row in jsondata:
cursor.execute('SELECT order_id from tbl_mo WHERE order_id = %s',
(row['order_id'], ))
exists_mo = cursor.fetchall()
list_mo_esi.append(row['order_id'])
if len(exists_mo) != 0:
print("updating order#", row['order_id'])
cursor.execute('UPDATE tbl_mo SET volume = %s, price = %s WHERE order_id = %s',
(row['volume_remain'], row['price'], row['order_id'], ))
mariadb_connection.commit()
else:
cursor.execute('INSERT INTO tbl_mo (type_id, order_id, ordertype,volume, price) VALUES (%s,%s,%s,%s,%s)',
(row['type_id'], row['order_id'], row['is_buy_order'], row['volume_remain'], row['price'], ))
mariadb_connection.commit()
urls.task_done()
except mariadb.Error as error:
mariadb_connection.rollback()
print("Failed retrieving itemtypes from tbl_items table {}".format(error))
The following finally part of my function is not executed, but should. 以下最后一部分我的函数没有执行,但应该执行。
finally:
list_mo_purge = list(set(list_mo_sql)-set(list_mo_esi))
cursor.execute('SELECT order_id FROM tbl_mo')
list_mo_sql = cursor.fetchall()
print(len(list_mo_esi))
print(len(list_mo_sql))
if mariadb_connection.is_connected():
cursor.close()
mariadb_connection.close()
main thread 主线程
for i in range(num_threads):
worker = Thread(target=import_mo, args=(urls,))
worker.setDaemon(True)
worker.start()
create_url()
urls.join()
After all tasks are completed my worker stop executing code right after urls.task_done(). 完成所有任务后,我的工作人员在urls.task_done()之后立即停止执行代码。 However, i have some more code after the function urls.task_done() i need to be executed for closing database connection and cleaning up my database from old entries.
但是,我在函数urls.task_done()之后还有一些代码需要执行以关闭数据库连接并从旧条目清理我的数据库。 How can I make this "finally"-part run?
我怎样才能让这个“终于” - 分开?
You are not breaking from the while. 你不是在打破。
You should do the following: 您应该执行以下操作:
if urls.empty():
break
Most likely your import
thread gets blocked at urls.get()
很可能你的
import
线程在urls.get()
被阻止
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.