python peewee多處理池錯誤

Question

堆棧：python3.4，PostgreSQL 9.4.7，peewee 2.8.0，psycopg2 2.6.1（dt dec pq3 ext lo64）

我需要能夠與每個工作人員的postgresql數據庫進行對話（選擇，插入，更新）。 我正在使用pythons多處理池創建10個工作程序，每個工作程序進行curl調用，然后根據發現的內容與數據庫進行對話。

在閱讀了互聯網上的一些線程之后，我認為連接池是必經之路。 因此，我將代碼放在了我的models.py文件之上。 我對連接池有疑問，因為我的理解是跨線程復用數據庫連接是不行的。

db = PooledPostgresqlExtDatabase(
    'uc',
    max_connections=32,
    stale_timeout=300,  # 5 minutes.
    **{'password': cfg['psql']['pass'], 
       'port': cfg['psql']['port'], 
       'register_hstore':False,
       'host': cfg['psql']['host'], 
       'user': cfg['psql']['user']})

現在到這個問題。 從某些工作人員與數據庫進行通訊時，出現隨機SQL錯誤。 在將peewee引入混合之前，我使用的是沒有包裝的“ psycopg2”庫。 我還在為每個工作人員創建一個新的數據庫連接。 沒有錯誤。

我得到的一個示例錯誤是：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/playhouse/postgres_ext.py", line 377, in execute_sql
    self.commit()
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3468, in commit
    self.get_conn().commit()
psycopg2.DatabaseError: error with no message from the libpq

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/dan/dev/link-checker/crawler/manager.py", line 17, in startWorker
    wrk.perform()
  File "/home/dan/dev/link-checker/crawler/worker.py", line 49, in perform
    self.pullUrls()
  File "/home/dan/dev/link-checker/crawler/worker.py", line 63, in pullUrls
    newUrlDict = UrlManager.createUrlWithInProgress(self._url['crawl'], source_url, self._url['base'])
  File "/home/dan/dev/link-checker/crawler/models.py", line 152, in createUrlWithInProgress
    newUrl = Url.create(**newUrlDict)
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 4494, in create
    inst.save(force_insert=True)
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 4680, in save
    pk_from_cursor = self.insert(**field_dict).execute()
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3213, in execute
    cursor = self._execute()
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 2628, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/usr/local/lib/python3.4/dist-packages/playhouse/postgres_ext.py", line 377, in execute_sql
    self.commit()
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3285, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 127, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/playhouse/postgres_ext.py", line 377, in execute_sql
    self.commit()
  File "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3468, in commit
    self.get_conn().commit()
peewee.DatabaseError: error with no message from the libpq

我還拖尾了postgresql文件，這就是我所看到的：

2016-04-19 20:34:23 EDT [26824-3] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:34:23 EDT [26824-4] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:34:23 EDT [26824-5] uc_user@uc WARNING:  there is no transaction in progress
2016-04-19 20:34:23 EDT [26824-6] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:34:23 EDT [26824-7] uc_user@uc WARNING:  there is no transaction in progress
2016-04-19 20:34:23 EDT [26824-8] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:34:23 EDT [26824-9] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:35:14 EDT [26976-1] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:35:14 EDT [26976-2] uc_user@uc WARNING:  there is no transaction in progress
2016-04-19 20:35:14 EDT [26976-3] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:35:14 EDT [26976-4] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:35:14 EDT [26976-5] uc_user@uc WARNING:  there is no transaction in progress
2016-04-19 20:35:14 EDT [26976-6] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:35:14 EDT [26976-7] uc_user@uc WARNING:  there is no transaction in progress
2016-04-19 20:35:14 EDT [26976-8] uc_user@uc WARNING:  there is already a transaction in progress
2016-04-19 20:35:14 EDT [26976-9] uc_user@uc WARNING:  there is no transaction in progress

我的直覺是連接池和多處理不能很好地結合在一起。 有沒有人成功地做到了這一點而沒有錯誤；如果是這樣，您能給我一個例子或給我一些可行的建議嗎？

我是否需要在工作進程中顯式創建與peewee的新連接，或者有更簡單的方法將peewee與多處理池庫一起使用。

感謝您的回答和閱讀。

Answer 1

我開始工作了，models.py文件中的所有代碼都將被工作人員使用。 我將其包裝在“使用db.execution_context作為ctx”中，如本頁所示：

http://docs.peewee-orm.com/zh-CN/latest/peewee/database.html#advanced-connection-management