簡體   English   中英

python asyncio任務未並行執行

[英]python asyncio task not executing in Parallel

我正在創建異步任務,以便它們可以像這樣並行執行

 for symbol in config.symbol_list:
                    tasks.append(asyncio.ensure_future(get_today_Data_async(symbol), loop=loop))
                loop.run_until_complete(asyncio.wait(tasks))

這是我想並行執行的任務

async def get_today_Data_async(symbol):

    periodType = 'day'
    period = 1
    frequencyType = 'minute'
    frequency = '1'
    use_last10_Min = False
    logging.info(f'Updating data {symbol} started...')
    try:
        logging.info(f'thread id - {threading.get_ident()} getting market data {symbol} periodType {periodType} period {period} frequencyType {frequencyType} frequency {frequency}')

        est = pytz.timezone('US/Eastern')
        if use_last10_Min:
            startDate = (datetime.datetime.now()- datetime.timedelta(minutes=10)).astimezone(tz=est).timestamp()
        else:
            startDate =(datetime.datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)).astimezone(tz=est).timestamp()
        endDate = (datetime.datetime.now()+datetime.timedelta(hours=48)).astimezone(tz=est).timestamp()
        endDate = str(endDate).split('.')[0] + '000'
        startDate = str(startDate).split('.')[0] + '000'

        reqDict = {'apikey': '' + config.client_id + '@AMER.OAUTHAP','endDate': endDate, 'frequencyType': frequencyType,'frequency': frequency,
                   'startDate': startDate, 'needExtendedHoursData': usePreMarket}

        header = {'Authorization': 'Bearer ' + config.token['access_token'] + '', 'content-type': 'application/json'}
        logging.info(f"thread id - {threading.get_ident()} datetime check {symbol} {datetime.datetime.now()}   {reqDict}")
        with await tlock:
            resp = requests.get("https://api.tdameritrade.com/v1/marketdata/" + symbol + "/pricehistory", params=reqDict)
        logging.info(f'thread id - {threading.get_ident()} datetime check {symbol} {datetime.datetime.now()} {resp.status_code}')
        if resp.status_code == 200 and not resp.json()['empty']:
            candles = resp.json()['candles']
            logging.info(f"symbol candel {symbol} {frequencyType} {frequency} {period} {get_one_hour(resp.json()['candles'])}")
            if not usePreMarket:
                newcandles = []
                EST = pytz.timezone('us/eastern')
                time_ist_end = datetime.datetime.now(EST).replace(hour=16, minute=00, second=00)
                time_ist_start = time_ist_end.replace(hour=9, minute=30, second=00)
                for x in candles:
                    tmp_date = datetime.datetime.fromtimestamp((x.get('datetime') / 1000), tz=pytz.timezone('US/Eastern'))
                    if tmp_date > time_ist_start and tmp_date < time_ist_end:
                        newcandles.append(x)
                if len(newcandles) > 0:
                    process_price(symbol,newcandles)
            else:
                if len(candles) > 0:
                    process_price(symbol, candles)

        logging.info(f" symbol - {symbol} status code {resp.status_code} resp {resp.text}")

    except Exception as e:
        traceback.print_exc()
        logging.error(f'Error in getting price {e}')
    logging.info(f'Updating data {symbol} completed...')

但是任務按順序執行,產生以下輸出

2020-10-14 20:22:43,293  - root - get_today_Data_async - 398 - INFO - Updating data AAPL started...
2020-10-14 20:22:45,066  - root - get_today_Data_async - 442 - INFO - Updating data AAPL completed...
2020-10-14 20:22:45,066  - root - get_today_Data_async - 398 - INFO - Updating data MSFT started...
2020-10-14 20:22:46,301  - root - get_today_Data_async - 442 - INFO - Updating data MSFT completed...
2020-10-14 20:22:46,301  - root - get_today_Data_async - 398 - INFO - Updating data AMZN started...
2020-10-14 20:22:47,573  - root - get_today_Data_async - 442 - INFO - Updating data AMZN completed...
2020-10-14 20:22:47,573  - root - get_today_Data_async - 398 - INFO - Updating data FB started...
2020-10-14 20:22:48,907  - root - get_today_Data_async - 442 - INFO - Updating data FB completed...
2020-10-14 20:22:48,907  - root - get_today_Data_async - 398 - INFO - Updating data GOOGL started...
2020-10-14 20:22:51,266  - root - get_today_Data_async - 442 - INFO - Updating data GOOGL completed...
2020-10-14 20:22:51,266  - root - get_today_Data_async - 398 - INFO - Updating data GOOG started...
2020-10-14 20:22:52,585  - root - get_today_Data_async - 442 - INFO - Updating data GOOG completed...
2020-10-14 20:22:52,585  - root - get_today_Data_async - 398 - INFO - Updating data JNJ started...
2020-10-14 20:22:54,041  - root - get_today_Data_async - 442 - INFO - Updating data JNJ completed...
2020-10-14 20:22:54,041  - root - get_today_Data_async - 398 - INFO - Updating data PG started...
2020-10-14 20:22:55,275  - root - get_today_Data_async - 442 - INFO - Updating data PG completed...
2020-10-14 20:22:55,275  - root - get_today_Data_async - 398 - INFO - Updating data V started...
2020-10-14 20:22:56,563  - root - get_today_Data_async - 442 - INFO - Updating data V completed..

這意味着任務正在按順序執行。 大約有 500 個符號。 你能幫我嗎,這樣我就可以並行執行任務

在python中,理論上在任何給定時間都沒有並行執行。

Python 的全局解釋器鎖 (GIL) 是一種復雜的機制,我不會在這里解釋,如果您願意,可以閱讀它,但它會阻止 Python 代碼同時在兩個不同的線程運行。

那么為什么仍然使用線程/並行處理? 在 Python 中,解決 I/O(輸入輸出)問題是並行處理的經典之選,我將舉例說明。 如果你有一個發出 HTTP 請求的代碼,因為網絡數據傳輸比 cpu 處理慢得多,為了使你的代碼最有效,你寧願在一個線程上發出請求,而不是讓程序卡住並等待響應,繼續使用其他線程發出請求,而不是針對每個返回的響應,處理從該響應中獲得的輸出。

這就是為什么在 Python 中,很多問題可能不應該是多線程的,而在其他語言中它有一些好處。

使用 python 實現真正並行處理的一種方法是使用multiprocessing模塊。 但請記住,它會比普通的 Python 執行有更多的 RAM 使用量,因為您在內存中有多個相同的堆棧,而且它不一定會更快,因為打開和關閉進程需要時間。

暫無
暫無

聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM