Combine asynchronous websockets with multithreaded functions / apis in Python

Question

Please may anyone advise the best practice / method to:

combine 2 threads:

( Thread1 = Dataprovider1_live_api, Thread2 = Dataprovider2_live_api)

with an asynchronous websocket client so that I may publish out the resultant merged dataframe ( updated_summary_df ) of Thread1 & Thread2 in the best way possible?

I believe the current method is certainly not correct & I have fudged it/ made it work for only a few 10s of seconds & is prone to poor performance & crashing given how I am merging threads & an asyncronous websocket function

please any advice, ideas or tips are greatly welcome & very much appreciated:)

I would also like to add an

await webscoket.recv()

but I couldn't get it to work as it's already rather messy. Please if anyone knows how I could also receive websocket messages at the same time this would be very very helpful, or even best practice.

I would also be open to inserting the websocket function below into my dataprovider1_api_loop & dataprovider2_api_loop threads if that is the correct way to do things?

Thread1 & Thread2 share dataframes between them using a queue that is combined together in Thread2 & then published out via a websocket client.

Below the func df_random() is a good proxy to what kind of dataframes are being produced in thread1 & thread2.

def df_random():
       
    summary_df_large = pd.DataFrame()
    ccy_bbgn = ['KWN','IRN','NTN','IHN','PPN','CCN']
    while True:

        time.sleep(0.9)
        int_ = random.randint(1,9)
        int_2 = random.randint(1,9)
        # for ccy in ccy_bbgn:
        ccy = ccy_bbgn[0]
        df = pd.DataFrame([{"CCY":ccy, "TENOR":"Tom Fix","BID": int_, "ASK": int_2,"MID":7, "SPREAD": 0.8, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0},
            {"CCY":ccy,"TENOR":"1W1M","BID": 1, "ASK": 5,"MID":7, "SPREAD": 0.07, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0},
            {"CCY":ccy,"TENOR":"1x2","BID": 1, "ASK": 5,"MID":7, "SPREAD": 0.07, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0},
            {"CCY":ccy,"TENOR":"1x3","BID": 1, "ASK": 5,"MID":7, "SPREAD": 0.07, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0},
            {"CCY":ccy,"TENOR":"1x6","BID": 1, "ASK": 5,"MID":7, "SPREAD": 0.07, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0},
            {"CCY":ccy,"TENOR":"1x9","BID": 1, "ASK": 5,"MID":7, "SPREAD": 0.1, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0},
            {"CCY":ccy,"TENOR":"1x12","BID": 1, "ASK": 5,"MID":7, "SPREAD": 0.1, "SKEW": 0,"BID_override":"","ASK_override": "", "BID_output": 5,"ASK_output": 6, "MID_output": 8, "POINTS_SCALE":0,"BB MID": 8, "CHK": 8, "DAYS":0}])
            
        
        print(df.head())
        print('------------')
        yield df

Here is the data class that creates the two threads that performs operations on shared dataframes via a queue object, (each thread receives seperate & different new data in via an api)

import asyncio
import websockets
import threading
import datetime
import json
import threading
import time
import random
import pandas as pd
import datetime
import queue
import sys, os
import numpy as np

class data:
    def start_publish(self):

        
        self.df_1 = pd.DataFrame(columns=['Instrument','BID','ASK'])
        q = queue.Queue() # used for the dataprovider1
        q_summary = queue.Queue() # used for the dataprovider2
        lock = threading.Lock() # Create lock obj

        
    
       
        self.t_1 = threading.Thread(target=self.dataprovider1_api_loop, args=(q,lock),name=f"dataprovider1",daemon = True)
        self.t_2 = threading.Thread(target=self.dataprovider2_api_loop, args=(q,q_summary, lock),name=f"dataprovider2",daemon = True)

        #Start threads :
        self.t_1.start() #dataprovider1
        self.t_2.start() #dataprovider2

This data class is initialised & then called via:

#Lines 450 & 451
data_obj = data()
data_obj.start_publish()'

We then define an asyncronous websocket function, and run the event

#Line 453 onwards
async def run_websocket():

    web_socket_host = 'ws://ipaddres:port'
    
    async with websockets.connect(web_socket_host) as websocket:
        
        while True:
        
            try:
                #sumary websocket publish#
                json_summary_arr = json.dumps({"SUM":data_obj.updated_summary_df.to_dict('records'),"DATETIME":datetime.datetime.now()},default=str)
                json_arr = json.dumps({"data_prov1":data_obj.df_bbg[["TENOR","BID","ASK",'TIME']].to_dict('records'),"DATETIME":datetime.datetime.now()},default=str)
                
                await websocket.send(json_arr)
                #sumary websocket publish#
                await websocket.send(json_summary_arr)

            except Exception as e:
                print(e)

            except websockets.ConnectionClosed as exc:
                print('Connection closed while sending')
                # self.log.debug('Connection closed while sending')
                disconnected = Disconnected(exc.code)
                # self.jobs.close(Result(disconnected))
                raise disconnected from exc 


asyncio.get_event_loop().run_until_complete(run_websocket())
print('Done')
#End

Unfortunately we get the error below which I believe is due to a timeout issue, which I'm not sure how to fix, please any advice, ideas or tips are greatly welcome & very much appreciated:)

my websocket server is on a seperate machine & running in C#, I'm rather sure its not the websocket server is not causing issues.

sent 1011 (unexpected error) keepalive ping timeout; no close frame received

Answer 1

First: There are 3 main exceptions related to a closed connection occasions when working with websockets python package

1. websockets.exceptions.ConnectionClosedError
2. websockets.exceptions.ConnectionClosedOK
3. websockets.exceptions.ConnectionClosed

2'nd: you must consider order of exception on handling ladder most unique exceptions (like the 3 above) may be caught on generic "Exception" reorder the exception handling from most specific to most generic (There are many references on so to this I am not the first to mention it ). first and most specific is ConnectionClosedOK, second ConnectionClosedError, third ConnectionClosed and only then handle generics like "Exception" or RunTime. three first types of exceptions are also generics "Exception" so they will be handled before (on your code)

3'rd: on development process (unlike deployment) separate IO/Networking related operations to separate exception handling so you can verify witch caused the exception alternatively use global flag to identify the cause:

while True:
    flag_cause="ok"
    try:
        #sumary websocket publish#
        flag_cause="dumps_num01"
        json_summary_arr = json.dumps({"SUM":data_obj.updated_summary_df.to_dict('records'),"DATETIME":datetime.datetime.now()},default=str)
        flag_cause="dumps_num02"
        json_arr = json.dumps({"data_prov1":data_obj.df_bbg[["TENOR","BID","ASK",'TIME']].to_dict('records'),"DATETIME":datetime.datetime.now()},default=str)
        flag_cause="dumps_send01"
        await websocket.send(json_arr)
        #sumary websocket publish#
        flag_cause="dumps_send02"
        await websocket.send(json_summary_arr)

The trick is that this flag is global so valid on exceptions context you may print it on exception to verify the cause.

4'th: websockets package utilize tasks asyncio loop context instead of threads the websocket package use threads. so either shift your work to websocket package only, or use tasks and micro services instead of threads.

Best. R. Ori