how to use multithreading with pika and rabbitmq to perform Requests and Responses RPC Messages

Question

I'm working on a Project with Rabbitmq, I'm using the RPC Pattern, basically I'm receiving or consuming Messages from a Queue, make some Processing and then send a Response back. Im using Pika, my goal is to use a Thread per Task so for every Task i ll make a Thread perticularly for that Task. I also read that the best Practice is to make only one Connection and under it many channels as i want to, but i get always this Error :
'start_consuming may not be called from the scope of ' pika.exceptions.RecursionError: start_consuming may not be called from the scope of another BlockingConnection or BlockingChannel callback.

I made some Research and found out that Pika is not thread safe and we should use for every Thread an Independant Connection and a channel. but i dont want to do that since it is considered bad Practice. So I wanted to ask here if someone already achieved to make this work. I read also that it is Possible if i didn't use BlockingConnection to instantiate my Connection and also that there is a Function called add_callback_threadsafe which can make this Possible. but there is unfortunally no Examples for that and I read the Documentation but it's complex and without Examples it was hard for me to grasp what they want to describe.

my Try was to declare two Classes. Each class will represent a Task Executer which receive or consume a message from a queue and based on that made some Processing and deliver a Response back. my idea was to share a rabbitmq Connection between the two Tasks but every Task will get an independant Channel. in the Code above the rabbit Parameter passed to the function is a Class that holds some Variables like Connection and other Functions like EventSubscriber which when called it will assign a new Channel and start consuming messages from that Particular Exchanges and routingKey. Next i declare a Thread and give the subscribe or Consume function as a Target to that Thread. the other Task Class look also the same as this Class that's why i ll only upload this Code. in the main Class i make a Connection to rabbitmq and pass it as Parameter to the constructor of the Two Task Classes.

class On_Deregistration:

def __init__(self, rabbit):
   self.event(rabbit) # this will call event function and pass the connection shared between all Tasks. rabbit parameter hold a connection to rabbitmq

def event(self, rabbit):
 
    self.Subscriber = rabbit.EventSubscriber(rabbit,  'testing.test',  'test', False,  onDeregistrationFromHRS # this func is task listener)

def subscribeAsync(self):
    self.Subscriber.subscribe() # here i call start_consuming

def start(self):
    """start Subscribtion in an Independant Thread  """
    thread = threading.Thread(target = self.subscribeAsync )  
    thread.start()
    if thread.isAlive():
        print("asynchronous subscription started")

---

MAin Class:

class App:

def __init__(self):

    self.rabbitMq = RabbitMqCommunicationInterface(host='localhost', port=5672)
    firstTask =  On_Deregistration(self.rabbitMq)
    secondTask =  secondTask(self.rabbitMq)

app = App()

error : 'start_consuming may not be called from the scope of '

pika.exceptions.RecursionError: start_consuming may not be called from the scope of another BlockingConnection or BlockingChannel callback

I searched for the cause of this Error and obviously is pika not thread safe but there must be a Solution for this. maybe Not using a BlockingConnection ? maybe someone can give me an Example how to do that because i tried it and didnt work. Maybe I'm missing something about how to Implement multithreading with rabbitmq

Answer 1

so after a long research, I figure out that Pika is not thread safe. well for the moment at least, maybe in new versions it will be thread safe. so now for my Project I stopped using Pika and I'm using b-rabbit , which is a thread safe wrapper around Rabbitpy. but I must say that Pika is a great Library and I find the API better described and structured than rabbitpy but for my Project it was mandatory to use multithreading and that's why Pika for the moment was a bad choice. I hope this helps someone in the Future