超过每个分区允许的最大接收器数量 eventthub 中的错误

Question

Like the code below, I am continuously receiving data from azure eventhub. Often I am seeing the error saying "Exceeded the maximum number of allowed receivers per partition" which I know where it comes from.

import os
import sys
import logging
import time
from azure.eventhub import EventHubClient, Receiver, Offset

logger = logging.getLogger("azure")

ADDRESS = ""
USER = ""
KEY = ""

CONSUMER_GROUP = "$default"
OFFSET = Offset("@latest")
PARTITION = "0"

total = 0
last_sn = -1
last_offset = "-1"
client = EventHubClient(ADDRESS, debug=False, username=USER, password=KEY)

receiver = client.add_receiver(CONSUMER_GROUP, PARTITION, prefetch=0, offset=OFFSET)    
client.run()
start_time = time.time()
while True:
    for event_data in receiver.receive(timeout=5000):
        print("Received: {}".format(event_data.body_as_str(encoding='UTF-8')))
        a = event_data.body_as_str(encoding='UTF-8')
        total += 1
    end_time = time.time()
    run_time = end_time - start_time
    print("Received {} messages in {} seconds".format(total, run_time))

The line here is the one that add receiver and if I add more than five reciever, it hits the limit of the possible number of receiver per partition.

receiver = client.add_receiver(CONSUMER_GROUP, PARTITION, prefetch=0

1. So, I tried to remove receiver with functions in it. like using receiver.client.clients.remove or receiver.client.clients.clear() , to clear the receivers added in it before. However, none of these methods seems to work.

The reason why I see this error is that because I run the whole code above whenever I need to stop running script to debug, so whenever I re-run it, I have to run the line receiver = client.add_receiver(CONSUMER_GROUP, PARTITION, prefetch=0

2. I also tried to run only part of the code below "add_receiver" line

client.run()
start_time = time.time()
while True:
    for event_data in receiver.receive(timeout=5000):
        print("Received: {}".format(event_data.body_as_str(encoding='UTF-8')))
        a = event_data.body_as_str(encoding='UTF-8')
        total += 1
    end_time = time.time()
    run_time = end_time - start_time
    print("Received {} messages in {} seconds".format(total, run_time))

However, I see another error saying that EventHubError: This receive handler is now closed.

Any possible ways to fix this issue?

Answer 1

After struggling, I think this may be a solution to this issue.

For the line 22 above, I could simplly add 'keep_alive' input to it, something like:

receiver = client.add_receiver(CONSUMER_GROUP, PARTITION, prefetch=0, offset=OFFSET, keep_alive = 10000000)  

In this way, I can leave the client receiver opened and not get the error message "EventHubError: This receive handler is now closed.", then simply run the only part from below:

start_time = time.time()
while True:
    for event_data in receiver.receive(timeout=5000):
        print("Received: {}".format(event_data.body_as_str(encoding='UTF-8')))
        a = event_data.body_as_str(encoding='UTF-8')
        total += 1
    end_time = time.time()
    run_time = end_time - start_time
    print("Received {} messages in {} seconds".format(total, run_time))