vhost in rabbitmq is not starting

Question

My rabbitmq was working fine. Suddenly, one of the vhosts is not starting up. On restarting rabbitmq server, it shows in admin UI, that error is starting up vhost, and all the queues are in down state. Here is the error when I restart the vhost. Please suggest. Our production message broker is down, need immediate help. Rabbitmq ver. - 3.8.3 Erlang ver. - 22.3

Trying to restart vhost 'r_t' on node 'rabbit@myserver' ...
Error:
Failed to start vhost 'r_t' on node 'rabbit@myserver'Reason: {:shutdown, {:failed_to_start_child, 
:rabbit_vhost_process, {:error, {{{:badarg, [{:erlang, :binary_to_term, [<<131, 104, 6, 100, 0, 13, 98, 
97, 115, 105, 99, 95, 109, 101, 115, 115, 97, 103, 101, 104, 4, 100, 0, 8, 114, 101, 115, 111, 117, 114,
99, 101, 109, 0, ...>>], []}, {:rabbit_queue_index, :parse_pub_record_body, 2, [file: 
'src/rabbit_queue_index.erl', line: 783]}, {:rabbit_queue_index, :"-segment_entries_foldr/3-fun-0-", 4, 
[file: 'src/rabbit_queue_index.erl', line: 1111]}, {:array, :sparse_foldr_3, 6, [file: 'array.erl', line:
1847]}, {:array, :sparse_foldr_2, 8, [file: 'array.erl', line: 1836]}, {:rabbit_queue_index, 
:scan_queue_segments, 3, [file: 'src/rabbit_queue_index.erl', line: 741]}, {:rabbit_queue_index, 
:queue_index_walker_reader, 2, [file: 'src/rabbit_queue_index.erl', line: 728]}, {:rabbit_queue_index, 
:"-queue_index_walker/1-fun-1-", 2, [file: 'src/rabbit_queue_index.erl', line: 710]}]}, {:gen_server2, 
:call, [#PID<10691.1882.0>, :out, :infinity]}}, {:child, :undefined, :msg_store_persistent, 
{:rabbit_msg_store, :start_link, [:msg_store_persistent, 
'/var/lib/rabbitmq/mnesia/rabbit@1myserver/msg_stores/vhosts/1SLGRHB3T7STV1U1TEB4MR6QS', [], 
{#Function<2.23124100/1 in :rabbit_queue_index>, {:start, [{:resource, "r_t", :queue, 
"product.import_royn_se"}, {:resource, "r_t", :queue, "customer.import_ronin_es"},  {:resource, "r_t", 
...}, {:resource, ...}, {...}, ...]}}]}, :transient, 30000, :worker, [:rabbit_msg_store]}}}}}

Answer 1

I got a workaround. We just exported and saved vhost definitions for existing vhost which was not starting. The deleted this vhost. Created same vhost and imported the definition back. So, we got all queues with same features.

Answer 2

-> For dump all messages

 from os import listdir from os.path import isfile, join import re base_path = '/docker/rabbitmq-nfe/data/mnesia/rabbit@rabbitmq_nfe/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues' list_queues = listdir(base_path) #GET ALL QUEUES for queue in list_queues: dir_queue = '%s/%s' % (base_path, queue) list_files = [f for f in listdir(dir_queue) if isfile(join(dir_queue, f)) and f not in ['.queue_name', 'journal.jif']] list_files.sort() name_queue = open('%s/.queue_name' % (dir_queue), 'r').read().split('QUEUE: ')[-1][:-1] payload_queue = [] #GET ALL FILES QUEUES for file in list_files: path_file = '%s/%s' % (dir_queue, file) binary_file = open(path_file, 'r') string_file = binary_file.read() string_file_decoded = string_file.decode('iso 8859-1') #GET ALL PAYLOAD FILES QUEUES list_payload_queue = ['{%s}' % f for f in re.split('\{(.*?)\}', string_file_decoded) if 'chave' in f and 'rede' in f] for idx, payload in enumerate(list_payload_queue): if payload.count('{') > 1: list_payload_queue[idx] = ['{%s' % f for f in payload.split('{') if 'chave' in f and 'rede' in f][0] payload_queue = payload_queue + list_payload_queue #SAVE BACKUP QUEUES print('FILA: %s ARQUIVOS: %s' % (name_queue, str(len(payload_queue)))) with open('/tmp/%s.log' % name_queue, 'w') as f: for line in payload_queue: f.write("%s\n" % line)