当有大量消息排队时，从MSMQ读取速度会变慢

Question

Short introduction

I have a SEDA based system, and used MSMQ for communication (event triggering) between the different applications/services.

One of these services gets messages by file, so I have a file listener that reads the file content and inserts this into a queue (or actually 4 different queues, but that's not very important for the first question).

Server is Windows Server 2008

First question - read slows down

My application that reads these messages at the other side normally reads about 20 messages from the queue per second, but when the service that posts messages start queuing some thousand messages, the read goes down, and the read application only reads 2-4 messages per second. When there is no posting to the queue, the read application can again read up to 20 messages per second.

The code in the reading application is pretty simple, developed in C#, I use the Read(TimeSpan timeout) function in System.Messaging.

Q: Why does the read slows down when there is a lot of messages posted to the queue?

Second question - limitations of TPS

An additional question is about the read itself. It seems there is no difference in how many messages I can read per second if I use 1 or 5 threads to read from the queue. I've also tried implementing a "round robin solution" where the post service are posting to a random set of 4 queues, and the read application had one thread listening to each of these queues, but there is still only 20 TPS even if I read from 1 queue with 1 thread, 1 queue with 4 threads or 4 queues (with one thread per queue).

I know the processing in the thread takes about 50 ms, so 20 TPS is quite correct if there is only one message processed at the time, but the clue with multi threading should be that messages are handled in parallel and not sequential.

There is about 110 different queues on the server.

Q: Why can't I get more than 20 messages out of my queue at the time even with multi threading and the use of several queues?

This is the code running today:

// There are 4 BackgroundWorkers running this function
void bw_DoWork(object sender, DoWorkEventArgs e) 
{
    using(var mq = new MessageQueue(".\\content"))
    {
        mq.Formatter = new BinaryMessageFormatter();

        // ShouldIRun is a bool set to false by OnStop()
        while(ShouldIRun)
        {
            try
            {
                using(var msg = mq.Receive(new TimeSpan(0,0,2))
                {
                    ProcessMessageBody(msg.Body); // This takes 50 ms to complete
                }
            }
            catch(MessageQueueException mqe)
            {
               // This occurs every time TimeSpan in Receive() is reached
               if(mqe.MessageQueueErrorCode == MessageQueueErrorCode.IOTimeout) 
                   continue;
            }
        }
    }

But even if there are 4 threads, it seems all waits for the function to enter the "Receive" point again. I've also tried using 4 different queues (content1, content2, content3 and content4), but still i get 1 message processed every 50 ms.

Does this have anything to do with the TimeSpan in Receive(), and/or is it possible to omit this?

Another question is if the use of private queues, instad of public will solve anything?

Answer 1

Performance issues.
You don't mention if all the code is running on the server or if you have clients remotely accessing the queues on the server. From the speed, I'll assume the latter.
Also, are the queues transactional?
How large are the messages?

If you want to read a message from a queue, your application does not connect to the queue itself. Everything goes between the local queue manager and the remote queue manager. The queue manager is the only process that writes to, and reads from queues. Therefore having multiple queues or a single queue won't necessarily perform any differently.

The MSMQ queue manager is therefore going to be a bottleneck at some point as there is only so much work it can do at the same time. Your first question shows this - when you put a high load on the queue manager putting messages IN, your ability to take messages OUT slows down. I'd recommend looking at performance monitor to see if MQSVC.EXE is maxed out, for example.

Answer 2

Why are you using timespan? - that is a bad thing and here is why.

When developing services and queue you need to program in a theadsafe manner. Each item in the queue will spawn a new thread. Using timespan is forcing each of the threads to use a single timer event thread. These events are having to wait for their turn at the event thread.

The norm is 1 thread per queue events - This is generally your System.Messaging.ReceiveCompletedEventArgs event. Another thread is your onStart event...

20 threads or 20 reads per second is probably correct. Generally when thread pooling you can only spawn 36 threads at a time in .net.

My advice is drop the timer event an make your queue simply process the data.

do something more like this;

namespace MessageService 
{ 

public partial class MessageService : ServiceBase 

{ 

    public MessageService() 

    { 

        InitializeComponent(); 

    } 



    private string MessageDirectory = ConfigurationManager.AppSettings["MessageDirectory"]; 

    private string MessageQueue = ConfigurationManager.AppSettings["MessageQueue"]; 



    private System.Messaging.MessageQueue messageQueue = null; 



    private ManualResetEvent manualResetEvent = new ManualResetEvent(true); 





    protected override void OnStart(string[] args) 

    { 

        // Create directories if needed 

        if (!System.IO.Directory.Exists(MessageDirectory)) 

            System.IO.Directory.CreateDirectory(MessageDirectory); 



        // Create new message queue instance 

        messageQueue = new System.Messaging.MessageQueue(MessageQueue); 



        try 

        {    

            // Set formatter to allow ASCII text 

            messageQueue.Formatter = new System.Messaging.ActiveXMessageFormatter(); 

            // Assign event handler when message is received 

            messageQueue.ReceiveCompleted += 

                new System.Messaging.ReceiveCompletedEventHandler(messageQueue_ReceiveCompleted); 

            // Start listening 



            messageQueue.BeginReceive(); 

        } 

        catch (Exception e) 

        { 



        } 

    } 



    protected override void OnStop() 

    { 

        //Make process synchronous before closing the queue 

        manualResetEvent.WaitOne(); 





        // Clean up 

        if (this.messageQueue != null) 

        { 

            this.messageQueue.Close(); 

            this.messageQueue = null; 

        } 

    } 



    public void messageQueue_ReceiveCompleted(object sender, System.Messaging.ReceiveCompletedEventArgs e) 

    { 

        manualResetEvent.Reset(); 

        System.Messaging.Message completeMessage = null; 

        System.IO.FileStream fileStream = null; 

        System.IO.StreamWriter streamWriter = null; 

        string fileName = null; 

        byte[] bytes = new byte[2500000]; 

        string xmlstr = string.Empty;                

            try 

            { 

                // Receive the message 

                completeMessage = this.messageQueue.EndReceive(e.AsyncResult);                    

                completeMessage.BodyStream.Read(bytes, 0, bytes.Length); 



                System.Text.ASCIIEncoding ascii = new System.Text.ASCIIEncoding(); 



                long len = completeMessage.BodyStream.Length; 

                int intlen = Convert.ToInt32(len);                   

                xmlstr = ascii.GetString(bytes, 0, intlen);                   

            } 

            catch (Exception ex0) 

            { 

                //Error converting message to string                    

            } 

        }