生产者 - 消费者的Java多线程

Question

I am following a video tutorial on Java multi-threading. It introduces on using Java to implement the famous "Producer-consumer" problem.

It used wait() and notifyAll() to ensure the proper communication between producer threads and consumer threads.

The tutor intentionally created several producer threads while only one consumer threads, but he left a question unanswered: "It is always best practice to have equal number of producer and consumer threads, if there are more producer threads than consumer, there will be problems".

However, he didn't specify what that problem is. I personally imagine that would only be a situation that the basket is full. Could experts help here? Thanks.

Answer 1

You could have any number of producers and any number of consumers.

If the producers produce too fast, the queue will fill up, until you get memory problems or the producers are forced to stay idle until there's some place left in the queue.

If the consumers consume too fast, they will stay idle most of the time.

Answer 2

It is always best practice to have equal number of producers and consumer threads

I personally dont agree with this.

How many producers and how consumer threads to have entirely depends on the situation. There is no straight forward rule to that.

I will tell you my case,

We had a program which was reading the CSV file,creating XML from it and publishing those XMLs on JMS Queue. This program was single threaded in the begining but the performance was not upto the mark.We found that publishing message to queue was taking much of time.So we decided to introduce multithreading.

We Devided the logic in 2 threads-

1. Producer

   Reads the CSV and creates XML

2. Consumer

   Sends the XML to JMS queue.

We decided to have only one producer, because we did not wanted to open multiple handles for the file though its possible. On the other hand, we kept the number of consumer threads to be created configurable and it worked fine and we got expected performance gain.

To summarize, you need make sure that the speed of producer threads and consumer threads is almost matching.Otherwise if consumer thread is slower than producer, the records will pile up in memory(RAM) which can cause OutOfMemory in extreame cases.

Answer 3

I don't see any issue besides the matter of Producer efficiency. If you have more Producers than Consumers, and they all produce/consume at the same rate, the extra Producers will effectively be idle.

See What's best value for make -j , for example, where you're recommended to give n+1 jobs to your n cores, so that they're always under full load.

Answer 4

Well, the problem is the optimum use of resources namely the CPU and the memory. As you rightly guessed, the basket would be full (Hence more memory consumed), the producer thread would be idle for while till the consumer empties the basket for new tasks which will result into underutilization of CPU.

If there are equal number of producer and consumer, both the thread groups (producer and consumer) will be busy in their task and the basket would also have a steady storage.

Now, too large or too less producer/consumers will also be a problem. The large number of treads will hog the CPU and hence causing other problem (slowing etc). So there is always an optimum number which depend on the hardware resources available on the machine.

Answer 5

@JB Nizet summarized the situation pretty well, in that you can have too many producers or too many consumers. The determining factor is simply the rate at which producers produce versus the rate at which consumers consume. In other words, if it takes much more time to produce a resource than to consume it, than it is best to have more producers than consumers. If, however, it takes more time to consume a resource than produce it, then it is best to have more consumers than producers.

The tutor is probably assuming that the consumption process is more efficient than the production process, which may be true in many or possibly even most cases, but it's not necessarily a good assumption to make. Just try to keep this in mind when designing producer/consumer systems.

Answer 6

I think it depends not on consumer/producer numbers but speed first of all. You may have 1 consumer which can process 1 message in 10 ms and 2 producers producing 1 message in 100 ms each.