Issue with Iterator Mediator in WSO2 ESB

Question

I have an scenario like this:

1. A proxy service in ESB do a call to a service and receive a payload, between 2.3 and 4.9MB.
2. This payload have a list of Persons.
3. With this list I use the iterator mediator to iterate over it and inside each element I call a DBReport mediator to store the values inside the Person element.

When the payload have 2.3MB all work fine and the data is store in my DB in a few minutes, about 5000 elements.

But with the 4.9MB payload it´s really slowly as you can see rapidly the RAM take 2.7 of 3.0GB max. At that moment the JMX monitoring frozen and just 123 record of 10 000 elements in 3minutes was stored in DB

After a while the processing continue and stop for a while and you can see the delay between each DBReport call:

TID: [0] [ESB] [2013-06-12 16:00:00,480]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:6a11bc08-8e1f-4c37-a031-fe9c84e395da, Direction: response, TEST = 06469 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:00:09,591]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:439fa810-3333-4cbf-bb68-08eb843ca408, Direction: response, TEST = 06471 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:00:15,555]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:24b5dacd-53dd-4d78-8f57-24944cdf2a84, Direction: response, TEST = 06475 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:01:33,420]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:2a8c77a9-ae60-4f84-901d-3b4f76a6b480, Direction: response, TEST = 06477 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:01:41,489]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:1f0632c3-91ee-4fae-b47e-2c88b73aed69, Direction: response, TEST = 06478 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:02:04,190]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:02b2b0ae-3d52-48c9-ad09-04e983501e3d, Direction: response, TEST = 06505 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:05,620]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:ae2b72d2-124d-4f34-8033-9036a2b0419a, Direction: response, TEST = 06507 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:07,411]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:ac951f44-c212-4a4d-93c2-9473677acf32, Direction: response, TEST = 06511 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:14,232]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:99e2804e-f70b-449f-9b5f-46f65d81638b, Direction: response, TEST = 06514 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:15,757]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:b0f1dfca-2eda-415c-9c11-3ea70168044e, Direction: response, TEST = 06516 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:11:57,247]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:9d92ecc9-3b70-440d-8195-c743da44ccd5, Direction: response, TEST = 06517 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:12:13,862]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:fac67fed-989c-44e2-91cb-c2b3b40460ee, Direction: response, TEST = 06522 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:12:28,531]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:635d74b9-23ae-435f-909e-c2c6ae171407, Direction: response, TEST = 06523 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:12:45,257]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     SOAPAction: , MessageID: urn:uuid:59a006e2-e2ef-4241-9439-10d9db5830c3, Direction: response, TEST = 06524 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:19:46,585]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/an                     onymous, WSAction: , SOAPAction: , MessageID: urn:uuid:5d867db0-3e88-4798-9360-0f9a49b9c5ae, Direction: response, TEST = 06525 {org.apache.synap                     se.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:20:06,866]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:ad48e642-eeb0-49ef-a25f-63f2f10534c0, Direction: response, TEST = 06530 {org.apache.synapse.mediators.builtin.LogMediator}

Any idea?

Answer 1

DBReport mediator is executed synchronously in WSO2 ESB which means the thread that execute this DBReport mediator is stucked until the Database operation is completed. That means the performance is lower than when the execution happens asynchronously.

Therefore to gain the maximum performance, please use WSO2 Data Services server and use DSS to execute the insert Database operation. That way you can gain the maximum performance.

It is better if you can add your configuration files. That will help to identify the root cause of this issue.