wso2 ESB-Websphere MQ JMS失去连接

Question

WSO2 ESB sometimes lost JMS connection to Websphere MQ. Establishing new connection last at least 1-2 seconds. This cause timeouts in proxy service using this connection.

WSO2 ESB 4.8.1, IBM Webpshere MQ 7.0.1.12

Strange stacktrace:

2014-12-18 06:47:51,183 [-] [JMSCCThreadPoolWorker-7] ERROR ServiceTaskManager JMS Connection failed : JMSWMQ1107: A problem with this connection has occurred. - shutting down worker tasks
2014-12-18 06:47:52,212 [-] [JMSCCThreadPoolWorker-7] ERROR ServiceTaskManager Error closing shared Connection
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0019: Failed to disconnect from queue manager 'name' using connection mode '1' and host name 'host(1441)'. Please see the linked exception for more information.
    at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
    at com.ibm.msg.client.wmq.internal.WMQConnection.close(WMQConnection.java:742)
    at com.ibm.msg.client.jms.internal.JmsConnectionImpl.close(JmsConnectionImpl.java:352)
    at com.ibm.mq.jms.MQConnection.close(MQConnection.java:93)
    at org.apache.axis2.transport.jms.ServiceTaskManager.stop(ServiceTaskManager.java:237)
    at org.apache.axis2.transport.jms.ServiceTaskManager.start(ServiceTaskManager.java:166)
    at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.onException(ServiceTaskManager.java:631)
    at com.ibm.msg.client.jms.internal.JmsProviderExceptionListener.run(JmsProviderExceptionListener.java:429)
    at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)
    at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:100)
    at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)
    at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:298)
    at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1220)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
    ... 12 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009
    at com.ibm.mq.jmqi.remote.internal.RemoteHconn.enterCall(RemoteHconn.java:428)
    at com.ibm.mq.jmqi.remote.internal.RemoteHconn.enterCall(RemoteHconn.java:350)
    at com.ibm.mq.jmqi.remote.internal.RemoteHconn.enterCall(RemoteHconn.java:325)
    at com.ibm.mq.jmqi.remote.internal.RemoteFAP.MQDISC(RemoteFAP.java:2541)
    at com.ibm.msg.client.wmq.internal.WMQConnection.close(WMQConnection.java:724)
    ... 11 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9213: A communications error for 'TCP' occurred. [1=java.net.SocketException[Unrecognized Windows Sockets error: 0: recv failed],4=TCP,5=sockInStream.read]
    at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.receive(RemoteTCPConnection.java:1418)
    at com.ibm.mq.jmqi.remote.internal.RemoteRcvThread.receiveBuffer(RemoteRcvThread.java:737)
    at com.ibm.mq.jmqi.remote.internal.RemoteRcvThread.receiveOneTSH(RemoteRcvThread.java:701)
    at com.ibm.mq.jmqi.remote.internal.RemoteRcvThread.run(RemoteRcvThread.java:146)
    ... 5 more
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.receive(RemoteTCPConnection.java:1409)
    ... 8 more
2014-12-18 06:49:53,302 [-] [DefaultQuartzScheduler_Worker-1] ERROR MessageInjector Proxy Service: monitoring_Service not found
2014-12-18 06:49:53,302 [-] [DefaultQuartzScheduler_Worker-1] ERROR JobRunShell Job synapse.simple.quartz.monitoring_Task threw an unhandled Exception: 
org.apache.synapse.SynapseException: Proxy Service: monitoring_Service not found
    at org.apache.synapse.startup.tasks.MessageInjector.handleError(MessageInjector.java:333)
    at org.apache.synapse.startup.tasks.MessageInjector.execute(MessageInjector.java:217)
    at org.apache.synapse.startup.quartz.SimpleQuartzJob.execute(SimpleQuartzJob.java:88)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
2014-12-18 06:49:53,302 [-] [DefaultQuartzScheduler_Worker-1] ERROR ErrorLogger Job (synapse.simple.quartz.monitoring_Task threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.apache.synapse.SynapseException: Proxy Service: monitoring_Service not found]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: org.apache.synapse.SynapseException: Proxy Service: monitoring_Service not found
    at org.apache.synapse.startup.tasks.MessageInjector.handleError(MessageInjector.java:333)
    at org.apache.synapse.startup.tasks.MessageInjector.execute(MessageInjector.java:217)
    at org.apache.synapse.startup.quartz.SimpleQuartzJob.execute(SimpleQuartzJob.java:88)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    ... 1 more

Answer 1

The key error you're seeing within the stacktrace is:

Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122)

Which indicates that the JVM received something it wasn't expecting over a Windows socket. It looks like you're using an Oracle JVM rather than the IBM one shipped with MQ.

I'd suggest you try upgrading to a newer JVM if one is available. Note that the MQ 7.0.1 client you're using supports Java 5 and Java 6. If you want to use Java 7 you'll need to use an MQ client of 7.1 or later. All IBM MQ clients are available free of charge via the MQC support packs:

MQ 7.1 clients: http://www-01.ibm.com/support/docview.wss?uid=swg24031412
MQ 7.5 clients: http://www-01.ibm.com/support/docview.wss?uid=swg24032744
MQ 8.0 clients: http://www-01.ibm.com/support/docview.wss?uid=swg24037500

If a newer JVM isn't available from Oracle, I'd suggest opening a support ticket with them so they can investigate.