Class Loading Issue when Running Gemfire 7.x OQL in Weblogic 10.3.x

I have a very simple OQL like "SELECT * FROM /FxLimitsRegion" that ran successfully in my unit/integration test. But I got the following exception when trying to run it inside my WebLogic EAR (our Gemfire cache and Weblogic application server run in the same JVM):

com.gemstone.gemfire.cache.query.QueryInvalidException: Syntax error in query:  Invalid class or can't make instance, com.gemstone.gemfire.cache.query.internal.parse.ASTSelect
    at com.gemstone.gemfire.cache.query.internal.QCompiler.compileQuery(QCompiler.java:80)
    at com.gemstone.gemfire.cache.query.internal.DefaultQuery.(DefaultQuery.java:156)
    at com.gemstone.gemfire.cache.query.internal.DefaultQueryService.newQuery(DefaultQueryService.java:107)
    at com.gemstone.gemfire.internal.cache.LocalDataSet.query(LocalDataSet.java:129)
    at com.jpmorgan.gcrm.sef.business.cache.impl.SEFCacheManagerGemfireImpl.queryPartitionedRegionLocally(SEFCacheManagerGemfireImpl.java:187)
    at com.jpmorgan.gcrm.fno.business.datamanager.impl.FxDataManager.selectActiveFxLimits(FxDataManager.java:57)
    ... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid class or can't make instance, com.gemstone.gemfire.cache.query.internal.parse.ASTSelect
    at antlr.ASTFactory.createUsingCtor(ASTFactory.java:251)
    at antlr.ASTFactory.create(ASTFactory.java:210)
    at com.gemstone.gemfire.cache.query.internal.parse.OQLParser.selectExpr(OQLParser.java:1090)
    at com.gemstone.gemfire.cache.query.internal.parse.OQLParser.query(OQLParser.java:217)
    at com.gemstone.gemfire.cache.query.internal.parse.OQLParser.queryProgram(OQLParser.java:115)
    at com.gemstone.gemfire.cache.query.internal.QCompiler.compileQuery(QCompiler.java:76)

The class "com.gemstone.gemfire.cache.query.internal.parse.ASTSelect" to load was included in the gemfire.jar in my EAR. So the exception seems to be confusing in the beginning.
From the above stack trace, you can see it relates to the ANTLR jar. Both my EAR and Weblogic itself have ANTLR. But my EAR uses version 2.7.7 while Weblogic 10.3.2 uses a pretty old version.
So we need to know which ANTLR version was used. Here is the call workflow:
1. my EAR thread issued a OQL ->
2. Gemfire found  ANTLR to parse the OQL to AST ->
3. ANTLR tried to load Gemfire class "com.gemstone.gemfire.cache.query.internal.parse.ASTSelect"

By default, Step2 found the Weblogic ANTLR by the Weblogic system class loader. But the Gemfire class is in the child EAR classloader. So Step 3 threw the exception.
If we can tell Weblogic to use my EAR ANTLR, the exception should be gone. In order to do so, I just added the following excerpt to your weblogic-application.xml:
         <prefer-application-packages>
               <package-name>antlr.*</package-name>
         </prefer-application-packages>
New version of ANTLR changed to load classes from the thread context class loader by default which was the EAR classloader in my case. So if the Weblogic had new version of ANTLR, the exception should have not been there.

Connecting to Websphere MQ from Weblogic using Certificate

Here is the requirement: we need to connect to a remote Websphere MQ sever (v7.x) from a Weblogic servier (v10.3) using a certifcate by configuring MQ destination and connection factories in a JMS foreign server.
The difficulty thing is how to authenticate the JMS client on Weblogic with the MQ server using a SSL certficate.

Originally I though I could configure my keystore on the "keystore" and "SSL" tabs of a Weblogic service. But this option threw the following exception:

[EJB:011014]The Message-Driven EJB failed while creating a JMS Connection. The error was:

com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name 'NAUMIB1.jpmchase.net(1414)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.

Nested exception: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').

Nested exception: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'NAUMIB1.jpmchase.net(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target],3=NAUMIB1.jpmchase.net/169.69.169.122:1414 (NAUMIB1.jpmchase.net),4=SSLSocket.startHandshake,5=default]],3=NAUMIB1.jpmchase.net(1414),5=RemoteTCPConnection.protocolConnect]

Nested exception: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target],3=NAUMIB1.jpmchase.net/169.69.169.122:1414 (NAUMIB1.jpmchase.net),4=SSLSocket.startHandshake,5=default]

Nested exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Nested exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Nested exception: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name 'NAUMIB1.jpmchase.net(1414)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.

       at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)

       at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)

       at com.ibm.msg.client.wmq.internal.WMQConnection.(WMQConnection.java:440)

       at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:7062)

       at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6453)

       at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:295)

       at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6230)

       at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:115)

       at weblogic.ejb.container.internal.JMSConnectionPoller.getConnection(JMSConnectionPoller.java:473)

       at weblogic.ejb.container.internal.JMSConnectionPoller.createJMSConnection(JMSConnectionPoller.java:1962)

       at weblogic.ejb.container.internal.JMSConnectionPoller.connect(JMSConnectionPoller.java:780)

       at weblogic.ejb.container.internal.MDConnectionManager.startConnectionPolling(MDConnectionManager.java:241)

       at weblogic.ejb.container.manager.MessageDrivenManager.start(MessageDrivenManager.java:584)

       at weblogic.ejb.container.deployer.MessageDrivenBeanInfoImpl.mdManagerStart(MessageDrivenBeanInfoImpl.java:1174)

       at weblogic.ejb.container.deployer.MessageDrivenBeanInfoImpl.deployMessageDrivenBeans(MessageDrivenBeanInfoImpl.java:1154)

       at weblogic.ejb.container.deployer.EJBDeployer.deployMessageDrivenBeans(EJBDeployer.java:1675)

       at weblogic.ejb.container.deployer.EJBDeployer.deployMessageDrivenBeansUsingModuleCL(EJBDeployer.java:1450)

       at weblogic.ejb.container.deployer.MDBServiceImpl.startService(MDBServiceImpl.java:69)

       at weblogic.server.ServiceActivator.start(ServiceActivator.java:98)

       at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

       at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)

       at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)

       at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').

       at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)

       ... 21 more

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'NAUMIB1.jpmchase.net(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target],3=NAUMIB1.jpmchase.net/169.69.169.122:1414 (NAUMIB1.jpmchase.net),4=SSLSocket.startHandshake,5=default]],3=NAUMIB1.jpmchase.net(1414),5=RemoteTCPConnection.protocolConnect]

       at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:2010)

       at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1227)

       at com.ibm.msg.client.wmq.internal.WMQConnection.(WMQConnection.java:355)

       ... 20 more

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target],3=NAUMIB1.jpmchase.net/169.69.169.122:1414 (NAUMIB1.jpmchase.net),4=SSLSocket.startHandshake,5=default]

       at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1020)

       at com.ibm.mq.jmqi.remote.internal.system.RemoteConnection.connect(RemoteConnection.java:1112)

       at com.ibm.mq.jmqi.remote.internal.system.RemoteConnectionPool.getConnection(RemoteConnectionPool.java:350)

       at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1599)

       ... 22 more

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1611)

       at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)

       at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)

       at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1035)

       at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:124)

       at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)

       at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)

       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)

       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1112)

       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1139)

       at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)

       at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1005)

       at java.security.AccessController.doPrivileged(Native Method)

       at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1000)

       ... 25 more

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:294)

       at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:200)

       at sun.security.validator.Validator.validate(Validator.java:218)

       at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)

       at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)

       at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)

       at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1014)

       ... 35 more

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)

       at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)

       at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:289)

       ... 41 more

The above exception is complaining about the CA's used in the MQ server certificate are not available in my keystore. But I am sure I have these CA's in the keystore.

The real problem is the keystore you configured on Weblogic server "keystore" tab is only recognizable when the Weblogic server DIRECTLY connects to an SSL server.

In my case, my JMS MDB on Weblogic connects to the MQ server through IBM MQ API. So Weblogic is not directly connecting to the MQ server. You can see ths from the IBM packages "com.ibm.msg.client"  and "com.ibm.mq.jmqi" in the above exception.

Finally in order to connect successfully, you just need to define the following 4 properties on the Weblogic startup cmd line:-D javax.net.ssl.trustStore=
-D javax.net.ssl.trustStorePassword=
-D javax.net.ssl.keyStore=
-D javax.net.ssl.keyStorePassword=