Error zookeeper SSL when execute zookeeper shell

Question

i configured ssl in zookeeper, when i see log tls runs fine, but when i run zookeeper shell suddenly error and exit this log

this log after execute zookeeper shell: [2022-11-08 03:47:16,761] WARN zookeeper.ssl.keyStore.location not specified (org.apache.zookeeper.common.X509Util) [2022-11-08 03:47:16,762] WARN zookeeper.ssl.trustStore.location not specified (org.apache.zookeeper.common.X509Util) [2022-11-08 03:47:16,886] ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [2022-11-08 03:47:16,900] WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) at io.netty.c hannel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) at io.netty.ch annel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) at io.netty.handler.codec.B yteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)... 17 more [2022-11-08 03:47:18,617] ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [2022-11-08 03:47:18,622] WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.ch annel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) at io.netty.channel.nio.NioEventLo op.processSelectedKeys(NioEventLoop.java:584) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) at io.netty.handler.codec.B yteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)

this log before execute zookeeper sheell
[2022-11-08 03:44:09,105] WARN Send worker leaving thread id 3 my id = 2 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2022-11-08 03:44:09,131] INFO Accepted TLS connection from /192.168.30.232:52204 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket)
[2022-11-08 03:44:09,141] INFO Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x12, n.peerEpoch:0x15, n.zxid:0x1400000002, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2022-11-08 03:44:09,141] INFO Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x12, n.peerEpoch:0x15, n.zxid:0x1400000002, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2022-11-08 03:44:09,142] INFO Peer state changed: following (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,142] INFO FOLLOWING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,148] INFO leaderConnectDelayDuringRetryMs: 100 (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,148] INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,148] INFO zookeeper.learner.asyncSending = false (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,150] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2022-11-08 03:44:09,151] INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-11-08 03:44:09,153] INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-11-08 03:44:09,155] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2022-11-08 03:44:09,155] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2022-11-08 03:44:09,156] INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-11-08 03:44:09,156] INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-11-08 03:44:09,156] INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-11-08 03:44:09,156] INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-11-08 03:44:09,156] INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-11-08 03:44:09,156] INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-11-08 03:44:09,158] INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-11-08 03:44:09,158] INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-11-08 03:44:09,158] INFO Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 clientPortListenBacklog -1 datadir /var/log/zookeeper-logs/version-2 snapdir /var/log/zookeeper-logs/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-11-08 03:44:09,159] INFO FOLLOWING - LEADER ELECTION TOOK - 575 MS (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,162] INFO Peer state changed: following - discovery (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,236] INFO Successfully connected to leader, using address: /192.168.30.232:2888 (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,273] INFO Peer state changed: following - synchronization (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,276] INFO Getting a diff from the leader 0x1400000002 (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,276] INFO Peer state changed: following - synchronization - diff (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,280] INFO Learner received NEWLEADER message (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,280] INFO Dynamic reconfig is disabled, we don't store the last seen config. (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,314] INFO Peer state changed: following - synchronization (org.apache.zookeeper.server.quorum.QuorumPeer)
[2022-11-08 03:44:09,320] INFO Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1 (org.apache.zookeeper.server.quorum.CommitProcessor)
[2022-11-08 03:44:09,320] INFO Configuring CommitProcessor with 2 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor)
[2022-11-08 03:44:09,325] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2022-11-08 03:44:09,342] INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner)
[2022-11-08 03:44:09,343] INFO Peer state changed: following - broadcast (org.apache.zookeeper.server.quorum.QuorumPeer)


i have change CA, keystore, trustore but this can't help me

Answer 1

There is an issue in your Zk SSL settings. Mine is like this and SSL works:

secureClientPort=12181
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.protocol=TLSv1.2
ssl.trustStore.location=XXX
ssl.trustStore.password=XXX
ssl.keyStore.location=XXX
ssl.keyStore.password=XXX
ssl.clientAuth=none

You can find more details here and here .