{"id":2366,"date":"2022-10-07T20:30:43","date_gmt":"2022-10-07T20:30:43","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=2366"},"modified":"2022-10-07T20:31:12","modified_gmt":"2022-10-07T20:31:12","slug":"zookeeper-mtls","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2022\/10\/07\/zookeeper-mtls\/","title":{"rendered":"Zookeeper mTLS"},"content":{"rendered":"\n<figure class=\"wp-block-pullquote has-white-color has-luminous-vivid-orange-to-vivid-red-gradient-background has-text-color has-background has-small-font-size\" style=\"border-radius:2px;font-style:normal;font-weight:300\"><blockquote><p>A bad way to spend one&#8217;s Friday, but at least I am ending it on a good note. <\/p><cite>~ Friday guru<\/cite><\/blockquote><\/figure>\n\n\n\n<p>I was struggling with the mTLS configuration for zookeeper configuration and somehow no matter what I do I kept getting an error as below<\/p>\n\n\n\n<pre class=\"wp-block-code has-black-color has-pale-ocean-gradient-background has-text-color has-background has-small-font-size\"><code>Oct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]: &#91;2022-10-07 16:11:26,818] WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL\/TLS record: 0000002d0000000000000000000000000000753000000000000000000000001000000000000000000000000000&gt;\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at java.base\/java.lang.Thread.run(Thread.java:829)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]: Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL\/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)\nOct 07 16:11:26 centos3.brokers.test zookeeper-server-start.sh&#91;18171]:         ... 17 more\n<\/code><\/pre>\n\n\n\n<p>The error was notifying of something, but it was hard to decipher and stupid old me, I just kept overlooking at the core issue. <\/p>\n\n\n\n<p>Thanks to not so clear apache documentation (which later I realized was all this time pointing to the issue) I kept chasing my own tail until by pure luck I tried the <code>zk-tls-config-file<\/code> as below<\/p>\n\n\n\n<pre class=\"wp-block-code has-black-color has-blush-light-purple-gradient-background has-text-color has-background has-small-font-size\"><code>ssl.clientAuth=need\nzookeeper.ssl.client.enable=true\nzookeeper.connect=centos3.brokers.test:2182,centos2.brokers.test:2182,centos1.brokers.test:2182\nsecureClientPort=2182\nzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty\nzookeeper.ssl.trustStore.location=\/opt\/CA\/netops-kafka\/ssl\/kafka.broker.truststore.jks\nzookeeper.ssl.trustStore.password=changeit\nzookeeper.ssl.keystore.location=\/opt\/CA\/netops-kafka\/ssl\/kafka.broker.keystore.jks\nzookeeper.ssl.keystore.password=changeit\nzookeeper.set.acl=true\nzookeeper.client.secure=true<\/code><\/pre>\n\n\n\n<p>Once I had these properties I realized the issue is that the server is getting an un-encrypted packet and hence it is throwing the error that it is.<\/p>\n\n\n\n<pre class=\"wp-block-code has-white-color has-black-background-color has-text-color has-background has-small-font-size\"><code>bin\/zookeeper-shell.sh centos3.brokers.test:2182 -zk-tls-config-file .\/config\/zookeeper.config\nConnecting to centos3.brokers.test:2182\nWelcome to ZooKeeper!\nJLine support is disabled\n&#91;2022-10-07 16:26:55,683] WARN zookeeper.ssl.trustStore.location not specified (org.apache.zookeeper.common.X509Util)\n\nWATCHER::\n\nWatchedEvent state:SyncConnected type:None path:null<\/code><\/pre>\n\n\n\n<p>BANG! she works now.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"104\" src=\"https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-07-at-4.27.51-PM-1024x104.png\" alt=\"\" class=\"wp-image-2368\" srcset=\"https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-07-at-4.27.51-PM-1024x104.png 1024w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-07-at-4.27.51-PM-300x31.png 300w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-07-at-4.27.51-PM-1536x157.png 1536w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-07-at-4.27.51-PM-2048x209.png 2048w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-07-at-4.27.51-PM-300x31@2x.png 600w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>A bad way to spend one&#8217;s Friday, but at least I am ending it on a good note. ~ Friday guru I was struggling with the mTLS configuration for zookeeper configuration and somehow no matter what I do I kept getting an error as below The error was notifying of something, but it was hard [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":2367,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[34],"tags":[266,225,21],"class_list":["post-2366","post","type-post","status-publish","format-image","has-post-thumbnail","hentry","category-technical","tag-error","tag-kafka","tag-learn","post_format-post-format-image"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/2366","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=2366"}],"version-history":[{"count":2,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/2366\/revisions"}],"predecessor-version":[{"id":2370,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/2366\/revisions\/2370"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/2367"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=2366"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=2366"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=2366"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}