Kubernetes secrets and spring boot configuration

Question

Our service is running in kubernetes cluster. I'm trying to make our service to be secured by SSL.

For that purpose I added to application.properties:

security.require-ssl=true 
server.ssl.key-store-type=JKS
server.ssl.key-store=serviceCertificates.jks
server.ssl.key-store-password=${KEYSTORE_PASSWORD}
server.ssl.key-alias=certificate

The keystore password I want to take from kubernetes secret, that is defined in the cluster.
When the service starts running I get an error Password verification failed :

"org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-8080]]\\n\\tat org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)\\n\\tat org.apache.catalina.core.StandardService.addConnector(StandardService.java:225)\\n\\tat org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:256)\\n\\tat org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:198)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:300)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162)\\n\\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplica tionContext.java:140)\\n\\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)\\n\\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:327)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)\\n\\tat com.ibm.securityservices.cryptoutils.Application.main(Application.java:9)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\\n\\tat org.springframew ork.boot.loader.Launcher.launch(Launcher.java:50)\\n\\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)\\nCaused by: org.apache.catalina.LifecycleException: Protocol handler start failed\\n\\tat org.apache.catalina.connector.Connector.startInternal(Connector.java:1020)\\n\\tat org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)\\n\\t... 21 common frames omitted\\nCaused by: java.lang.IllegalArgumentException: Keystore was tampered with, or password was incorrect\\n\\tat org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:116)\\n\\tat org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:87)\\n\\tat org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:225)\\n\\tat org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1150)\\n\\tat org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:591)\\n\\tat org.apache.catalina.connector.Connector.startInternal(Connector.ja va:1018)\\n\\t... 22 common frames omitted\\nCaused by: java.io.IOException: Keystore was tampered with, or password was incorrect\\n\\tat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)\\n\\tat sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)\\n\\tat sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)\\n\\tat sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)\\n\\tat java.security.KeyStore.load(KeyStore.java:1445)\\n\\tat org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:139)\\n\\tat org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:204)\\n\\tat org.apache.tomcat.util.net.jsse.JSSEUtil.getKeyManagers(JSSEUtil.java:184)\\n\\tat org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)\\n\\t... 27 common frames omitted\\nCaused by: java.security.UnrecoverableKeyException: Password verification failed\\n\\t at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)\\n\\t... 35 common frames omitted\\n"}

My investigation:
1. If I print in the code

    System.out.println("KEYSTORE_PASSWORD: "+ System.getenv("KEYSTORE_PASSWORD"));   

I see its correct value.
2. If I set hard coded constant password value in application properties, it works, service is up and running.

So I guess that the problem is to set secret value to application property.
Your help and advice will be appreciated

Answer 1

I think there is typo or hidden character in your secret descriptor. You can exec into the pod , verify the system property and also try decrypting the password using command line tools.