在私有Kubernetes集群上设置大三角帆-Clouddriver证书错误

Question

I tried deploying Spinnaker using halyard and running into issue with clouddriver.

Have added a corporate Docker registry which needs corporate CA.

Clouddriver was failing with the below error.

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_212]
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_212]
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_212]
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[na:1.8.0_212]
    ... 100 common frames omitted

2019-08-07 04:53:14.237 ERROR 1 --- [0.0-7002-exec-3] c.n.s.k.w.e.GenericExceptionHandlers     : Internal Server Error

com.netflix.spinnaker.clouddriver.core.AlwaysUpHealthIndicator$HealthIndicatorWrappedException: retrofit.RetrofitError: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at com.netflix.spinnaker.clouddriver.core.AlwaysUpHealthIndicator.health(AlwaysUpHealthIndicator.java:49) ~[clouddriver-core.jar:na]
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
    at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
    at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:53) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_212]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_212]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
    at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) ~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
    at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
    at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:278) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]

So, created secret with cacerts which is imported with corporate CA based on the below link.

https://www.spinnaker.io/reference/halyard/custom/#using-custom-volumes

In ~/.hal/default/service-settings/clouddriver.yml,

kubernetes:
  volumes:
  - id: spin-truststore
    type: secret
    mountPath: /app/certs/

But, still getting the same cert error.

Also, tried the below approach.

Added the below entry ~/.hal/default/profile/clouddriver-local.yml along with the above volumes.

okHttpClient:
  enabled: true
  keyStore: /app/certs/cacert
  keyStorePassword: changeit
  trustStore: /app/certs/cacert
  trustStorePassword: changeit
  propagateSpinnakerHeaders: true
  connectTimeoutMs: 60000
  readTimeoutMs: 60000

Now, getting stream too big error.

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [retrofit.client.OkClient]: Factory method 'okClient' threw exception; nested exception is java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    ... 107 common frames omitted
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
    at sun.security.util.DerInputStream.getLength(DerInputStream.java:599) ~[na:1.8.0_212]
    at sun.security.util.DerValue.init(DerValue.java:391) ~[na:1.8.0_212]
    at sun.security.util.DerValue.<init>(DerValue.java:332) ~[na:1.8.0_212]
    at sun.security.util.DerValue.<init>(DerValue.java:345) ~[na:1.8.0_212]
    at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1938) ~[na:

Update

The cert error is gone now after updating the mount path as below. Here, I did not add the file clouddriver-local.yml.

mountPath: /etc/ssl/certs/java

But, getting different errors now,

2019-08-07 06:09:55.364 ERROR 1 --- [ecutionAction-2] .d.r.p.a.DockerRegistryImageCachingAgent : Could not load tags for gcp-spinnaker/spinnaker-marketplace/front50 in https://docker.xyz.com

retrofit.RetrofitError: 429 Too Many Requests
    at retrofit.RetrofitError.httpError(RetrofitError.java:40) ~[retrofit-1.9.0.jar:na]
    at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:388) ~[retrofit-1.9.0.jar:na]
    at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) ~[retrofit-1.9.0.jar:na]
    at com.sun.proxy.$Proxy134.getTags(Unknown Source) ~[na:na]

Also the below error.

Error from server (Forbidden): podsecuritypolicies.extensions is forbidden: User "XXXXXXXX" cannot list resource "podsecuritypolicies" in API group "extensions" at the cluster scope
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "XXXXXX" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope

    at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor.list(KubectlJobExecutor.java:421) ~[clouddriver-kubernetes.jar:na]
    at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.lambda$list$18(KubernetesV2Credentials.java:464) ~[clouddriver-kubernetes.jar:na]
    at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.runAndRecordMetrics(KubernetesV2Credentials.java:598) ~[clouddriver-kubernetes.jar:na]
    at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.list(KubernetesV2Credentials.java:460) ~[clouddriver-kubernetes.jar:na]
    at com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesV2CachingAgent.lambda$loadPrimaryResourceList$0(KubernetesV2CachingAgent.java:88) ~[clouddriver-kubernetes.jar:na]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_212]

Solution 1:-

Create a custom image with the corporate cert imported.

https://medium.com/@edwin.a.avalos/updating-spinnaker-halyard-releases-with-custom-containers-373494a532b9

Answer 1

I'm having the same problem. Apparently, the 'too big' error is due to a wrong crt-file. The keystore/truststore couldn't be read by keytool neither as JKS nor PKCS#12 keystore.

See here