Can't read configMap with name: [my-configmap] in namespace:[my-namespace]. Ignoring

Question

Deploying the spring application only works in the default namespace. Deploying the same application to other namespaces fails with the following error:

I have noticed this has something to do with the service (named: Kubernetes) in the default namespace which is having the clusterIP of 10.242.0.1 . Deploying the application to other namespaces still routes to this Kubernetes service in the default namespace. Adding service link to this service doesn't help. Creating a replica of this service with endpoints in the other namespaces doesn't solve the issue either.

2021-05-13 11:46:24.514  WARN [some-service,,] 1 --- [           main] s.c.k.f.c.Fabric8ConfigMapPropertySource : Can't read configMap with name: [my-configmap] in namespace:[my-namespace]. Ignoring.

io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [ConfigMap]  with name: [my-configmap]  in namespace: [my-namespace]  failed.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:225) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:186) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:84) ~[kubernetes-client-4.13.2.jar:na]
        at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.getData(Fabric8ConfigMapPropertySource.java:62) [spring-cloud-kubernetes-fabric8-config-2.0.2.jar:2.0.2]
        at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySource.<init>(Fabric8ConfigMapPropertySource.java:50) [spring-cloud-kubernetes-fabric8-config-2.0.2.jar:2.0.2]
        at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapPropertySourceLocator.getMapPropertySource(Fabric8ConfigMapPropertySourceLocator.java:51) [spring-cloud-kubernetes-fabric8-config-2.0.2.jar:2.0.2]
        at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.getMapPropertySourceForSingleConfigMap(ConfigMapPropertySourceLocator.java:81) [spring-cloud-kubernetes-commons-2.0.2.jar:2.0.2]
        at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.lambda$locate$0(ConfigMapPropertySourceLocator.java:67) [spring-cloud-kubernetes-commons-2.0.2.jar:2.0.2]
        at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_282]
        at org.springframework.cloud.kubernetes.commons.config.ConfigMapPropertySourceLocator.locate(ConfigMapPropertySourceLocator.java:67) [spring-cloud-kubernetes-commons-2.0.2.jar:2.0.2]
        at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:51) ~[spring-cloud-context-3.0.2.jar:3.0.2]
        at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:47) ~[spring-cloud-context-3.0.2.jar:3.0.2]
        at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:95) ~[spring-cloud-context-3.0.2.jar:3.0.2]
        at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:650) ~[spring-boot-2.4.5.jar:2.4.5]
        at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:403) ~[spring-boot-2.4.5.jar:2.4.5]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.4.5.jar:2.4.5]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1340) ~[spring-boot-2.4.5.jar:2.4.5]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) ~[spring-boot-2.4.5.jar:2.4.5]
        at com.ubs.wma.service.accountgroup.AccountGroupServiceApplication.main(AccountGroupServiceApplication.java:19) ~[classes/:na]
Caused by: java.net.ConnectException: Failed to connect to /10.242.0.1:443
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[okhttp-3.14.9.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:133) ~[kubernetes-client-4.13.2.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at io.fabric8.kubernetes.client.utils.OIDCTokenRefreshInterceptor.intercept(OIDCTokenRefreshInterceptor.java:41) ~[kubernetes-client-4.13.2.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[kubernetes-client-4.13.2.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:151) ~[kubernetes-client-4.13.2.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:na]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[okhttp-3.14.9.jar:na]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[okhttp-3.14.9.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:472) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:402) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:384) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:935) ~[kubernetes-client-4.13.2.jar:na]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:220) ~[kubernetes-client-4.13.2.jar:na]
        ... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_282]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_282]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_282]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_282]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_282]
        at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_282]
        at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130) ~[okhttp-3.14.9.jar:na]
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263) ~[okhttp-3.14.9.jar:na]
        ... 54 common frames omitted                                            

Added Role and RoleBinding resources to my deployment like this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-reader
rules:
  - apiGroups: ["", "extensions", "apps"]
    resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
    verbs: ["get", "list", "watch"]

---

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-reader-binding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: default
    apiGroup: ""
roleRef:
  kind: Role
  name: namespace-reader
  apiGroup: ""

My bootstrap.yaml config is looking like this:

spring:
  cloud:
    kubernetes:
      config:
        sources:
          - name: some-service
        namespace: ${MY_POD_NAMESPACE}

My service deployment is looking like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-service
spec:
  selector:
    matchLabels:
      app: some-service
  template:
    metadata:
      labels:
        app: some-service
    spec:
      containers:
        - name: some-service
          image: registry/some-service:latest
          resources:
            requests:
              memory: 320Mi
            limits:
              memory: 512Mi
          ports:
            - containerPort: 8080
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "some-profile"
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 40
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 40
          lifecycle:
            preStop:
              exec:
                command: [ "sh", "-c", "sleep 10" ]

Finally, my configmap looks as follows:

apiVersion: v1
data:
  application-properties.yaml: 'property: "value"'
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
  name: my-configmap

Answer 1

I have found a workaround that works for me. Created an admin clusterrolebinding resource for my namespace my-namespace like so:

kubectl create clusterrolebinding admin-my-namespace --clusterrole=cluster-admin --serviceaccount=my-namespace:default

Deleted the Role and the RoleBinding resources as they are not needed any longer.

This works but would appreciate why Role and Rolebinding to ServiceAccount wasn't enough at the first place.

Answer 2

Check for network policy configured which blocks accessing it