Flink Kubernetes deployment - the HPA controller was unable to get the target's current scale:

Question

I am deploying the flink stateful app using the below-mentioned YAML file.

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: operational-reporting-15gb
spec:
  image:.azurecr.io/stateful-app-v2
  flinkVersion: v1_15
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    state.savepoints.dir: abfs://flinktest@.dfs.core.windows.net/savepoints.v2
    state.checkpoints.dir: abfs://flinktest@.dfs.core.windows.net/checkpoints.v2
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: abfs://flinktest@.dfs.core.windows.net/ha.v2
  serviceAccount: flink
  jobManager:
    resource:
      memory: "15360m"
      cpu: 2
  taskManager:
    resource:
      memory: "15360m"
      cpu: 3
  podTemplate:
    spec:
      containers:
        - name: flink-main-container
          volumeMounts:
            - mountPath: /flink-data
              name: flink-volume
      volumes:
        - name: flink-volume
          emptyDir: {}
  job:
    jarURI: local:///opt/operationalReporting.jar
    parallelism: 1
    upgradeMode: savepoint
    state: running

Flink jobs are running perfectly. For auto-scaling I created HPA using the following code.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: basic-hpa
  namespace: default
spec:
  minReplicas: 1
  maxReplicas: 15
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageValue: 100m
  scaleTargetRef:
    apiVersion: flink.apache.org/v1beta1
    kind: FlinkDeployment
    name: operational-reporting-15gb

While describing the auto scaling I am getting below mentioned error.

Type Status Reason Message

---

AbleToScale False FailedGetScale the HPA controller was unable to get the target's current scale: flinkdeployments.flink.apache.org "operational-reporting-15gb" not found Events: Type Reason Age From Message

---

Warning FailedGetScale 4m4s (x121 over 34m) horizontal-pod-autoscaler flinkdeployments.flink.apache.org "operational-reporting-15gb" not found

For HPA target is showing UNKNOW. Kindly help

Answer 1

I assume you are following the HPA example of the Kubernetes Operator. Thanks for giving it a try, it is an experimental feature as outlined in the docs, we only have limited experience with it at the moment.

That said checking the obvious is your FlinkDeployment named operational-reporting-15gb running in the default namespace? Otherwise please adjust the namespace of your HPA accordingly.