kubernetes hpa 中外部指标的值很奇怪

Question

I'm studying kubernetes and testing some example.

I got a problem with applying external metrics to hpa.

i made a external metrics with prometheus adapter.

so I can get external metrics using

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/

command.

result is below.

{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[{"name":"redis_keys","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}]}

and i can get metrics value using

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/redis_keys

command.

result is below.

{"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/external.metrics.k8s.io/v1beta1/namespaces/default/redis_keys"},"items":[{"metricName":"redis_keys","metricLabels":{},"timestamp":"2020-10-28T08:39:09Z","value":"23"}]}

and i applied the metric to hpa.

below is hpa configuration.

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: taskqueue-autoscaler
spec:
  scaleTargetRef:
          #apiVersion: extensions/v1beta1
    apiVersion: apps/v1
    kind: Deployment
    name: taskqueue-consumer
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metricName: redis_keys
      targetAverageValue: 20

after mading hpa,

i tested this command.

kubectl get hpa

and result is weird.

NAME                   REFERENCE                       TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
taskqueue-autoscaler   Deployment/taskqueue-consumer   11500m/20 (avg)   1         10        2          63m

i think it's value(11500m) is wrong because result of query value is 23.

Where should i see in this case?

Answer 1

Actually it's right, but its also complicated because it gets into a couple of different things with the HPA resource that are not immediately obvious. So I will tackle explaining this one thing at a time.

First, the units of measurement. The Metrics API will try to return whole units when possible, but does return milli-units as well, which can result in the above. Really, what you are seeing is 11.5 converted to 11500m, but the two are the same. Check out this link on the "Quantities" of HPA metrics that covers it a bit more.

Next, you are seeing two replicas at the moment with a value from the Metrics API of 23. Since you have set the metric to be the AverageValue of the External Metric, it is dividing the value of the metric by the number of replicas in the cluster to result in 11.5 or 11500m when you view the HPA resource. This explains why you are seeing only 2 replicas, while the value of the metric is "above" your threshold. Check out this link on Autoscaling with Multiple & Custom metrics, specifically the section about "Object" metrics . And lower down the page they slip in this line regarding External Metrics, confirming why you are seeing the above.

External metrics support both the Value and AverageValue target types, which function exactly the same as when you use the Object type.

Hope this helps and a few tweaks should make it line up better with your expectations. Good luck out there!