[英]Kubernetes AWS Cloudwatch adapter not fetching custom metric value for EKS HPA autoscaling
I'm trying to enable AWS EKS autoscaling based on a custom Cloudwatch metric via the Kubernetes Cloudwatch adapter.我正在尝试通过 Kubernetes Cloudwatch 适配器基于自定义 Cloudwatch 指标启用 AWS EKS 自动缩放。 I have pushed custom metrics to AWS Cloudwatch, and validated they appear in Cloudwatch console as well as are retrievable using the boto3 client get_metric_data.
我已将自定义指标推送到 AWS Cloudwatch,并验证它们出现在 Cloudwatch 控制台中,并且可以使用 boto3 客户端 get_metric_data 进行检索。 This is the code I use to publish my custom metric to Cloudwatch:
这是我用来将自定义指标发布到 Cloudwatch 的代码:
import boto3
from datetime import datetime
client = boto3.client('cloudwatch')
cloudwatch_response = client.put_metric_data(
Namespace='TestMetricNS',
MetricData=[
{
'MetricName': 'TotalUnprocessed',
'Timestamp': datetime.now(),
'Value': 40,
'Unit': 'Megabytes',
}
]
)
I have the following yaml files for establishing the external metric and the hpa autoscaler in kubernetes:我有以下 yaml 文件用于在 kubernetes 中建立外部指标和 hpa 自动缩放器:
extMetricCustom.yaml: extMetricCustom.yaml:
apiVersion: metrics.aws/v1alpha1
kind: ExternalMetric
metadata:
name: test-custom-metric
spec:
name: test-custom-metric
resource:
resource: "deployment"
queries:
- id: sqs_test
metricStat:
metric:
namespace: "TestMetricNS"
metricName: "TotalUnprocessed"
period: 60
stat: Average
unit: Megabytes
returnData: true
hpaCustomMetric.yaml hpaCustomMetric.yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: test-scaler
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: sqs-consumer
minReplicas: 1
maxReplicas: 4
metrics:
- type: External
external:
metricName: test-custom-metric
targetAverageValue: 2
When I assess whether the Kubernetes Cloudwatch adapter is properly grabbing my custom metric (kubectl get hpa), it always displays that the metric is 0:当我评估 Kubernetes Cloudwatch 适配器是否正确获取我的自定义指标 (kubectl get hpa) 时,它始终显示指标为 0:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
test-scaler Deployment/sqs-consumer 0/2 (avg) 1 4 1 161m
How can I properly autoscale based off my Cloudwatch custom metric?如何根据我的 Cloudwatch 自定义指标正确自动缩放?
Worked with OP on this out-of-band and still had the tab open for this question later in the day, so posting the outcome here for posterity for anyone that stumbles upon it.在这个带外问题上与 OP 合作,并且当天晚些时候仍然为这个问题打开了选项卡,因此将结果张贴在这里,供任何偶然发现它的人使用。
The root cause of the issue was a timezone conflict.问题的根本原因是时区冲突。 The metrics monitor was based on "current" metrics, but the the following line from the metric generator script was producing time stamps without a timezone specified and also was in a local timezone.
指标监视器基于“当前”指标,但指标生成器脚本中的以下行正在生成没有指定时区的时间戳,并且也在本地时区。
'Timestamp': datetime.now(),
Since there was "no data" for the current timezone (only data X hours in the past due to a -X UTC offset), the system did not initiate scaling because there was a value of "0"/nil/null effectively.由于当前时区“没有数据”(由于 -X UTC 偏移,只有过去 X 小时的数据),系统没有启动缩放,因为有效地存在值“0”/nil/null。 Instead, a UTC time string can be specified to ensure the generated metrics are timely:
相反,可以指定 UTC 时间字符串以确保生成的指标及时:
'Timestamp': datetime.utcnow(),
A secondary consideration was that the Kubernetes Nodes need access to poll the metrics from CloudWatch.次要考虑是 Kubernetes 节点需要访问以从 CloudWatch 轮询指标。 This is done by attaching this policy to the nodes's IAM role:
这是通过将此策略附加到节点的 IAM 角色来完成的:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData"
],
"Resource": "*"
}
]
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.