[英]CloudWatch resource access policy error while creating Amazon Elasticsearch Service via Cloud Formation
I am trying to create an elastic search domain with enabled LogPublishingOptions
.我正在尝试创建一个启用
LogPublishingOptions
的弹性搜索域。 While enabling LogPublishingOptions ES says it does not sufficient permissions to create a LogStream on Cloudwatch.虽然启用 LogPublishingOptions ES 表示它没有足够的权限在 Cloudwatch 上创建 LogStream。
I tried creating a policy with a role and attaching the policy to the LogGroup which is referred by ES but it ain't working.我尝试创建一个具有角色的策略并将该策略附加到 ES 引用的 LogGroup 但它不起作用。 Following is my elastic search cloud formation template,
以下是我的弹性搜索云形成模板,
AWSTemplateFormatVersion: 2010-09-09
Resources:
MYLOGGROUP:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: index_slow
MYESROLE:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: es.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AmazonESFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
RoleName: !Join
- '-'
- - es
- !Ref 'AWS::Region'
PolicyDocESIndexSlow :
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:PutLogEvents
- logs:CreateLogStream
Resource: 'arn:aws:logs:*'
PolicyName: !Ref MYLOGGROUP
Roles:
- !Ref MYESROLE
MYESDOMAIN:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: 'es-domain'
ElasticsearchVersion: '7.4'
ElasticsearchClusterConfig:
DedicatedMasterCount: 3
DedicatedMasterEnabled: True
DedicatedMasterType: 'r5.large.elasticsearch'
InstanceCount: '2'
InstanceType: 'r5.large.elasticsearch'
EBSOptions:
EBSEnabled: True
VolumeSize: 10
VolumeType: 'gp2'
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: '*'
Action: 'es:*'
Resource: '*'
AdvancedOptions:
rest.action.multi.allow_explicit_index: True
LogPublishingOptions:
INDEX_SLOW_LOGS:
CloudWatchLogsLogGroupArn: !GetAtt
- MYLOGGROUP
- Arn
Enabled: True
VPCOptions:
SubnetIds:
- !Ref MYSUBNET
SecurityGroupIds:
- !Ref MYSECURITYGROUP
MYVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
MYSUBNET:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref MYVPC
CidrBlock: 10.0.0.0/16
MYSECURITYGROUP:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: security group for elastic search domain
VpcId: !Ref MYVPC
GroupName: 'SG for ES'
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
Upon execution, it creates all resources except MYESDOMAIN.执行时,它会创建除 MYESDOMAIN 之外的所有资源。 It says
它说
The Resource Access Policy specified for the CloudWatch Logs log group index_slow does not grant sufficient permissions for Amazon Elasticsearch Service to create a log stream.为 CloudWatch Logs 日志组 index_slow 指定的资源访问策略未授予 Amazon Elasticsearch 服务创建日志 stream 的足够权限。 Please check the Resource Access Policy.
请检查资源访问策略。 (Service: AWSElasticsearch; Status Code: 400; Error Code: ValidationException)
(服务:AWSElasticsearch;状态代码:400;错误代码:ValidationException)
Any idea what's missing here?知道这里缺少什么吗?
I believe there is some confusion here about what policies should be updated/set to enable ES writing to a log group.我相信这里有一些关于应该更新/设置哪些策略以启用 ES 写入日志组的混淆。
I think you should apply the PolicyDocESIndexSlow
policy to CloudWatch Logs .我认为您应该将
PolicyDocESIndexSlow
策略应用于CloudWatch Logs 。
And this can't be done in CloudFormation from what I remember.根据我的记忆,这不能在 CloudFormation 中完成。 You have to use put-resource-policy , corresponding API call, or console as shown in:
您必须使用put-resource-policy 、相应的 API 调用或控制台,如下所示:
The final code would be something like this,最终的代码将是这样的,
DeployES lambda_function.py部署ES lambda_function.py
import logging
import time
import boto3
import json
from crhelper import CfnResource
logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)
try:
# Init code goes here
pass
except Exception as e:
helper.init_failure(e)
@helper.create
@helper.update
def create(event, _):
logger.info("Got Create/Update")
my_log_group_arn = event['ResourceProperties']['MYLOGGROUPArn']
client = boto3.client('logs')
policy_document = dict()
policy_document['Version'] = '2012-10-17'
policy_document['Statement'] = [{
'Sid': 'ESLogsToCloudWatchLogs',
'Effect': 'Allow',
'Principal': {
'Service': [
'es.amazonaws.com'
]
},
'Action': 'logs:*',
}]
policy_document['Statement'][0]['Resource'] = my_log_group_arn
client.put_resource_policy(policyName='ESIndexSlowPolicy', policyDocument=json.dumps(policy_document))
helper.Data['success'] = True
helper.Data['message'] = 'ES policy deployment successful'
# To return an error to Cloud Formation you raise an exception:
if not helper.Data["success"]:
raise Exception('Error message to cloud formation')
return "MYESIDDEFAULT"
@helper.delete
def delete(event, _):
logger.info("Got Delete")
# Delete never returns anything. Should not fail if the underlying resources are already deleted.
# Desired state.
try:
client = boto3.client('logs')
client.delete_resource_policy(policyName='ESIndexSlowPolicy')
except Exception as ex:
logger.critical(f'ES policy delete failed with error [{repr(ex)}]')
def lambda_handler(event, context):
helper(event, context)
And some additional components in CF template以及 CF 模板中的一些附加组件
MYLAMBDAROLE:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
- 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
- 'arn:aws:iam::aws:policy/AmazonESFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
RoleName: !Join
- '-'
- - lambda-role
- !Ref 'AWS::Region'
MYLAMBDADEPLOY:
Type: 'AWS::Lambda::Function'
Properties:
Code:
S3Bucket: es-bucket-for-lambda-ta86asdf596
S3Key: es.zip
FunctionName: deploy_es
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt
- MYLAMBDAROLE
- Arn
Runtime: python3.8
Timeout: 60
MYESSETUP:
Type: 'Custom::MYESSETUP'
Properties:
ServiceToken: !GetAtt
- MYLAMBDADEPLOY
- Arn
MYLOGGROUPArn: !GetAtt
- MYLOGGROUP
- Arn
DependsOn:
- MYLAMBDADEPLOY
- MYLOGGROUP
And just add below DependsOn
to MYESDOMAIN只需在
DependsOn
下面添加到 MYESDOMAIN
DependsOn:
- MYESSETUP
There is a CloudFormation resource called AWS::Logs::ResourcePolicy
which allows defining policies for CloudWatch Logs in CF.有一个名为
AWS::Logs::ResourcePolicy
的 CloudFormation 资源,它允许在 CF 中为 CloudWatch Logs 定义策略。 The main issue I found is that it only accepts a real string as the value.我发现的主要问题是它只接受一个真正的字符串作为值。 Trying to assemble a string using Ref, Join, etc kept being rejected.
尝试使用 Ref、Join 等组合字符串一直被拒绝。 If anyone can make that work that would be fab.
如果有人能完成这项工作,那就太棒了。
Writing it in YAML is easier as JSON requires escaping all the "
characters.将其写入 YAML 更容易,因为 JSON 需要 escaping 所有
"
字符。
OSLogGroupPolicy:
Type: AWS::Logs::ResourcePolicy
Properties:
PolicyName: AllowES
PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'
Adding a DependsOn
in the AWS::Elasticsearch::Domain
resource for the AWS::Logs::ResourcePolicy
fixed this for me.在
AWS::Logs::ResourcePolicy
的AWS::Elasticsearch::Domain
资源中添加DependsOn
为我解决了这个问题。
Example code:示例代码:
ESLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Sub '/aws/OpenSearchService/domains/${NamePrefix}-es/application-logs'
RetentionInDays: 30
ESLogGroupPolicy:
Type: AWS::Logs::ResourcePolicy
DependsOn: ESLogGroup
Properties:
PolicyName: !Sub "es-logs-access-policy"
PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'
ESDomain:
Type: AWS::Elasticsearch::Domain
DependsOn: [ESLogGroupPolicy]
Properties:
DomainName: !Sub "${NamePrefix}-es"
...
LogPublishingOptions:
ES_APPLICATION_LOGS:
CloudWatchLogsLogGroupArn: !GetAtt ESLogGroup.Arn
Enabled: true
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.