[英]How to create a kinesis firehose delivery stream with dynamic partitions enabled using python cdk?
I am trying to create a firehose delivery stream with dynamic partitions enabled.我正在尝试创建启用了动态分区的消防软管交付 stream。 Below is what I have got so far.以下是我到目前为止所得到的。
analytics_delivery_stream = kinesisfirehose.CfnDeliveryStream(
self, "AnalyticsDeliveryStream",
delivery_stream_name='analytics',
extended_s3_destination_configuration=kinesisfirehose.CfnDeliveryStream.ExtendedS3DestinationConfigurationProperty(
bucket_arn=f'arn:aws:s3:::{analytic_bucket_name}',
buffering_hints=kinesisfirehose.CfnDeliveryStream.BufferingHintsProperty(
interval_in_seconds=60
),
dynamic_partitioning_configuration = kinesisfirehose.CfnDeliveryStream.DynamicPartitioningConfigurationProperty(
enabled=True,
retry_options=kinesisfirehose.CfnDeliveryStream.RetryOptionsProperty(
duration_in_seconds=123
)),
compression_format="UNCOMPRESSED",
role_arn=firehose_role.role_arn,
prefix="!{partitionKeyFromQuery:log_type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/",
error_output_prefix="errors/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}/",
)
)
When I run this, I get below error.当我运行它时,我得到以下错误。
Processing Configuration is not enabled when DataPartitioning is enabled.
I found below references to Processing Configuration in the docs.我在文档中找到了以下对处理配置的引用。
processing_configuration=kinesisfirehose.CfnDeliveryStream.ProcessingConfigurationProperty(
enabled=False,
processors=[kinesisfirehose.CfnDeliveryStream.ProcessorProperty(
type="type",
# the properties below are optional
parameters=[kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty(
parameter_name="parameterName",
parameter_value="parameterValue"
)]
)]
),
I am not sure what values to put for parameters or type inside processing_configuration.我不确定要为参数设置什么值或在 processing_configuration 中输入什么值。
I have logs being put into firehose with below structure.我将原木放入具有以下结构的消防软管中。
type A - {'log_type':'type_A_log',....other props....} type A - {'log_type':'type_A_log',....其他道具....}
type B - {'log_type':'type_B_log',....other props....} B 型 - {'log_type':'type_B_log',....其他道具....}
Using dynamic partitioning, I want to achieve the scenario where all logs of type A go into type_A_log directory inside s3 and type B log into type_B_log directory.使用动态分区,我想实现A类go的所有日志进入s3内的type_A_log目录,B类日志进入type_B_log目录的场景。
Can someone please help here?有人可以在这里帮忙吗? I am going down a rabbithole.我要去一个兔子洞。
I am not sure what values to put for parameters or type inside processing_configuration.我不确定要为参数设置什么值或在 processing_configuration 中输入什么值。
Here's the documentation for CfnDeliveryStream.ProcessorProperty
: https://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_kinesisfirehose/CfnDeliveryStream.html#processorproperty这是CfnDeliveryStream.ProcessorProperty
的文档: https ://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_kinesisfirehose/CfnDeliveryStream.html#processorproperty
There's only 1 valid value for type
: Lambda
. type
只有 1 个有效值: Lambda
。
Here's the documentation for CfnDeliveryStream.ProcessorParameterProperty
: https://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_kinesisfirehose/CfnDeliveryStream.html#aws_cdk.aws_kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty这是CfnDeliveryStream.ProcessorParameterProperty
的文档: https ://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_kinesisfirehose/CfnDeliveryStream.html#aws_cdk.aws_kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty
The parameters
list describes the configuration for the Lambda function that is doing the data partitioning. parameters
列表描述了执行数据分区的 Lambda 函数的配置。
Here's the documentation for the valid values for parameter_name
: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-processorparameter.html以下是parameter_name
有效值的文档: https ://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-processorparameter.html
At the very minimum, you'll need to provide the LambdaArn
parameter.至少,您需要提供LambdaArn
参数。
Example:例子:
kinesisfirehose.CfnDeliveryStream.ProcessorProperty(
type="Lambda",
parameters=[
kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty(
parameter_name="LambdaArn",
parameter_value="<lambda_arn_value>"
),
...
]
)
There's only 1 valid value for type: Lambda.类型只有 1 个有效值:Lambda。
However on AWS documentation for AWS::KinesisFirehose::DeliveryStream Processor但是在AWS::KinesisFirehose::DeliveryStream 处理器的 AWS 文档中
Type类型
The type of processor. Valid values: Lambda.
Required: Yes
Type: String
**Allowed values: AppendDelimiterToRecord | Lambda | MetadataExtraction | RecordDeAggregation**
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.