Do I need to create two kinesis delivery streams to send messages under two different paths in the same s3 bucket?

Question

I have a lambda function which receives logs from cloudwatch using a subscription filter and has to send them to firehose to be delivered to an s3 bucket.

The same lambda function receives two kinds of logs. These two log formats need to be written into two different paths under the same s3 destination.

For eg.

1. Log type A should be stored under - /logTypeA directory.
2. Log type B should be stored under - /logTypeB directory.

I am using cdk to create the necessary firehose and s3 setup.

Do I need to create two different delivery streams - one for log type A and one for log type B or is there a way to do this using a single delivery stream ? Didn't find anything useful in the tech docs.

Answer 1

Do I need to create two different delivery streams - one for log type A and one for log type B or is there a way to do this using a single delivery stream?

Using a single Kinesis Delivery Stream might be possible with Dynamic Partitioning . It depends on the data and schema of the logs. If the schema of the logs contains a field that designates the schema to be of Type A vs Type B, then partitioning can happen in-line.

Example schema:

{
    "log_type": "log_type_A"
},
{
    "log_type": "log_type_B"
}

Example Prefix configuration:

"Prefix": "log_type=!{partitionKeyFromQuery:log_type}"

Example output to S3:

s3://<bucket>/log_type=log_type_A/...
s3://<bucket>/log_type=log_type_B/...

For more advanced partitioning, logic can be implemented with AWS Lambda ( partitionKeyFromLambda ). See the example in the AWS documentation for more details.

Example Prefix configuration:

"Prefix": "log_type=!{partitionKeyFromLambda:log_type}"

References

1. Custom Prefixes for Amazon S3 Objects (AWS)