从 Kinesis 数据 stream 到 Kinesis Firehose 到 S3 存储桶的 CloudFront 实时日志

Question

I was able to create the CloudFront realtime logs through the console, and now I'd like to setup through Terraform.

I currently have a CloudFront distribution that points back to an S3 bucket.

resource "aws_cloudfront_distribution" "www_distribution" {
  default_cache_behavior {
    realtime_log_config_arn = aws_cloudfront_realtime_log_config.analytics.arn
    ...
  }
  ...
}

I created the realtime log configuration.

resource "aws_cloudfront_realtime_log_config" "analytics" {
  name          = "analytics"
  sampling_rate = 100
  fields        = [
    ...
  ]

  endpoint {
    stream_type = "Kinesis"

    kinesis_stream_config {
      role_arn   = aws_iam_role.analytics.arn
      stream_arn = aws_kinesis_stream.analytics.arn
    }
  }

  depends_on = [aws_iam_role_policy.analytics]
}

That is then managed by a Kinesis data stream

resource "aws_kinesis_stream" "analytics" {
  name             = "blog-cloudfront-analytics"
  shard_count      = 1
  retention_period = 48

  shard_level_metrics = [
    "IncomingBytes",
    "OutgoingBytes",
  ]
}

I'd like for this to be consumed by a Kinesis Firehose stream.

resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
  name        = "example-cloudfront-analytics"
  destination = "extended_s3"

  kinesis_source_configuration {
    kinesis_stream_arn = aws_kinesis_stream.analytics.arn
    role_arn = aws_iam_role.kinesis_firehose.arn
  }

  extended_s3_configuration {
     cloudwatch_logging_options {
      log_group_name = "/aws/lambda/example_cloudfront_analytics"
      log_stream_name = "example_stream"
      enabled = true
    }
    role_arn   = aws_iam_role.firehose_role.arn
    bucket_arn = aws_s3_bucket.bucket.arn
  }
}
resource "aws_s3_bucket" "bucket" {
  bucket = "example-cloudfront-analytics"
  acl    = "private"
}

I've applied this configuration, but the 'monitor' tab in the Kinesis data stream console shows that nothing is being sent to the stream. How do I set this up?

Update

Here are the IAM roles used for the different services mentioned before.

---

This is the one used for the Kinesis Firehose

data "aws_iam_policy_document" "kinesis_firehose" {
  statement {
    effect="Allow"
    actions = [
      "kinesis:*",
      "firehose:*"
    ]
    resources = [
      aws_kinesis_stream.analytics.arn,
      aws_kinesis_firehose_delivery_stream.extended_s3_stream.arn
    ]
    sid = "kinesisId"
  }
}
resource "aws_iam_role" "kinesis_firehose" {
   name = "cloudfront_kinesis_role"
   assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}
resource "aws_iam_role_policy" "kinesis_firehose_stream" {
  policy = data.aws_iam_policy_document.kinesis_firehose.json
  role   = aws_iam_role.kinesis_firehose.id
}

This is the one use inside the realtime log .

resource "aws_iam_role" "analytics" {
  name = "cloudfront-realtime-log"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "kinesis.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}
resource "aws_iam_role_policy" "analytics" {
  name = "cloudfront-realtime-log"
  role = aws_iam_role.analytics.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
        "Effect": "Allow",
        "Action": [
          "kinesis:DescribeStreamSummary",
          "kinesis:DescribeStream",
          "kinesis:PutRecord",
          "kinesis:PutRecords"
        ],
        "Resource": "${aws_kinesis_stream.analytics.arn}"
    }
  ]
}
EOF
}

This is the one used for the S3 bucket.

resource "aws_iam_role" "firehose_role" {
  name = "firehose_cloudfront"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}
data "aws_iam_policy_document" "kinesis_firehose_s3" {
  statement {
    effect="Allow"
    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
    ]
    resources = [
      aws_s3_bucket.bucket.arn,
      "${aws_s3_bucket.bucket.arn}/*",
    ]
    sid = "kinesisId"
  }
}
resource "aws_iam_role_policy" "kinesis_firehose_stream_s3" {
  policy = data.aws_iam_policy_document.kinesis_firehose_s3.json
  role   = aws_iam_role.firehose_role.id
}

Answer 1

Your aws_cloudfront_realtime_log_config.analytics uses role aws_iam_role.analytics.arn . However, its principle is kinesis.amazonaws.com . It should be cloudfront.amazonaws.com , as shown in the TF docs :

resource "aws_iam_role" "analytics" {
  name = "cloudfront-realtime-log"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

There could be other issues, which haven't been yet identified.