從 Kinesis 數據 stream 到 Kinesis Firehose 到 S3 存儲桶的 CloudFront 實時日志

Question

我能夠通過控制台創建 CloudFront 實時日志，現在我想通過 Terraform 進行設置。

我目前有一個指向 S3 存儲桶的 CloudFront 分配。

resource "aws_cloudfront_distribution" "www_distribution" {
  default_cache_behavior {
    realtime_log_config_arn = aws_cloudfront_realtime_log_config.analytics.arn
    ...
  }
  ...
}

我創建了實時日志配置。

resource "aws_cloudfront_realtime_log_config" "analytics" {
  name          = "analytics"
  sampling_rate = 100
  fields        = [
    ...
  ]

  endpoint {
    stream_type = "Kinesis"

    kinesis_stream_config {
      role_arn   = aws_iam_role.analytics.arn
      stream_arn = aws_kinesis_stream.analytics.arn
    }
  }

  depends_on = [aws_iam_role_policy.analytics]
}

然后由 Kinesis 數據 stream 管理

resource "aws_kinesis_stream" "analytics" {
  name             = "blog-cloudfront-analytics"
  shard_count      = 1
  retention_period = 48

  shard_level_metrics = [
    "IncomingBytes",
    "OutgoingBytes",
  ]
}

我希望它被 Kinesis Firehose stream 消耗。

resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
  name        = "example-cloudfront-analytics"
  destination = "extended_s3"

  kinesis_source_configuration {
    kinesis_stream_arn = aws_kinesis_stream.analytics.arn
    role_arn = aws_iam_role.kinesis_firehose.arn
  }

  extended_s3_configuration {
     cloudwatch_logging_options {
      log_group_name = "/aws/lambda/example_cloudfront_analytics"
      log_stream_name = "example_stream"
      enabled = true
    }
    role_arn   = aws_iam_role.firehose_role.arn
    bucket_arn = aws_s3_bucket.bucket.arn
  }
}
resource "aws_s3_bucket" "bucket" {
  bucket = "example-cloudfront-analytics"
  acl    = "private"
}

我已應用此配置，但 Kinesis 數據 stream 控制台中的“監視器”選項卡顯示沒有任何內容發送到 stream。 我該如何設置？

更新

以下是用於前面提到的不同服務的 IAM 角色。

---

這是用於Kinesis Firehose的那個

data "aws_iam_policy_document" "kinesis_firehose" {
  statement {
    effect="Allow"
    actions = [
      "kinesis:*",
      "firehose:*"
    ]
    resources = [
      aws_kinesis_stream.analytics.arn,
      aws_kinesis_firehose_delivery_stream.extended_s3_stream.arn
    ]
    sid = "kinesisId"
  }
}
resource "aws_iam_role" "kinesis_firehose" {
   name = "cloudfront_kinesis_role"
   assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}
resource "aws_iam_role_policy" "kinesis_firehose_stream" {
  policy = data.aws_iam_policy_document.kinesis_firehose.json
  role   = aws_iam_role.kinesis_firehose.id
}

這是實時日志中的一種用途。

resource "aws_iam_role" "analytics" {
  name = "cloudfront-realtime-log"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "kinesis.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}
resource "aws_iam_role_policy" "analytics" {
  name = "cloudfront-realtime-log"
  role = aws_iam_role.analytics.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
        "Effect": "Allow",
        "Action": [
          "kinesis:DescribeStreamSummary",
          "kinesis:DescribeStream",
          "kinesis:PutRecord",
          "kinesis:PutRecords"
        ],
        "Resource": "${aws_kinesis_stream.analytics.arn}"
    }
  ]
}
EOF
}

這是用於S3存儲桶的那個。

resource "aws_iam_role" "firehose_role" {
  name = "firehose_cloudfront"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}
data "aws_iam_policy_document" "kinesis_firehose_s3" {
  statement {
    effect="Allow"
    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
    ]
    resources = [
      aws_s3_bucket.bucket.arn,
      "${aws_s3_bucket.bucket.arn}/*",
    ]
    sid = "kinesisId"
  }
}
resource "aws_iam_role_policy" "kinesis_firehose_stream_s3" {
  policy = data.aws_iam_policy_document.kinesis_firehose_s3.json
  role   = aws_iam_role.firehose_role.id
}

Answer 1

您的aws_cloudfront_realtime_log_config.analytics使用角色aws_iam_role.analytics.arn 。 但是，它的原理是kinesis.amazonaws.com 。 它應該是cloudfront.amazonaws.com ，如TF 文檔中所示：

resource "aws_iam_role" "analytics" {
  name = "cloudfront-realtime-log"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

可能還有其他尚未確定的問題。