Amazon Kinesis: Caught exception while sync'ing Kinesis shards and leases

Question

I am trying to make Snowplow work on AWS. When I am trying to run stream-enrich service on instance, I am getting this exception:

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases
[cw-metrics-publisher] WARN com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable - Could not publish 4 datums to CloudWatch

I don't think error is due to Cloud Watch:

Caught exception while sync'ing Kinesis shards and leases

Answer 1

As mentioned in the comments above, this error will crop when you're lacking permissions to AWS resources required by Kinesis Client Library (KCL) . This can be the DynamoDB, CloudWatch, or Kinesis. For the Stream Enrich component of Snowplow, you'll need the following permissions:

• Read permission to input kinesis stream (collector good)
• Write permission to output kinesis streams (enrich good & enrich bad)
• List permission to kinesis streams
• Read/write/create permission to DynamoDB state table (table name is the “appName” value in your stream enrich application.conf )
• PutMetricData to Cloudwatch

A templated version of an IAM policy that meets these needs is as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
      ],
      "Resource": [
        "${collector_stream_out_good}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
          "kinesis:ListStreams"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [
        "${enricher_stream_out_good}",
        "${enricher_stream_out_bad}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DescribeTable",
        "dynamodb:Scan",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": [
        "${enricher_state_table}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    }
  ]
}

I've written up a blog post that covers required IAM permissions for Stream Enrich and other Snowplow components since documentation on the exact required permissions was sparse/non-existent in the Snowplow documentation.

Hope that helps!

Answer 2

So I had this problem when setting up Snowplow. I'm using terraform to automate the infrastructure and got this error after a destroy and re-apply. Here's what I learned.

You give the enricher DynamoDB privilages so it can create a table. If this table is already created before the enricher creates it (but not destroyed by terraform in my case) it is not able to create a table with the same name. It also seemingly won't link to existing tables.

My solution was to delete the existing DynamoDB table via the AWS console, terminate my enricher, and start up a new one. The error no longer appeared and my enricher worked as intended.

Answer 3

I faced this issue today. For me, the issue was that, I changed the kinesis stream names without changing the appName in the enrich configuration.

Once I changed the appName to a new name and deployed an updated to snowplow enrich , I was able to get rid of the error.