ValidationException - 成员的长度必须小于或等于 500

Question

My Snowplow enricher loads data to a Kinesis Data Stream but Kinesis responds with error 400 and the message Member must have length less than or equal to 500 (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException and I cannot find this exception in the AmazonKinesis documentation. I also do not understand the message, 500 what, bytes? When I look at Kinesis within the AWS console, there is no setting set to 500 and also I do not see anything related to maximum sizes.

Can someone tell me what this 500 is about?

Answer 1

The error means that the Enrich app is trying to write more than 500 records to Kinesis at once. The PutRequests action of the Kinesis API has a limit of 500 entries. This is documented here .

In your Enrich config file, you have buffer section , where you can specify:

      byteLimit = 10000000
      recordLimit = 480 
      timeLimit = 60000

A flush is triggered when one of those limits is hit, whichever occurs first.