error “security token included in the request is invalid” from AWS Client

Question

Hi I'm using Kinesis as my data buffer, and I got this error below, indicating that my access key is invalid. I tried to google around, and other posting answers suggest to change the access key to be the right one.

But my case is somehow special, this error comes at run-time, which means the access key is correct when my application started but after running for some time the access key is invalid all of a sudden.

I suspect "AmazonCloudWatch" and "AmazonDynamoDBv2" service client library is causing the problem? Or my way to create "AmazonKinesisClient" is not correct?

I'm not authorized to post thread under these two service on AWS forum, so I post it here. Thank you very much.

Scala code to initiate AmazonKinesisClient:

  val awsCredentialsProvider = new DefaultAWSCredentialsProviderChain()
  val kinesisClient = new AmazonKinesisClient(awsCredentialsProvider)

error msg:

15/10/21 17:05:48 INFO CWPublisherRunnable: Successfully published 20 datums.
15/10/21 17:05:49 INFO CWPublisherRunnable: Successfully published 3 datums.
15/10/21 17:05:59 INFO CWPublisherRunnable: Successfully published 18 datums.
15/10/21 17:06:00 INFO BlockManager: Removing RDD 115350
15/10/21 17:06:00 INFO BlockManager: Removing RDD 115349
15/10/21 17:06:00 INFO BlockManager: Removing RDD 115348
15/10/21 17:06:00 INFO MemoryStore: ensureFreeSpace(3016) called with curMem=40094, maxMem=186832650
15/10/21 17:06:00 INFO MemoryStore: Block input-0-1445447160600 stored as values in memory (estimated size 2.9 KB, free 178.1 MB)
15/10/21 17:06:00 INFO BlockGenerator: Pushed block input-0-1445447160600
15/10/21 17:06:05 INFO MemoryStore: ensureFreeSpace(2792) called with curMem=43110, maxMem=186832650
15/10/21 17:06:05 INFO MemoryStore: Block input-0-1445447165600 stored as values in memory (estimated size 2.7 KB, free 178.1 MB)
15/10/21 17:06:05 INFO BlockGenerator: Pushed block input-0-1445447165600
15/10/21 17:06:09 INFO CWPublisherRunnable: Successfully published 20 datums.
15/10/21 17:06:09 INFO CWPublisherRunnable: Successfully published 5 datums.
15/10/21 17:06:12 INFO Worker: Current stream shard assignments: shardId-000000000001, shardId-000000000000
15/10/21 17:06:12 INFO Worker: Sleeping ...
15/10/21 17:06:18 INFO BlockManager: Removing RDD 115354
15/10/21 17:06:18 INFO BlockManager: Removing RDD 115353
15/10/21 17:06:18 INFO BlockManager: Removing RDD 115352
15/10/21 17:06:19 INFO CWPublisherRunnable: Successfully published 18 datums.
15/10/21 17:06:21 INFO MemoryStore: ensureFreeSpace(2536) called with curMem=45902, maxMem=186832650
15/10/21 17:06:21 INFO MemoryStore: Block input-0-1445447181600 stored as values in memory (estimated size 2.5 KB, free 178.1 MB)
15/10/21 17:06:21 INFO BlockGenerator: Pushed block input-0-1445447181600
15/10/21 17:06:29 INFO CWPublisherRunnable: Successfully published 20 datums.
15/10/21 17:06:30 WARN CWPublisherRunnable: Could not publish 5 datums to CloudWatch
com.amazonaws.AmazonServiceException: The security token included in the request is invalid. (Service: AmazonCloudWatch; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: 12560958-7816-11e5-8781-e3e348826d8f)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:883)
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:354)
    at com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:63)
    at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:129)
    at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:75)
    at java.lang.Thread.run(Thread.java:744)
15/10/21 17:06:31 ERROR LeaseCoordinator: LeasingException encountered in lease renewing thread
com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.AmazonServiceException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: P76PVB575SQUA5QPDTD4BSDDTJVV4KQNSO5AEMVJF66Q9ASUAAJG)
    at com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:557)
    at com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:371)
    at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:102)
    at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:83)
    at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator.runRenewer(LeaseCoordinator.java:194)
    at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable.run(LeaseCoordinator.java:130)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.amazonaws.AmazonServiceException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: P76PVB575SQUA5QPDTD4BSDDTJVV4KQNSO5AEMVJF66Q9ASUAAJG)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2908)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1256)
    at com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:362)
    ... 11 more

Answer 1

This happens because credentials are rotated and sometimes you try to access kinesis with credentials that are just expired.

The solution suggested by Amazon is to refresh the credentials 5 minutes before the expiration. More details here https://aws.amazon.com/premiumsupport/knowledge-center/security-token-expired/