使用AWS Cli访问DynamoDB中的AWS Kinesis表

Question

I am new to AWS Kinesis. Trying to learn how to build distributed application using AWS DynamoDB. Could someone tell me how to access the tables in DynamoDB used by my streams in AWS Kinesis using the AWS Cli? Can I query the tables?

Answer 1

Surely. Kinesis-stream uses Amazon Dynamodb table for consumer offset management.

• your table will have the same name as your consumer name (defined via KCL). you can see all consumer_offset_tables via console ( https://console.aws.amazon.com/dynamodb/home )

• consumer could be running in multiple instances, but only one instance (THE leaseOwner )can consume from one partition (or Shard what Kinesis calls it). If this consumer fails another instance of the same consumer will take over and continue processing from the checkpoint .

• checkpoint is the last processed event

• the shard that the consumer instance is processing is called leaseKey which is unique for a table.

• The data structure of the key value based document in Dynamodb would be as below,

where

"S" - Char array

"N" - Number

{
  "leaseOwner": {
    "S": "SmartConsumerStream_Consumer-192.168.1.83"
  },
  "checkpoint": {
    "S": "49570630332110756564477900867375857710984404992079691778"
  },
  "checkpointSubSequenceNumber": {
    "N": "0"
  },
  "leaseCounter": {
    "N": "16"
  },
  "leaseKey": {
    "S": "shardId-000000000000"
  },
  "ownerSwitchesSinceCheckpoint": {
    "N": "0"
  }
}

• You can use Dynamodb API to get the current offset for given partitionKey or leaseKey . You can only query by leaseKey because thats the indexed key in table. Its created by Kinesis-stream itself.

• You can use the stream-driver I'm writing at here , which gives you interface to get the consumer offset very very easily.

• Here are kinesis-stream tests which might be helpful too.

In summary, get the consumer offset using JAVA api

-- put aws credentials in ~/.aws/credentials

public Map<String, String> getConsumerPosition() {
    DynamoDB dynamoDB = new DynamoDB(getOffsetConnection()); //
    Table consumerOffsetTable = dynamoDB.getTable("your_consumer_id");

    Map<String, Object> leaseOwner = consumerOffsetTable.getItem("leaseKey", "shardId-000000000000").asMap();

    return new HashMap<String, String>(){{
        put(leaseOwner.get("leaseKey").toString(), leaseOwner.get("checkpoint").toString());
    }};
}

public AmazonDynamoDB getOffsetConnection() {
    AmazonDynamoDBClient dynamoDB = new AmazonDynamoDBClient(getAuthProfileCredentials(), getHttpConfiguration());

    return dynamoDB;
}

private ProfileCredentialsProvider getAuthProfileCredentials() {
    return new ProfileCredentialsProvider("your-aws-auth-profile_in_~/.aws/credentials");
}


private ClientConfiguration getHttpConfiguration() {

    ClientConfiguration clientConfiguration = new ClientConfiguration();

    return clientConfiguration;
}

Hope it helps, let me know if I can help some other ways.