Maximum Partition key length of my data in Dynamo DB

Question

I have an use case to place constraints on the key size in my application. I tried to find the max length of partition key so far in my DynamoDB table. This will help me to know my data before placing any internal constraints on the data that I am using as a partition key in Dynamo DB.

Example: Let's say here is my table with a partition key (idempotent_id). I want to know the max length of partition keys so far (in this case 7).

idempotent_id
1234
12
1234567
12345

I tried using Dynamo DB console from my aws account. I looked at query and scan api of DynamoDB. But nothing seems good fit for me. May be this is something we can't find using DynamoDB? or may be I am searching wrongly? Any help would be appreciated.

Answer 1

Take a look at data types section and their limits: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html

String

The length of a String is constrained by the maximum item size of 400 KB.

Strings are Unicode with UTF-8 binary encoding. Because UTF-8 is a variable width encoding, DynamoDB determines the length of a String using its UTF-8 bytes.

Number

A Number can have up to 38 digits of precision, and can be positive, negative, or zero.

Positive range: 1E-130 to 9.9999999999999999999999999999999999999E+125 Negative range: -9.9999999999999999999999999999999999999E+125 to -1E-130 DynamoDB uses JSON strings to represent Number data in requests and replies. For more information, see DynamoDB Low-Level API.

If number precision is important, you should pass numbers to DynamoDB using strings that you convert from a number type.

Binary

The length of a Binary is constrained by the maximum item size of 400 KB.

Applications that work with Binary attributes must encode the data in Base64 format before sending it to DynamoDB. Upon receipt of the data, DynamoDB decodes it into an unsigned byte array and uses that as the length of the attribute.

Answer 2

From your comment :

I am trying to find the maximum size/length of idempotent_id so far in my single table.

In order to do this without any auxiliary data, you will need to perform a full table scan and get the result attributes you care for from each item. You can use a ProjectionExpressions to reduce the amount of data retrieved.

You could store the value in another attribute and create a GSI on that which will give you the ability to query that index in an ordering.

Another option would be to use something like DynamoDB Streams to listen to events and keep track of the max size in a different storage medium.

Answer 3

Partition Keys and Sort Keys Partition Key Length The minimum length of a partition key value is 1 byte. The maximum length is 2048 bytes.

Partition Key Values There is no practical limit on the number of distinct partition key values, for tables or for secondary indexes.

Sort Key Length The minimum length of a sort key value is 1 byte. The maximum length is 1024 bytes.

Sort Key Values In general, there is no practical limit on the number of distinct sort key values per partition key value.

The exception is for tables with secondary indexes. With a local secondary index, there is a limit on item collection sizes: For every distinct partition key value, the total sizes of all table and index items cannot exceed 10 GB. This might constrain the number of sort keys per partition key value. For more information, see Item Collection Size Limit.

Answer 4

Range Key Example

string = '1USER:01G23WSPRVXA8BWK5ERD0TDYT501G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sljs;gjasjfdaskfjas;dfjaskdfjaskldfjkasjfjkasjfjlsjdfjskjfjlsajf;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sljs;gjasjfdaskfjas;dfjaskdfjaskldfjkasjfjkasjfjlsjdfjskjfjlsajf;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sljs;gjasjfdaskfjas;dfjaskdfjaskldfjkasjfjkasjfjlsjdfjskjfjlsajf;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sl'

string.length => 1024

You can also check string bytes HERE

What is interesting, is if I add one more character to this string and then create a new item with this string as the range key I get the following error.

UnhandledPromiseRejectionWarning: ValidationException: Hash primary key values must be under 2048 bytes, and range primary key values must be under 1024 bytes

The measured reality of it is that the range key length can be <= 1024 where the error message says only that of < 1024. The same difference for the Primary Hash Key of 2048; The warning says < 2048 but the database excepts <= 2048.

Note: Both keys must be at least one character. The following is an error message when the Range key is a blank string.

UnhandledPromiseRejectionWarning: ValidationException: One or more parameter values are not valid. The AttributeValue for a key attribute cannot contain an empty string value. Key: SK

Primary (Hash) Key Example

string = '01G23XW4MCMTGMRWD4R78NMA29:USERS31:1USER:01G23WSPRVXA8BWK5ERD0TDYT501G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sljs;gjasjfdaskfjas;dfjaskdfjaskldfjkasjfjkasjfjlsjdfjskjfjlsajf;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sljs;gjasjfdaskfjas;dfjaskdfjaskldfjkasjfjkasjfjlsjdfjskjfjlsajf;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;sljs;gjasjfdaskfjas;dfjaskdfjaskldfjkasjfjkasjfjlsjdfjskjfjlsajf;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;slskahlsfdjlashghjskhglhaskjghlhsdghjkahskghlahkdhgkjashdjkghjakshghasjkdhg;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;slskahlsfdjlashghjskhglhaskjghlhsdghjkahskghlahkdhgkjashdjkghjakshghasjkdhg;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;slskahlsfdjlashghjskhglhaskjghlhsdghjkahskghlahkdhgkjashdjkghjakshghasjkdhg;jslfjklsajjj;sdfjsfjas;fjsa;fjs;ldfkjsadk;lsfasjjf;jfj;jdksfjsljfksfksf;afjs;lfjsljflsjkssdjfsjfsljskdkdkdkdkdkdkddkdkksajglks;jgsag;ljd;lagjdkllgkjsklgj01G0G9WD6E1JXPZTFBJAAFKJ9N:CLIENTONE:NAVBAR:V1:lsj;lajgjsldglsjgjasgjsjdgjsdjgjajjg;slskahlsfdjlashghjskhglhaskjghlhsdghjkahsdsagsdgdggg'


string.length => 2048