Lambda with DynamoDB Trigger on a table Partition Key with more than 500000 distinct values

Question

We are currently designing a dynamodb table to store certain file attributes. There are 2 main columns

1. Date:- This contains the date in YYMMDD format for ex:-20190618
2. FileName:- xxxxxxxxxxx.json

Currently the partition key is Date and sort key is FileName. We expect about 500000 files with distinct file names on each day (this can increase over period of time) . The File names will repeated same each day ie a typical schema is as shown below

Date FileName 20190617 abcd.json 20190618 abcd.json

We have a series of queries that is based on Date and a dynamodb trigger. The queries are working great. Currently what we are observing is that the number of concurrent lambda executions are limited to 2, since we are partition by date. While trying to improve the concurrency of lambda we came across 2 solutions

1) Referring the following link ( https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-sharding.html ) , one idea is add a fixed number of random suffixes for Date Field ie (20190617.1 to 20190617.500) to split the data in to 500 partitions with 1000 records each . This would ensure an amount of concurrency and also there will be minimal changed to query

2) Second option is to change partitioning of table as follows Partition Key :- FileName and SortKey :- Date. This will result in about 500000 partitions , (which can increase) . For querying by date we will need to add a GSI, but we will achieve more concurrency in Lambda

we have not created a table with 500000 partitions (which can increase). Any body has such experience... If so please comment

Any help is appreciated

Answer 1

You seem to be under the mistaken impression that there's a one to one correspondence between partition keys and partitions.

This is not the case.

The number of partitions is driven by table size and through-put. The partition key is hashed by DDB and the data is stored in a particular partition.

You could have 100k partition keys and only a single partition.

If you're pushing the limits of DDB, then yeah you might end up with only a single partition key in a partition...but that's not typical.

The DDB Whitepaper provides some details into how DDB works...

Answer 2

Partitioning by file name doesn't make a lot of sense if your access pattern is to query by date.

Instead, the idea of increasing the number of partitions for each date by adding a suffix seems fine. But rather than adding a random suffix, you might consider adding a stable suffix based on the name of the file:

You could use the first letter of the file name, to get about 30 partitions - assuming the file names are random. The only trouble is some letter might be more common than others giving skewed subpartitions

Or, you could take a hash of the file name and use that as the suffix for the partition key. The hash function could be a relatively simple hash function that produces a target numeric value corresponding to the number of sub partitions you would like to have for each date.

If you end up with about 10000-50000 items per partition it would probably be great.

Hope this helps