DynamoDB 数据建模 - 作为项目的分层数据结构

Question

The access patterns that I'm interested in is the last item for a given exchange and an account name.

+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
|              PK               |                    SK                     | Account |                  Address                   | AddressTag | Exchange | Instrument | Network | Quantity  | TransactionFee |                           TransactionId                            |       TransferDate        |            TransferId            | TransferType |          UpdatedAt           |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| Exchange#Binance#Account#main | TransferDate#12/17/2022 4:59:12 PM +02:00 | main    | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | Binance  | USDT       | ETH     | 97.500139 |            3.2 | 0x46d28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-17T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 |            0 | 2023-01-28T20:19:59.9181573Z |
| Exchange#Binance#Account#main | TransferDate#12/17/2022 5:38:23 PM +02:00 | main    | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | Binance  | USDT       | ETH     | 3107.4889 |            3.2 | 0xbb2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2022-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 |            0 | 2023-01-28T20:20:00.4536839Z |
| Exchange#FTX#Account#main     | TransferDate#12/17/2021 5:38:23 PM +02:00 | main    | 0x476d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | FTX      | USDT       | ETH     |        20 |            3.2 | 0xaa2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2021-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 |            0 | 2023-01-28T20:20:00.5723855Z |
| Exchange#FTX#Account#main     | TransferDate#12/19/2022 4:59:12 PM +02:00 | main    | 0xc46d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | FTX      | USDT       | ETH     |        15 |            3.2 | 0xddd28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-19T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 |            0 | 2023-01-28T20:20:00.5207119Z |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+

First of all, it seems to be working as expected but as I'm still learning I'm not so sure whether the partition key and the sort key I chose are good enough or not. This is important as "Uneven distribution of data due to the wrong choice of partition key" can cause reading/writing above the limit issues.

There was a similar example in the documentation and what they say about TransactionId being a partition key is as following:

In most cases you won't use TransactionID for any query purposes, so you lose the ability to use the partition key to perform a fast lookup of data. To expand this reasoning, consider the traditional order history view on an e-commerce site. Normally orders are retrieved by customer ID or Order ID, not a UID such as a transaction ID that was synthetically generated during checkout. It's better to choose a natural partition key than generate a synthetic one that won't be used for querying.

Another interesting part of the documentation is about the composite key

Composite sort keys let you define hierarchical (one-to-many) relationships in your data that you can query at any level of the hierarchy

[country]#[region]#[state]#[county]#[city]#[neighborhood]

This would let you make efficient range queries for a list of locations at any one of these levels of aggregation, from country, to a neighborhood, and everything in between.

I'm also interested in the "Get all user transfers by date range" access pattern but I'm not sure how I could achieve it. So here we are.

C# implementation

public async Task<UserTransferDto?> GetLastAsync(string exchange, string account)
{
    var queryRequest = new QueryRequest
    {
        TableName = TableName,
        KeyConditionExpression = "#pk = :pk",
        ExpressionAttributeNames = new Dictionary<string, string>
        {
            { "#pk", "PK" }
        },
        ExpressionAttributeValues = new Dictionary<string, AttributeValue>
        {
            { ":pk", new AttributeValue { S = $"Exchange#{exchange}#Account#{account}" } }
        },
        ScanIndexForward = false,
        Limit = 1
    };

    var response = await _dynamoDb.QueryAsync(queryRequest);
    if (response.Items.Count == 0)
    {
        return null;
    }

    var itemAsDocument = Document.FromAttributeMap(response.Items[0]);
    return JsonSerializer.Deserialize<UserTransferDto>(itemAsDocument.ToJson());;
}

public class UserTransferDto
{
    [JsonPropertyName("PK")]
    public string Pk => $"Exchange#{Exchange}#Account#{Account}";

    [JsonPropertyName("SK")]
    public string Sk => $"TransferDate#{TransferDate}";

    public required string Exchange { get; init; }

    public required string Account { get; init; }

    public required DateTimeOffset TransferDate { get; init; }

    public required string TransferId { get; init; }

    public required TransferType TransferType { get; init; }

    public required string Instrument { get; init; }

    public required string Network { get; init; }

    public required decimal Quantity { get; init; }

    public required string Address { get; init; }

    public string? AddressTag { get; init; }

    public decimal? TransactionFee { get; init; }

    public string? TransactionId { get; init; }

    public DateTime UpdatedAt { get; set; }
}

public enum TransferType
{
    Withdraw = 0,
    Deposit = 1
}

Sources:

• https://youtu.be/HaEPXoXVf2k?t=720
• https://youtu.be/HaEPXoXVf2k?t=798
• Hierarchical Data Structures as Items https://youtu.be/HaEPXoXVf2k?t=2775 Access Patterns https://youtu.be/HaEPXoXVf2k?t=2903
• https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/
• https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html

Answer 1

Your base table design works well for getting the latest item for a given exchange and account (via a Query with that as the PK and getting the last sortable from the SK), except that you're using non-sortable human time stamps instead of sortable time stamps. You should use 2023-01-28 12:56:08 and so on so that the times sort right as strings.

For the other query to find the latest across all exchanges and accounts, you can create a GSI which has a singular PK and the times as the SK. Just beware that you're limited in how many writes per second you can do to the same PK. Above 1,000 write units per second you'll need to shard it and then do a query for each shard to get the latest per shard and then the latest overall.

This is a pattern described in https://youtu.be/0iGR8GnIItQ