DynamoDB takes too much time to respond for a single item

Question

I have a Go REST service which connects to DynamoDB to retrieve some products. The table has around 100.000 products, and I usually retrieve around 20.0000 (which then get filtered down by other processes before actually replying but thats irrelevant to my problem).

The problem I see is my entire request takes around 4 seconds, which is very high for my goals. I started logging each layer in my app until finally reaching the smallest part which is each individual retrieval of an item from the DB.

To my surprise, each item retrieval takes approximately 3.6 seconds ??.!.

Regardless of the below code, what might be the causes of this? Is something I am missing on the configuration for dynamo?

func (db DynamoRepository) FetchProducts(c *gin.Context, pID string) ([]models.Product, error) {
    output := make([]models.Product, 0)
    chProduct := make(chan *models.Product)

    sem := make(chan struct{}, 40000)
    wait := sync.WaitGroup{}
    wait.Add(1)
    go func (wg *sync.WaitGroup) {
        defer wg.Done()
        for i := 0; i < len(itemIDs); i++ {
            p := <-chProduct
            if p != nil {
                output = append(output, *p)
            }
        }
        close(chProduct)
    }(&wait)
    for index := range itemIDs {
        sem <- struct{}{}
        go func(ix int, s chan struct{}, ch chan *models.Product) {
            defer func() {
               <- s
            }()
            infoSlice := strings.Split(itemIDs[ix], "_")
            if len(infoSlice) != 2 {
                ch <- nil
                return
            }
            key := map[string]*dynamodb.AttributeValue{
                "PK": {
                    N: aws.String(pID),
                },
                "SK": {
                    S: aws.String("product"),
                },
            }
            start := time.Now()
            result, err := db.DB.GetByKey(c, db.Config.AWS.Table, key)
            fmt.Printf("Item %s took %d\n", productID, time.Since(start).Milliseconds())
            if err != nil {
                ch <- nil
                return
            }
            if result.Item == nil {
                ch <- nil
                return
            }
            var product models.Product
            err = dynamodbattribute.UnmarshalMap(result.Item, &product)
            if err != nil {
                ch <- nil
                return
            }
            ch <- &product
        }(index, sem, chProduct)
    }
    wait.Wait()

    if len(output) == 0 {
        return nil, nil
    }
    return output, nil
}

The implementation of GetByKey:

func (provider *DynamoDBProvider) GetByKey(ctx context.Context, tableName string, key map[string]*dynamodb.AttributeValue) (*dynamodb.GetItemOutput, error) {
    input := &dynamodb.GetItemInput{
        TableName: aws.String(tableName),
        Key:       key,
    }

    result, err := provider.client.GetItem(input)

    return result, err
} 

Answer 1

This is what I ended up doing, and it greatly improved the latency. I followed this guide to create a custom HTTP client:

https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/custom-http.html

The latency went from almost 4 seconds to 500ms~. Why? I have no idea but it worked.