查询 50 GB 数据时哪些性能更​​好？ 是带有条件的 MYSQL SELECT 还是带有过滤器表达式的 Dynamodb SCAN？

Question

I'm retrieving some traffic data of a website using "scan" option in Dynamodb. I have used filterExpression to filter those out. I will be doing scanning against a large table which will have more than 20GB of data.

I found that DynamoDB scans throguh the entire table and filter the results out. The document says it only returns 1MB of data and then i have to loop through again to get the rest. It seems to be bad way to make this work. got the reference from here: Dynamodb filter expression not returning all results

For a small table that should be fine.

MySQL dose the same I guess. I'm not sure.

Which is faster to read is it MySQL select or DynamoDB scan on a large set of data. ?

Is there any other alternative? what are your thoughts and suggestions?

I'm trying to migrate those traffic data into Dynamodb table and then query it out. It seems like a bad idea to me now.

$params = [
      'TableName' => $tableName,
      'FilterExpression' => $this->filter.'=:'.$this->filter.' AND #dy > :since AND #dy < :now',
      'ExpressionAttributeNames'=> [ '#dy' => 'day' ],
      'ExpressionAttributeValues'=> $eav
    ];

    var_dump($params);

    try {
      $result = $dynamodb->scan($params);

After considering the suggestion this is what worked for me

$params = [ 
'TableName' => $tableName,
 'IndexName' => self::GLOBAL_SECONDARY_INDEX_NAME, 
'ProjectionExpression' => '#dy, t_counter , traffic_type_id', 'KeyConditionExpression' => 'country=:country AND #dy between :since AND :to', 
'FilterExpression' => 'traffic_type_id=:traffic_type_id' 'ExpressionAttributeNames' => ['#dy' => 'day'],
'ExpressionAttributeValues' => $eav 
]; 

Answer 1

If your data is like Key-Value pair and you have fixed fields on which you want to index, use DynamoDB - you can create indexes on all fields you want to query and it will work great

If you require complex querying on multiple indexes, then any RDBMS is good.

If you can query on just about anything, think about Elastic search

If your queries are very simple, but you have large data to be retrieved in each query. Think about S3 . Maybe you can index metadata in DynamoDb and actual data can be in S3