如何通过键索引存储在S3中的JSON文件？

Question

Suppose I want to store hundreds of JSON files in S3. All these JSON files have the same schema. I would like to search these JSON files by keys and values: eg find all JSON files with a key a value = "abc*" and a key x value = "xyz". I expect the search to return the file names and the keys that match the query.

What is the best way to index JSON files stored in S3 by keys?

This is a follow-up of my previous question

Answer 1

You might want to consider using S3 Select .

With Amazon S3 Select, you can use simple structured query language (SQL) statements to filter the contents of Amazon S3 objects and retrieve just the subset of data that you need. By using Amazon S3 Select to filter this data, you can reduce the amount of data that Amazon S3 transfers, which reduces the cost and latency to retrieve this data.

Amazon S3 Select works on objects stored in CSV, JSON, or Apache Parquet format.

Full docs on S3 Select .

Here's a nice blog post on how to use S3 Select .

https://aws.amazon.com/blogs/storage/querying-data-without-servers-or-databases-using-amazon-s3-select/

And a sample code would look like this:

import boto3

# S3 bucket to query (Change this to your bucket)
S3_BUCKET = 'greg-college-data'

s3 = boto3.client('s3')

r = s3.select_object_content(
        Bucket=S3_BUCKET,
        Key='COLLEGE_DATA_2015.csv',
        ExpressionType='SQL',
        Expression="select \"INSTNM\" from s3object s where s.\"STABBR\" in ['OR', 'IA']",
        InputSerialization={'CSV': {"FileHeaderInfo": "Use"}},
        OutputSerialization={'CSV': {}},
)

for event in r['Payload']:
    if 'Records' in event:
        records = event['Records']['Payload'].decode('utf-8')
        print(records)

Code soruce .