how to use DISTINCT in amazon s3 query

Question

I am trying to filter distinct id's using the amazon s3 select query. I tried the below query but I am getting errors running it. Could someone help me why I am getting an error?. Please suggest a proper query

SELECT distinct s.f0 FROM s3object s

Error

Unsupported SQL operation DISTINCT. Please check the service documentation for supported operations.

Answer 1

Amazon S3 Select does not support the DISTINCT operator.

S3 Select is not a full query engine. Instead, it is a quick way to filter the contents of a file. The S3 Select 'engine' simply progresses through each line and returns data if applicable. It is not 'smart' enough to do a DISTINCT operation.

If you want more SQL capabilities, consider using Amazon Athena. It can run SQL queries against data stored in Amazon S3, even across multiple files. It supports a much larger range of SQL functions.