从 s3 存储桶中获取唯一文件

Question

I am worknig on .net c# project. My code is fetching files from s3 bucket. In s3 bucket there are some duplicate files but their names are different. I want to fetch only unique files from s3 bucket. I am using this query to fetch all files from s3 bucket

ListObjectsV2Request listRequest = new ListObjectsV2Request { BucketName = awsBucketName, Prefix = fullPath };
var listResult = await client.ListObjectsV2Async(listRequest);
var obj = listResult.S3Objects.Where(x => x.Key.EndsWith(".pdf") && x.Size > 0)
                    .OrderByDescending(x => x.LastModified)

How to get files with unique content to avoid duplicates?

Should I just read all files 1 by 1 and remove duplicate ones? Or is there any easy way to avoid duplicates?

Answer 1

You can group by the ETag which files that have the same content will have the same Etag and grouping them by ETag will group all the duplicates so you can choose only one of them

ListObjectsV2Request listRequest = new ListObjectsV2Request { BucketName = awsBucketName, Prefix = fullPath };
var listResult = await client.ListObjectsV2Async(listRequest);
var obj = listResult.S3Objects.Where(x => x.Key.EndsWith(".pdf") && x.Size > 0)
                    .OrderByDescending(x => x.LastModified).GroupBy(x => x.ETag)
                    .Select(x => x.First())
                    .ToList();