通过Athena和/或Glue将S3数据ETL转换为CSV

Question

I have an S3 bucket ( com.example.myorg.images ) full of image files, all of them following the same naming convention:

<PRODUCT_ID>_<NUMBER>.jpg

Where <PRODUCT_ID> is a long number (a primary key in an RDS table), and <NUMBER> is always one of three values: 100, 200 or 300. So for example, the bucket might contain:

• 1394203949_100.jpg
• 1394203949_200.jpg
• 1394203949_300.jpg
• 1394203950_100.jpg
• 1394203950_200.jpg
• 1394203950_300.jpg
• ...etc.

I would like to write either an Athena or Glue ETL process that queries the S3 bucket for all the images inside of it, and somehow, extracts the UNIQUE <PRODUCT_ID> values into a table or list.

It's my understanding that Athena will then back up this table/list into a downloadable CSV; if true, then I will, separately, process that headerless CSV the way I need it on the command-line.

So for instance, if the 6 images above were the only images in the bucket, then this process would:

1. Query S3 and obtain a table/list consisting of 1394203949 and 1394203950
2. Create a downloadable CSV looking like this:

Could be a file on S3 or even in-memory:

1394203949,1394203950

Having no prior experience with either Athena or Glue, I'm attempting to accomplish this with an Athena query, but I'm having difficulty seeing the forest through the trees.

My best attempt at the 1st part (the S3 query):

CREATE EXTERNAL TABLE IF NOT EXISTS products_with_thumbnails (
  product_id string
) 
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY ','
  ESCAPED BY '\\'
  LINES TERMINATED BY '\n'
LOCATION 's3://com.example.myorg.images/';

Which would set up my in-memory table I believe with the file names of everything in the S3 bucket, but then:

• How do I make this table contain only unique product IDs (no dupes)?
• How do I extract out only the <PRODUCT_ID> segment of the filenames ( 1394203949 as opposed to 1394203949_100.jpg )?

I'm not partial to Athena or Glue, and would be happy with any solution that accomplishes what I need. Worst case I could write a Lambda that accomplishes all of this ETL at the application layer, but if there is a Hive-like or ETL-oriented AWS service that exists for doing this kind of stuff anyways, I'd rather just leverage that!

Thanks in advance!

Answer 1

Athena queries inside of files, not file listings, so using only Athena for this will not work (there are ways of abusing it to make it happen, but they will be expensive and slow and not what you want).

If the number of images is less than a hundred thousand or so I think your best bet is to just write a script that does more or less the equivalent of aws s3 ls --recursive s3://some-bucket/ | perl -ne '/(\\d+)_d+\\.jpg$/ && print "$1\\n"' | uniq aws s3 ls --recursive s3://some-bucket/ | perl -ne '/(\\d+)_d+\\.jpg$/ && print "$1\\n"' | uniq aws s3 ls --recursive s3://some-bucket/ | perl -ne '/(\\d+)_d+\\.jpg$/ && print "$1\\n"' | uniq .

If it's more than that I suggest using S3 Inventory and perhaps Athena for the processing. You can find instructions on how to enable S3 Inventory, and query the inventory with Athena here: https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html

With an S3 Inventory set up your query could look something like this:

SELECT DISTINCT regexp_extract(key, '(\d+)_\d+\.jpg', 1)
FROM the_inventory_table_name

It might be less work to write a script that processes the inventory than setting up Athena tables, though. I really recommend using S3 Inventory instead of listing S3 directly when there are many objects, though.

Answer 2

Looks like you can create a partitioned file of your S3 inventory, in S3, partitioned by date:

CREATE EXTERNAL TABLE my_inventory(
  `bucket` string,
  key string,
  version_id string,
  is_latest boolean,
  is_delete_marker boolean,
  size bigint,
  last_modified_date timestamp,
  e_tag string,
  storage_class string,
  is_multipart_uploaded boolean,
  replication_status string,
  encryption_status string,
  object_lock_retain_until_date timestamp,
  object_lock_mode string,
  object_lock_legal_hold_status string
  )
  PARTITIONED BY (dt string)
  ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
  STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
  OUTPUTFORMAT  'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
  LOCATION 's3://com.example.myorg.mybucket/com.example.myorg.mybucket/com.example.myorg.mybucket-ORC/hive/';

Then, anytime you want to query that my_inventory table, first repair the partitioned file by creating a new partition for the current date:

MSCK REPAIR TABLE my_inventory;

And finally you can query it via PrestoDB's SQL-like syntax:

SELECT key FROM my_inventory WHERE dt <= '<YYYY-MM-DD>-00-00';

Where <YYYY-MM-DD> is the current date in YYYY-MM-DD format.

You can then download the query results as a CSV file and process it however you like.