映射一系列warc.gz文件，EMR

Question

I have been running a streaming step in AWS/EMR with a mapper and reducer written in Python to map some of the archives in Common Crawl for sentiment analysis.

I am moving from the older common crawl textData format to the newer warc.gz format and I need to know how I might go about specifying a range of warc.gz files for my EMR input.

For example:

In the older format I could specify an input range of textData files as such:

s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690165636/textData-000[0-9][0-9]

but the new format looks like this:

first file:

s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2016-07/segments/1454702039825.90/warc/CC-MAIN-20160205195359-00000-ip-10-236-182-209.ec2.internal.warc.gz

second file:

s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2016-07/segments/1454702039825.90/warc/CC-MAIN-20160205195359-00001-ip-10-236-182-209.ec2.internal.warc.gz

How would I specify to map a range of these warc.gz files?

Answer 1

I'm pretty sure you can use the same method you were using previously. To just read the two files you would use:

s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2016-07/segments/1454702039825.90/warc/CC-MAIN-20160205195359-0000[0-1]-ip-10-236-182-209.ec2.internal.warc.gz

Also since these paths are richer than the previous one you have additional ways to specify sets of data to process.

CC-MAIN-2016-07 is CC-MAIN-YYYY-ww - Ability to specify a set of years or weeks to process.

CC-MAIN-20160205195359 is CC-MAIN-YYYYMMDDHHmmss - You can choose a date or time range.

Answer 2

You can download the list of warc file of july 2016 via

https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-30/warc.paths.gz
https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-30/wat.paths.gz
https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-30/wet.paths.gz

for accessing via browser attach this to the path mentioned in the file

commoncrawl.s3.amazonaws.com/

In Your case to access via s3 try appending this to the path

s3://commoncrawl/