WARC 文件中的记录数

Question

I currently parsing WARC files from CommonCrawl corpus and I would like to know upfront, without iterating through all WARC records, how many records are there.

Does WARC 1.1 standard defines such information?

Answer 1

The WARC standard does not define a standard way to indicate the number of WARC records in the WARC file itself. The number of response records in Common Crawl WARC files is usually between 30,000 and 50,000 - note that there are also request and metadata records. The WARC standard recommends 1 GB as target size of WARC files which puts a natural limit to the number of records.