无法从从普通爬网爬网的warc文件中找到URL

Question

I have crawled data from common crawl and I want to find out url corresponding to each of the records.

for record in files:
     print record['WARC-Target-URI']

This outputs an empty list. I am referring to the following link https://dmorgan.info/posts/common-crawl-python/ . Do we get target uri corresponding to each of the record or just one target uri for one warc file path ?

Answer 1

The info you're after is part of the header. Try:

print record.header['WARC-Target-URI']