如何从Warc文件中读取记录的子集

Question

我正在尝试从Python的Common Crawl中解析.warc文件。

由于文件很大，因此我想从头几条记录的样本/子集开始。

如何截断文件，使其仅包含前X行，同时保留到位的换行符/回车符？

这是我已经尝试过的方法：

1. head -n 250 oldfile > newfile这将删除解析文件所需的一些返回。 如果我尝试在Hadoop作业中使用此文件（通过warc包读取它）， warc ：

     Traceback (most recent call last): File "test.py", line 46, in <module> TagGrabber.run() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/job.py", line 461, in run mr_job.execute() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/job.py", line 479, in execute super(MRJob, self).execute() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/launch.py", line 151, in execute self.run_job() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/launch.py", line 214, in run_job runner.run() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/runner.py", line 464, in run self._run() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/sim.py", line 173, in _run self._invoke_step(step_num, 'mapper') File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/sim.py", line 264, in _invoke_step self.per_step_runner_finish(step_num) File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/local.py", line 152, in per_step_runner_finish self._wait_for_process(proc_dict, step_num) File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/local.py", line 268, in _wait_for_process (proc_dict['args'], returncode, ''.join(tb_lines))) Exception: Command ['sh', '-ex', 'setup-wrapper.sh', '/var/cc-mrjob/venv/bin/python', 'test.py', '--step-num=0', '--mapper', '/tmp/test.root.20150520.071726.549519/input_part-00000'] returned non-zero exit status 1: Traceback (most recent call last): File "test.py", line 46, in <module> TagGrabber.run() File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 461, in run mr_job.execute() File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 470, in execute self.run_mapper(self.options.step_num) File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 535, in run_mapper for out_key, out_value in mapper(key, value) or (): File "/var/cc-mrjob/mrcc.py", line 33, in mapper for i, record in enumerate(f): File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 390, in __iter__ record = self.read_record() File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 373, in read_record header = self.read_header(fileobj) File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 331, in read_header raise IOError("Bad version line: %r" % version_line) IOError: Bad version line: 'WARC/1.0\\n' 

2. 与＃1相同，但带有tail命令

3. 与＃1相同，但在之后使用tr或sed替换丢失的换行符或^M （回车符）。 这导致warc程序包仍然抱怨预期的回车或换行符不到位。
4. unix2dos oldfile

Answer 1

正确处理换行符将很困难，因为.warc文件也可能包含二进制数据。 截断也可能会产生损坏的.warc文件，因为例如python库相信Content-Length标头是有效的。

warc python库一次仅从.warc文件中读取一条记录（避免一次将整个文件读取到内存中），因此可以仅使用python处理子集。 例如：

import warc
from itertools import islice

N = 10
warc_file = warc.open('/path/to/file.warc')
for record in islice(warc_file, N):
    do_stuff_with(record)