如何使用 Python 从多个 7z 中提取多个文件？

Question

I need to extract 700k jpg files that are dispersed among 50 7z files. I have a txt file that has one row for each file I need. The row contains the target 7z file and location and file name.

This is what the txt file looks like:

A20000to22000.7z, A20000to22000/rows/A21673.Lo1sign.jpg
B20000to22000.7z, B20000to22000/rows/B21673.Lo1sign.jpg

I currently am able to extract files with Python but only from one 7z at a time. I use this command to do that:

7zz e A0000to22000.7z @f1.txt

This is taking way too long though. Is there anyway to edit the command or use another approach so I can extract many different files from many different 7z files at once?

Answer 1

Updated Answer

With the new information that there are lots of files to retrieve from each archive, a modified approach is needed.

First we must generate a list of the files needed from each 7z archive, then process that list in parallel. So this code should do that:

awk -F, '{sub("7z","txt",$1); print $2 > $1}' joblist.txt

That should make a file called A20000to22000.txt that contains all the files to be extracted from the archive A20000to22000.7z and similarly for B20000to22000.7z it should produce B20000to22000.txt .

Don't proceed past here till the files ending in .txt look correct.

Now we need to process the .txt files in parallel with GNU Parallel . That should look something like this:

parallel --dry-run 7zz e {.}.7z @{} ::: *to*.txt 

I used *to*.txt in order to avoid processing the original joblist.txt .

If that command looks correct, remove --dry-run and run for real.

Original Answer

Assuming joblist.txt looks like this:

A20000to22000.7z, A20000to22000/rows/A21673.Lo1sign.jpg
B20000to22000.7z, B20000to22000/rows/B21673.Lo1sign.jpg

and that corresponds to needing to run a command like:

7zz e A20000to22000.7z A20000to22000/rows/A21673.Lo1sign.jpg

you can do that in parallel with GNU Parallel like this:

parallel --dry-run --colsep , 7zz e {1} {2} :::: joblist.txt

If it looks right, remove --dry-run and run for real.

---

Note that this is done in the terminal/shell and without Python, so it falls under the "another approach" you mentioned.