[英]Python wget download multiple files at once
寻找一次下载多个文件的干净的 Python Wget 解决方案。
网址将始终相同:
https://example.com/
到目前为止,我可以这样做:
import wget
print('Beginning file download with wget module')
url = 'https://example.com/new_folder/1.jpg'
wget.download(url)
但我还需要下载 -2.jpg, -3.jpg , -4.jpg, -5.jpg 并将 NWZV1WB 重命名为NEWCODE-1.jpg , NEWCODE-2.jpg ...
此外,我需要下载文件夹中的所有 content(22).jpg 文件,并将文件夹本地重命名为NEWCODE 之类的名称,但保留文件的原始名称
这里的 url 也总是相同的:
import wget
print('Beginning file download with wget module')
url = 'https://example.com/big/1.jpg' #there's 18 jpg inside
wget.download(url)
什么是最好的,wget(找不到很多关于它的文章)或 requests ? 任何帮助表示赞赏。
例如:
import wget
import os
import multiprocessing
def run_process(url, output_path):
wget.download(url, out=output_path)
# TODO: you can write your rename logic at here using os.rename
if __name__ == '__main__':
cpus = multiprocessing.cpu_count()
max_pool_size = 4
pool = multiprocessing.Pool(cpus if cpus < max_pool_size else max_pool_size)
base_dir = os.path.dirname(os.path.abspath(__file__))
target = "NEWCODE"
prefix_list = ["NWZV1WB", "AWU3JAD", "NW96MRD"]
download_list = []
name_list = list(range(1, 23))
name_list.extend(["zoom_side", "zoom_sole", "zoom_side-thumb"])
for prefix in prefix_list:
path = os.path.join(base_dir, prefix)
if not os.path.exists(path):
os.mkdir(path)
if not os.path.isdir(path):
exit()
for name in name_list:
download_list.append(['https://img2.tennis-warehouse.com/360/{p}/{n}.jpg'.format(n=name, p=prefix), path])
for url, path in download_list: # change here to download other files
print('Beginning file download with wget module {n}'.format(n=url))
pool.apply_async(run_process, args=(url, path, ))
# add your code here to download other files
pool.close()
pool.join()
print("finish")
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.