Web 抓取 - 将文件保存到嵌套文件夹

Question

I am downloading pdf files from different URLs using a built-in API.

My end result should be to download files from each unique link (identified as links in the code below) to unique folders ( folder_location in the code) on the desktop.

I am quite puzzled by how I should arrange codes to do this as I am still a novice. So far I have tried the following.

import os
import requests
from glob import glob
import time
from urllib.parse import urljoin
from bs4 import BeautifulSoup

links = ["P167897", "P173997", "P166309"]

folder_location = "/pdf/"


for link, folder in zip(links, folder_location):
    time.sleep(10)
    end_point = f"https://search.worldbank.org/api/v2/wds?" \
                f"format=json&includepublicdocs=1&" \
                f"fl=docna,lang,docty,repnb,docdt,doc_authr,available_in&" \
                f"os=0&rows=20&proid={link}&apilang=en"
    documents = requests.get(end_point).json()["documents"]
    for document_data in documents.values():
        try:
            pdf_url = document_data["pdfurl"]
            filename = os.path.join(folder,pdf_url.split('/')[-1])
            with open(filename, 'wb') as f:
                f.write(requests.get(pdf_url).content)

EDIT: To clarify, the objects in links are id based on which links to the pdf files are to be identified from the API.

Answer 1

You could try using the pathlib module.

Here's how:

import os
import time
from pathlib import Path

import requests

links = ["P167897", "P173997", "P166309"]

for link in links:
    end_point = f"https://search.worldbank.org/api/v2/wds?" \
                f"format=json&includepublicdocs=1&" \
                f"fl=docna,lang,docty,repnb,docdt,doc_authr,available_in&" \
                f"os=0&rows=20&proid={link}&apilang=en"
    documents = requests.get(end_point).json()["documents"]
    for document_data in documents.values():
        try:
            pdf_url = document_data["pdfurl"]
            file_path = Path(f"pdf/{link}/{pdf_url.rsplit('/')[-1]}")
            file_path.parent.mkdir(parents=True, exist_ok=True)
            with file_path.open("wb") as f:
                f.write(requests.get(pdf_url).content)
            time.sleep(10)
        except KeyError:
            continue

This outputs files to:

pdf/
└── P167897
    ├── Official-Documents-First-Restatement-to-the-Disbursement-Letter-for-Grant-D6810-SL-and-for-Additional-Financing-Grant-TF0B4694.pdf
    └── Official-Documents-Grant-Agreement-for-Additional-Financing-Grant-TF0B4694.pdf
    ...