Python Web 抓取 - 如何抓取此类网站？

Question

Okay, so I need to scrape the following webpage: https://www.programmableweb.com/category/all/apis?deadpool=1

It's a list of APIs. There are approx 22,000 APIs to scrape.

---

I need to:

1) Get the URL of each API in the table (pages 1-889), and also to scrape the following info:

• API name
• Description
• Category
• Submitted

2) I then need to scrape a bunch of information from each URL.

3) Export the data to a CSV

---

The thing is, I'm a bit lost of how to think about this project. From what I can see, there are no AJAX calls been made to populate the table, which means I'm going to have to parse the HTML directly (right?)

---

In my head, the logic would be something like this:

1. Use the requests & BS4 libraries to scrape the table

2. Then, somehow grab the HREF from every row

3. Access that HREF, scrape the data, move onto the next one

4. Rinse and repeat for all table rows.

---

Am I on the right track, is this possible with requests & BS4?

Here's are some screenshots of what I've been trying to explain.

Thank you SOO much for any help. This is hurting my head haha

Answer 1

You should read more about scraping if you are going to pursue it.

from bs4 import BeautifulSoup
import csv , os , requests
from urllib import parse


def SaveAsCsv(list_of_rows):
    try:
        with open('data.csv', mode='a',  newline='', encoding='utf-8') as outfile:
            csv.writer(outfile).writerow(list_of_rows)
    except PermissionError:
        print("Please make sure data.csv is closed\n")

if os.path.isfile('data.csv') and os.access('data.csv', os.R_OK):
    print("File data.csv Already exists \n")
else:
    SaveAsCsv([ 'api_name','api_link','api_desc','api_cat'])
BaseUrl = 'https://www.programmableweb.com/category/all/apis?deadpool=1&page={}'
for i in range(1, 890):
    print('## Getting Page {} out of 889'.format(i))    
    url = BaseUrl.format(i)
    res = requests.get(url)
    soup = BeautifulSoup(res.text,'html.parser')
    table_rows = soup.select('div.view-content > table[class="views-table cols-4 table"] > tbody tr')
    for row in table_rows:
        tds = row.select('td')
        api_name = tds[0].text.strip()
        api_link = parse.urljoin(url, tds[0].find('a').get('href'))
        api_desc = tds[1].text.strip()
        api_cat  = tds[2].text.strip()  if len(tds) >= 3 else ''
        SaveAsCsv([api_name,api_link,api_desc,api_cat])

Answer 2

Here we go using requests , BeautifulSoup and pandas :

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.programmableweb.com/category/all/apis?deadpool=1&page='

num = int(input('How Many Page to Parse?> '))
print('please wait....')
name = []
desc = []
cat = []
sub = []
for i in range(0, num):
    r = requests.get(f"{url}{i}")
    soup = BeautifulSoup(r.text, 'html.parser')
    for item1 in soup.findAll('td', attrs={'class': 'views-field views-field-title col-md-3'}):
        name.append(item1.text)
    for item2 in soup.findAll('td', attrs={'class': 'views-field views-field-search-api-excerpt views-field-field-api-description hidden-xs visible-md visible-sm col-md-8'}):
        desc.append(item2.text)
    for item3 in soup.findAll('td', attrs={'class': 'views-field views-field-field-article-primary-category'}):
        cat.append(item3.text)
    for item4 in soup.findAll('td', attrs={'class': 'views-field views-field-created'}):
        sub.append(item4.text)

result = []
for item in zip(name, desc, cat, sub):
    result.append(item)

df = pd.DataFrame(
    result, columns=['API Name', 'Description', 'Category', 'Submitted'])
df.to_csv('output.csv')

print('Task Completed, Result saved to output.csv file.')

Result can be viewed online: Check Here

Output Simple:

Now For href parsing:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.programmableweb.com/category/all/apis?deadpool=0&page='

num = int(input('How Many Page to Parse?> '))
print('please wait....')

links = []
for i in range(0, num):
    r = requests.get(f"{url}{i}")
    soup = BeautifulSoup(r.text, 'html.parser')
    for link in soup.findAll('td', attrs={'class': 'views-field views-field-title col-md-3'}):
        for href in link.findAll('a'):
            result = 'https://www.programmableweb.com'+href.get('href')
            links.append(result)

spans = []
for link in links:
    r = requests.get(link)
    soup = soup = BeautifulSoup(r.text, 'html.parser')
    span = [span.text for span in soup.select('div.field span')]
    spans.append(span)

data = []
for item in spans:
    data.append(item)

df = pd.DataFrame(data)
df.to_csv('data.csv')
print('Task Completed, Result saved to data.csv file.')

Check Result Online: Here

Sample View is Below:

In Case if you want those 2 csv files together so here's the code:

import pandas as pd

a = pd.read_csv("output.csv")
b = pd.read_csv("data.csv")
merged = a.merge(b)
merged.to_csv("final.csv", index=False)

Online Result: Here