如何在一个嵌套的“for-loop”中同时抓取两个页面并生成两个不同的列表？

Question

I'm scraping from two URLs that have the same DOM structure, and so I'm trying to find a way to scrape both of them at the same time.
The only caveat is that the data scraped from both these pages need to end up on distinctly named lists.

To explain with example, here is what I've tried:

import os
import requests
from bs4 import BeautifulSoup as bs


urls = ['https://www.basketball-reference.com/leaders/ws_career.html',
       'https://www.basketball-reference.com/leaders/ws_per_48_career.html',]

ws_list = []
ws48_list = []

categories = [ws_list, ws48_list]

for url in urls:
    response = requests.get(url, headers=headers)
    soup = bs(response.content, 'html.parser')
    section = soup.find('table', class_='stats_table')
    for a in section.find_all('a'):
        player_name = a.text
        for cat_list in categories:
            cat_list.append(player_name)
print(ws48_list)
print(ws_list)

This ends up printing two identical lists when I was shooting for 2 lists unique to its page.
How do I accomplish this? Would it be better practice to code it another way?

Answer 1

Just add them to the appropriate list and the problem is solved?

for i, url in enumerate(urls):
    response = requests.get(url)
    soup = bs(response.content, 'html.parser')
    section = soup.find('table', class_='stats_table')
    for a in section.find_all('a'):
        player_name = a.text
        categories[i].append(player_name)
print(ws48_list)
print(ws_list)

Answer 2

Instead of trying to append to already existing lists. Just create new ones. Make a function to do the scrape and pass each url in turn to it.

import os
import requests
from bs4 import BeautifulSoup as bs

urls = ['https://www.basketball-reference.com/leaders/ws_career.html',
       'https://www.basketball-reference.com/leaders/ws_per_48_career.html',]

def parse_page(url, headers={}):

    response = requests.get(url, headers=headers)
    soup = bs(response.content, 'html.parser')
    section = soup.find('table', class_='stats_table')
    return [a.text for a in section.find_all('a')]


ws_list, ws48_list = [parse_page(url) for url in urls]

print('ws_list = %r' % ws_list)
print('ws8_list = %r' % ws48_list)

Answer 3

You can use a function to define your scraping logic, then just call it for your urls.

import os
import requests
from bs4 import BeautifulSoup as bs

def scrape(url):    
    response = requests.get(url)
    soup = bs(response.content, 'html.parser')
    section = soup.find('table', class_='stats_table')
    names = []
    for a in section.find_all('a'):
        player_name = a.text
        names.append(player_name)
    return names    

ws_list = scrape('https://www.basketball-reference.com/leaders/ws_career.html')
ws48_list = scrape('https://www.basketball-reference.com/leaders/ws_per_48_career.html')

print(ws_list)
print(ws48_list)