简体   繁体   中英

Scraping URLs using BeautifulSoup

I am scraping a site for cricket schedules.I am using beatiful soup. Here is the url

www.ecb.c0.uk/stats/fixtures-results?m=1&y=2016

This refers to all fixtures in January 2016.
I am trying to scrape other years and months as well. Is there a way I could change the year in code as part of the scrape? Will I have to place a variable in the URL? Will I have to form a loop or loop through a list?

 from bs4 import BeautifulSoup
 import requests

 html = requests.get("http://www.ecb.co.uk/stats/fixtures-results?m=1& y=2016").text
 soup = BeautifulSoup(html,'lxml')

You can use two loops, first for years, second for months:

base_url = "http://www.ecb.co.uk/stats/fixtures-results?m={month}&y={year}"

for year in range(2000, 2017):
    for month in range(1, 13):
        requests.get(base_url.format(month=month, year=year))

Or more shorter variant with itertools :

for year, month in itertools.product(range(2000, 2017), range(1, 13)):
    requests.get(base_url.format(month=month, year=year))

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM