使用BeautifulSoup解析HTML的所有页面

Question

I'm having problems within my code which works perfectly with one page, but when I try to parse all the 28 pages it doesn't parse 27 pages, but parse only the first one.

The main idea is parse the data from the mentioned url which has 28 pages in overall and I made for loop for it in order to make BS parse from all the pages. However, it parses only the first page, but doesn't parse others.

I would like to get your recommendations and ways to make it work.

Code:

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

for t in range(28):
    url = "https://boss.az/vacancies?action=index&controller=vacancies&only_path=true&page={}&type=vacancies".format(t)
    r = requests.get(url)
    soup = bs(r.content, 'html.parser')

    titles = [i.text for i in soup.select('.results-i-title')]
    #print(titles)
    companies = [i.text for i in soup.select('.results-i-company')]
    #print(companies)
    summaries = [i.text for i in soup.select('.results-i-summary')]

df = pd.DataFrame(list(zip(titles, companies, summaries)), columns = ['Title', 'Company', 'Summary'])
df.to_csv(r'Data.csv', sep=',', encoding='utf-8-sig',index = False )

Answer 1

You are overwriting titles , companies and summaries with every iteration of the loop. Simply change titles = ... to titles += ... :

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

titles = []
companies = []
summaries = []

for t in range(28):
    url = "https://boss.az/vacancies?action=index&controller=vacancies&only_path=true&page={}&type=vacancies".format(t)
    r = requests.get(url)
    soup = bs(r.content, 'html.parser')

    titles += [i.text for i in soup.select('.results-i-title')]
    companies += [i.text for i in soup.select('.results-i-company')]
    summaries += [i.text for i in soup.select('.results-i-summary')]

df = pd.DataFrame(list(zip(titles, companies, summaries)), columns = ['Title', 'Company', 'Summary'])
df.to_csv(r'Data.csv', sep=',', encoding='utf-8-sig',index = False )