python爬行beautifulsoup如何爬行几个页面？

Question

Please Help. I want to get all the company names of each pages and they have 12 pages.

http://www.saramin.co.kr/zf_user/jobs/company-labs/list/page/1 http://www.saramin.co.kr/zf_user/jobs/company-labs/list/page/2 -- this website only changes the number.

So Here is my code so far. Can I get just the title (company name) of 12 pages? Thank you in advance.

from bs4 import BeautifulSoup
import requests

maximum = 0
page = 1

URL = 'http://www.saramin.co.kr/zf_user/jobs/company-labs/list/page/1'
response = requests.get(URL)
source = response.text
soup = BeautifulSoup(source, 'html.parser')

whole_source = ""
for page_number in range(1, maximum+1):
URL = 'http://www.saramin.co.kr/zf_user/jobs/company-labs/list/page/' + str(page_number)
response = requests.get(URL)

whole_source = whole_source + response.text
soup = BeautifulSoup(whole_source, 'html.parser')
find_company = soup.select("#content > div.wrap_analysis_data > div.public_con_box.public_list_wrap > ul > li:nth-child(13) > div > strong")

for company in find_company:
print(company.text)

---------Output of one page

---------page source :)

Answer 1

So, you want to remove all the headers and get only the string of the company name? Basically, you can use the soup.findAll to find the list of company in the format like this:

 <strong class="company"><span>중소기업진흥공단</span></strong> 

Then you use the .find function to extract information from the <span> tag:

 <span>중소기업진흥공단</span> 

After that, you use .contents function to get the string from the <span> tag:

'중소기업진흥공단'

So you write a loop to do the same for each page, and make a list called company_list to store the results from each page and append them together.

Here's the code:

from bs4 import BeautifulSoup
import requests

maximum = 12

company_list = [] # List for result storing
for page_number in range(1, maximum+1):
    URL = 'http://www.saramin.co.kr/zf_user/jobs/company-labs/list/page/{}'.format(page_number) 
    response = requests.get(URL)
    print(page_number)
    whole_source = response.text
    soup = BeautifulSoup(whole_source, 'html.parser')
    for entry in soup.findAll('strong', attrs={'class': 'company'}): # Finding all company names in the page
        company_list.append(entry.find('span').contents[0]) # Extracting name from the result

The company_list will give you all the company names you want

Answer 2

I figured it out eventually. Thank you for your answer though!

image : code captured in jupyter notebook

Here is my final code.

from urllib.request import urlopen 
from bs4 import BeautifulSoup

company_list=[]
for n in range(12):
    url = 'http://www.saramin.co.kr/zf_user/jobs/company-labs/list/page/{}'.format(n+1)
    webpage = urlopen(url)
    source = BeautifulSoup(webpage,'html.parser',from_encoding='utf-8')
    companys = source.findAll('strong',{'class':'company'})

    for company in companys:
    company_list.append(company.get_text().strip().replace('\n','').replace('\t','').replace('\r',''))

file = open('company_name1.txt','w',encoding='utf-8')

for company in company_list:
file.write(company+'\n')

file.close()