使用 Python Beautifulsoup 抓取表格和数据

Question

I have scraped the data from this table, using Python-Beautifulsoup, from all the pages for this website and into a dictionary, as seen from the code below.

However, I am also trying to scrape for each company which has it's own separate page ,into that dictionary also.

import requests 
from bs4 import BeautifulSoup
from pprint import pprint 

company_data = []

for i in range(1, 3):
    page = requests.get(f'https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm{i}?')
    soup = BeautifulSoup(page.text, "lxml")

    row_info = soup.select('div.accordion_heading.panel-group.s_list_table')

    for row_info in row_info:
        comapny_info = {}
        comapny_info['Name'] = row_info.select_one('div.col_1 a').text.strip()

pprint(company_data)

Answer 1

I have just done with only for 2M company I believe that helps.

import requests
from bs4 import BeautifulSoup
res=requests.get("https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm").text
soup=BeautifulSoup(res,'html.parser')
comapny_info={}
comapny_info['Profile'] = soup.select('div.text-desc-members')
if len(soup.select('div.text-desc-members'))==0:
  comapny_info['Profile']  = soup.select('div.list-sub')[0].text.strip()

comapny_info['ACOP']=[item['href'] for item in soup.select(".table.table-striped a.files")]
comapny_info['QuestionAnswer']=["Question:" + q.text.strip() + " Answer:" +a.text.strip() for q ,a in zip(soup.select("div.list-reports .m_question"),soup.select("div.list-reports .m_answer")) ]

print(comapny_info)