如何將具有多個頁面和內部鏈接的網站抓取到熊貓數據框中？

Question

我需要從以下鏈接中的每個公司獲取數據，並獲取鏈接中的所有內容。 我需要每個公司的數據都連續。 我遇到的問題是我不確定如何准確地做到這一點。 我不知道采取哪種方法以及從哪里開始。

這是網站： https : //www.adgm.com/public-registers/fsra

我試圖至少將信息放入我的代碼中，然后嘗試從IDE中進行打印，但是我失敗了，我不明白為什么。

import requests
import pandas as pd
from bs4 import BeautifulSoup

res = requests.get("https://www.adgm.com/public-registers/fsra")
soup = BeautifulSoup(res.content,'html.parser')
table  = soup.find_all('.every-accord')

for element in table:
    print(element.text)

這是我一直在嘗試的代碼。 每個表格行都在我嘗試獲取的“每個協議”類中。 它沒有給我任何錯誤，但是我也沒有任何結果。

感謝您的任何幫助。

Answer 1

您可以遍歷容器：

import requests
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.adgm.com/public-registers/fsra').text, 'html.parser')
results = [[c.text for c in i.find_all('div', {'class':'col-sm-6'})]+[i.a['href'], i.find('div', {'class':'col-lg-5'}).text] for i in d.find_all('div', {'class':'every-accord'})]
no_headers = [[i for i in c[1:] if i not in {'Company Status', 'Address'}] for c in results]

輸出：

[['160024', 'Active', 'Level 7, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/aarna-capital-limited', 'Aarna Capital Limited'], ['160007', 'Active', 'Unit 8, 6th floor Al Khatem Tower, Abu Dhabi Global Markets Square, Al Maryah Island Abu Dhabi, United Arab Emirates P.O. Box 764605', '/public-registers/fsra/fsf/aberdeen-asset-middle-east-limited', 'Aberdeen Asset Middle East Limited'], ['180041', 'Active', 'Floor 22, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island', '/public-registers/fsra/fsf/abu-dhabi-catalyst-partners-limited', 'Abu Dhabi Catalyst Partners Limited'], ['180021', 'Active', 'Unit 5, 6th Floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/ad-global-investors-limited', 'AD Global Investors Limited'], ['180039', 'Active', '3419, 34th Floor, Al Maqam Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/ad-investment-management-limited', 'AD Investment Management Limited'], ['170036', 'Active', '10th Floor, Al Sila Tower, ADGM Square, Al Maryah Island', '/public-registers/fsra/fsf/adcb-asset-management-ltd', 'ADCB Asset Management Ltd.'], ['160006', 'Active', 'Level 34, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/adcm-altus-investment-management-limited', 'ADCM Altus Investment Management Limited'], ['160005', 'Active', '33rd floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/adcorp-ltd', 'ADCORP Ltd'], ['180024', 'Active', 'Unit 10, Level 6, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/adnoc-reinsurance-limited', 'ADNOC Reinsurance Limited'], ['170025', 'Active', 'Office 712, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/ads-investment-solutions-limited', 'ADS Investment Solutions Limited']]

編輯：格式化results列：

new_results = [{**{j[i]:j[i+1] for i in range(0, len(j), 2)}, **{'link':a, 'name':b}} for *j, a, b in results]

輸出：

[{'Financial Services Permission Number': '160024', 'Company Status': 'Active', 'Address': 'Level 7, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/aarna-capital-limited', 'name': 'Aarna Capital Limited'}, {'Financial Services Permission Number': '160007', 'Company Status': 'Active', 'Address': 'Unit 8, 6th floor Al Khatem Tower, Abu Dhabi Global Markets Square, Al Maryah Island Abu Dhabi, United Arab Emirates P.O. Box 764605', 'link': '/public-registers/fsra/fsf/aberdeen-asset-middle-east-limited', 'name': 'Aberdeen Asset Middle East Limited'}, {'Financial Services Permission Number': '180041', 'Company Status': 'Active', 'Address': 'Floor 22, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island', 'link': '/public-registers/fsra/fsf/abu-dhabi-catalyst-partners-limited', 'name': 'Abu Dhabi Catalyst Partners Limited'}, {'Financial Services Permission Number': '180021', 'Company Status': 'Active', 'Address': 'Unit 5, 6th Floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/ad-global-investors-limited', 'name': 'AD Global Investors Limited'}, {'Financial Services Permission Number': '180039', 'Company Status': 'Active', 'Address': '3419, 34th Floor, Al Maqam Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/ad-investment-management-limited', 'name': 'AD Investment Management Limited'}, {'Financial Services Permission Number': '170036', 'Company Status': 'Active', 'Address': '10th Floor, Al Sila Tower, ADGM Square, Al Maryah Island', 'link': '/public-registers/fsra/fsf/adcb-asset-management-ltd', 'name': 'ADCB Asset Management Ltd.'}, {'Financial Services Permission Number': '160006', 'Company Status': 'Active', 'Address': 'Level 34, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/adcm-altus-investment-management-limited', 'name': 'ADCM Altus Investment Management Limited'}, {'Financial Services Permission Number': '160005', 'Company Status': 'Active', 'Address': '33rd floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/adcorp-ltd', 'name': 'ADCORP Ltd'}, {'Financial Services Permission Number': '180024', 'Company Status': 'Active', 'Address': 'Unit 10, Level 6, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/adnoc-reinsurance-limited', 'name': 'ADNOC Reinsurance Limited'}, {'Financial Services Permission Number': '170025', 'Company Status': 'Active', 'Address': 'Office 712, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/ads-investment-solutions-limited', 'name': 'ADS Investment Solutions Limited'}]