使用Python和/so-sitemap.xml/的Beautiful Soup进行网络抓取

Question

我正在尝试抓取一个页面website / post-sitemap.xml，其中包含针对wordpress网站发布的所有网址。 第一步，我需要列出后站点地图中存在的所有URL的列表。 当我使用request.get并检查输出时，它也会同时打开所有内部url，这很奇怪。 我的意图是首先列出所有url，然后使用循环，在下一个函数中将抓取单个url。 下面是我到目前为止完成的代码。 如果python专家可以帮助我，则需要所有URL作为列表作为最终输出。

我试过使用request.get和openurl，但是似乎没有任何东西只能打开/post-sitemap.xml的基本URL。

import pandas as pd
import numpy as np
from urllib.request import urlopen
from bs4 import BeautifulSoup
import requests
import re

class wordpress_ext_url_cleanup(object):
    def __init__(self,wp_url):
        self.wp_url_raw = wp_url
        self.wp_url = wp_url + '/post-sitemap.xml/'

    def identify_ext_url(self):
        html = requests.get(self.wp_url)
        print(self.wp_url)
        print(html.text)
        soup = BeautifulSoup(html.text,'lxml')
        #print(soup.get_text())
        raw_data = soup.find_all('tr')
        print (raw_data)
        #for link in raw_data:
            #print(link.get("href"))

def main():
    print ("Inside Main Function");
    url="http://punefirst dot com" #(knowingly removed the . so it doesnt look spammy)
    first_call = wordpress_ext_url_cleanup(url)
    first_call.identify_ext_url()


if __name__ == '__main__':
    main()

我需要将发布站点地图中存在的所有548个URL作为一个列表，我将其用于下一个函数进行进一步的抓取。

Answer 1

从服务器返回的文档是XML，并使用XSLT转换为HTML格式（ 此处提供更多信息 ）。 要解析此XML中的所有链接，可以使用以下脚本：

import requests
from bs4 import BeautifulSoup

url = 'http://punefirst.com/post-sitemap.xml/'

soup = BeautifulSoup(requests.get(url).text, 'lxml')

for loc in soup.select('url > loc'):
    print(loc.text)

印刷品：

http://punefirst.com
http://punefirst.com/hospitals/pcmc-hospitals/aditya-birla-memorial-hospital-chinchwad-pune
http://punefirst.com/hospitals/pcmc-hospitals/saijyoti-hospital-and-icu-chinchwad-pune
http://punefirst.com/hospitals/pcmc-hospitals/niramaya-hospital-chinchwad-pune
http://punefirst.com/hospitals/pcmc-hospitals/chetna-hospital-chinchwad-pune
http://punefirst.com/hospitals/hadapsar-hospitals/pbmas-h-v-desai-eye-hospital
http://punefirst.com/hospitals/punecentral-hospitals/shree-sai-prasad-hospital
http://punefirst.com/hospitals/punecentral-hospitals/sadhu-vaswani-missions-medical-complex
http://punefirst.com/hospitals/katraj-kondhwa-hospitals/shivneri-hospital
http://punefirst.com/hospitals/punecentral-hospitals/kelkar-nursing-home
http://punefirst.com/hospitals/pcmc-hospitals/shrinam-hospital
http://punefirst.com/hospitals/pcmc-hospitals/dhanwantari-hospital-nigdi
http://punefirst.com/hospitals/punecentral-hospitals/dr-tarabai-limaye-hospital
http://punefirst.com/hospitals/katraj-kondhwa-hospitals/satyanand-hospital-kondhwa-pune

...and so on.